Re: [discuss] Release Geronimo 1.2

Jay D. McHugh Fri, 06 Apr 2007 13:38:29 -0700

Chris (do you go by Chris or Christopher?),

Here is a patch that I just wrote that allows the exit routine ofConnectionTrackingCoordinator to finish cleanly after a number (5) ofattempts at removing the resource.

If it fails after five tries, then the routine exits and throws aResourceException (that will hopefully be caught further up the stack).


Would you like to try it to see if it solves your concurrency problem?

Jay

Christopher Blythe wrote:

Doubtful... everything tested fine under light browser based testing.As the exception suggests, this is a concurrency problem that youwould only hit under load.

On 4/6/07, * Jay D. McHugh* <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:


    If Matt had no problem deploying and testing DT, could it be a Java
    version or classpath issue?

    That could explain the difference in the exception during deployment
    (and the problems during deployment could possibly explain the run
    time
    problems).

    Jay

    Christopher Blythe wrote:
    > I use a commercial load driving tool... FYI, I'm fairly certain that
    > G-2.0 has the same issue.
    >
    > On 4/6/07, *David Jencks* < [EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>
    > <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>
    wrote:
    >
    >     I think we need to figure out why the
    >     concurrentModificationException is happening before we
    release.  I
    >     think that one possible reason is that we are multithreading
    >     stateless session bean instances.  I hope this isn't the
    cause....
    >     but IMO we need to find out.
    >
    >     Chris, how do you run the several clients?  manually or with
    a tool?
    >
    >     thanks
    >     david jencks
    >
    >
    >     On Apr 6, 2007, at 11:09 AM, Christopher Blythe wrote:
    >
    >>     Gave it a shot... no luck. As soon as I started 2 clients, the
    >>     same exceptions started to pile up. I have attached the
    >>     geronimo.log. Also, noticed the following exception during
    startup.
    >>
    >>     14:05:00,640 ERROR [TransportConnector] Could not accept
    >>     connection from /127.0.0.1:28428: java.io.IOException: Wire
    >>     format negociation timeout: peer did not send his wire format.
    >>     java.io.IOException: Wire format negociation timeout: peer did
    >>     not send his wire format.
    >>         at
    org.apache.activemq.transport.WireFormatNegotiator.oneway
    >>     (WireFormatNegotiator.java :88)
    >>         at

>>org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47)

    >>         at org.apache.activemq.broker.TransportConnection.dispatch
    >>     (TransportConnection.java :1138)
    >>         at

>>org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:805)

    >>         at
    >>     org.apache.activemq.broker.TransportConnection.start
    (TransportConnection.java
    >>     :885)
    >>         at org.apache.activemq.broker.TransportConnector$1.onAccept
    >>     (TransportConnector.java:148)
    >>         at
    >>     org.apache.activemq.transport.tcp.TcpTransportServer.run
    (TcpTransportServer.java:167)
    >>         at java.lang.Thread.run (Thread.java:797)
    >>
    >>
    >>
    >>     On 4/6/07, *Matt Hogstrom* < [EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>
    >>     <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote:
    >>
    >>         Only a very light load from a few browsers.  One thing
    to try
    >>         is to increase the number of SLSBs in the pool.
    >>
    >>         Can you add
    >>
    >>                         <session>
    >>                             <ejb-name>TradeJDBC</ejb-name>
    >>                             <jndi-name>ejb/TradeJDBC</jndi-name>
    >>                             <cache-size>100</cache-size>
    >>                         </session>
    >>
    >>         to your plan and redeploy.  I added some support for
    multiple
    >>         SLSBs in a pool for 1.2 which we did not have before.  This
    >>         will hopefully make it better and not worse :)
    >>
    >>         On Apr 6, 2007, at 11:32 AM, Christopher Blythe wrote:
    >>
    >>>         Matt...
    >>>
    >>>         You mentioned that you deployed DayTrader 1.2... did you
    >>>         happen to run it under load? JDBC/Direct mode looks good;
    >>>         however, I am still seeing
    ConcurrentModificationExceptions
    >>>         while attempting to run more than 1 client in Session
    Direct
    >>>         mode (
    https://issues.apache.org/jira/browse/GERONIMO-2708).
    >>>         These exceptions are thrown throughout the duration of the
    >>>         run. FYI - I deployed the same ear on Geronimo 1.1.1 and
    >>>         didn't have a problem scaling up the users for Session
    >>>         Direct mode.
    >>>
    >>>         java.util.ConcurrentModificationException
    >>>             at
    java.util.HashMap$HashIterator.remove(HashMap.java:861)
    >>>             at

>>>org.apache.geronimo.connector.outbound.connectiontracking.ConnectionTrackingCoordinator.exit

    >>>         ( ConnectionTrackingCoordinator.java :127)
    >>>             at

>>>org.apache.geronimo.connector.outbound.connectiontracking.ConnectionTrackingCoordinator$$FastClassByCGLIB$$5d33aabf.invoke(<generated>)


    >>>             at net.sf.cglib.reflect.FastMethod.invoke
    >>>         (FastMethod.java :53)
    >>>             at

>>>org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke

    (FastMethodInvoker.java:38)
    >>>             at

>>>org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:122)

    >>>             at
    >>>         org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
    >>>         (GBeanInstance.java:820)
    >>>             at

>>>org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)

    >>>             at

>>>org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)

    >>>             at

>>>org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept

    >>>         ( ProxyMethodInterceptor.java:96)
    >>>             at

>>>org.apache.geronimo.connector.outbound.connectiontracking.ConnectionTracker$$EnhancerByCGLIB$$b6b1324a.exit(<generated>)

    >>>             at
    >>>         org.apache.openejb.NoConnectionEnlistingInterceptor.invoke
    >>>         (NoConnectionEnlistingInterceptor.java:70)
    >>>             at
    >>>         org.apache.openejb.SystemExceptionInterceptor.invoke
    (SystemExceptionInterceptor.java:35)
    >>>             at

>>>org.apache.openejb.security.DefaultSubjectInterceptor.invoke(DefaultSubjectInterceptor.java

    >>>         :49)
    >>>             at

>>>org.apache.openejb.slsb.DefaultStatelessEjbContainer.invoke(DefaultStatelessEjbContainer.java:178)

    >>>             at

>>>org.apache.openejb.slsb.DefaultStatelessEjbContainer$$FastClassByCGLIB$$7ad7a562.invoke

    (<generated>)
    >>>
    >>>             at
    >>>         net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
    >>>             at

>>>org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke

    (FastMethodInvoker.java:38)
    >>>             at
    >>>         org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
    >>>         (GBeanOperation.java:122)
    >>>             at

>>>org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:820)

    >>>             at

>>>org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java

    :57)
    >>>             at

>>>org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke

    >>>         (RawOperationInvoker.java:35)
    >>>             at

>>>org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)

    >>>             at

>>>org.apache.openejb.StatelessEjbContainer$$EnhancerByCGLIB$$5c554f35.invoke


    >>>         (<generated>)
    >>>             at

>>>org.apache.openejb.AbstractEjbDeployment.invoke(AbstractEjbDeployment.java:195)

    >>>             at

>>>org.apache.openejb.proxy.EJBMethodInterceptor.intercept(EJBMethodInterceptor.java:145)

    >>>             at

>>>org.apache.openejb.proxy.SessionEJBObject$$EnhancerByCGLIB$$f5a9c1b2.login

    >>>         (<generated>)
    >>>             at

>>>org.apache.geronimo.samples.daytrader.TradeAction.login(TradeAction.java:449)

    >>>             at org.apache.geronimo.samples.daytrader.
    >>>         web.TradeServletAction.doLogin

>>><http://web.TradeServletAction.doLogin>(TradeServletAction.java:364)

    >>>             at org.apache.geronimo.samples.daytrader .
    >>>         web.TradeAppServlet.performTask

>>><http://web.TradeAppServlet.performTask>(TradeAppServlet.java:126)

    >>>             at org.apache.geronimo.samples.daytrader.
    >>>         web.TradeAppServlet.doPost

>>><http://web.TradeAppServlet.doPost>(TradeAppServlet.java :91)

    >>>             at javax.servlet.http.HttpServlet.service
    >>>         (HttpServlet.java:617)
    >>>             at

>>>javax.servlet.http.HttpServlet.service(HttpServlet.java :690)

    >>>             at

>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter

    >>>         (ApplicationFilterChain.java:252)
    >>>             at
    >>>         org.apache.catalina.core.ApplicationFilterChain.doFilter
    >>>         (ApplicationFilterChain.java:173)
    >>>             at org.apache.geronimo.samples.daytrader.
    >>>         web.OrdersAlertFilter.doFilter

>>><http://web.OrdersAlertFilter.doFilter>(OrdersAlertFilter.java:91)

    >>>             at

>>>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter

    (ApplicationFilterChain.java
    >>>         :202)
    >>>             at
    >>>         org.apache.catalina.core.ApplicationFilterChain.doFilter
    >>>         (ApplicationFilterChain.java :173)
    >>>             at

>>>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)

    >>>             at
    org.apache.catalina.core.StandardContextValve.invoke
    >>>         (StandardContextValve.java:178)
    >>>             at

>>>org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke(DefaultSubjectValve.java:56)

    >>>             at

>>>org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java

    >>>         :328)
    >>>             at

>>>org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke

    (GeronimoBeforeAfterValve.java:47)
    >>>             at

>>>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)

    >>>             at org.apache.catalina.valves.ErrorReportValve.invoke
    >>>         (ErrorReportValve.java:105)
    >>>             at

>>>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)

    >>>             at

>>>org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)

    >>>             at org.apache.catalina.connector.CoyoteAdapter.service
    >>>         (CoyoteAdapter.java :148)
    >>>             at

>>>org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)

    >>>             at

>>>org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection

    (Http11BaseProtocol.java
    >>>         :667)
    >>>             at

>>>org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)

    >>>             at

>>>org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)

    >>>             at

>>>org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run

    >>>         (ThreadPool.java:684)
    >>>             at java.lang.Thread.run(Thread.java:797)
    >>>
    >>>         On 4/5/07, *Jason Dillon* < [EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>
    >>>         <mailto:[EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>>> wrote:
    >>>
    >>>             Aight, no worries.  I still don't fully understand
    all
    >>>             that plugin stuff... yet ;-)
    >>>
    >>>             --jason
    >>>
    >>>
    >>>             On Apr 5, 2007, at 3:38 PM, Paul McMahan wrote:
    >>>
    >>>>             The change I have cued up replaces "
    1.2-SNAPSHOT" with
    >>>>             " 1.2" for all the catalog entries.  So it would
    break
    >>>>             anyone using the Geronimo plugin repo from a
    >>>>             1.2-SNAPSHOT server (maybe not a huge deal).  Also,
    >>>>             I've tested the catalog updates by looping http
    >>>>             requests to repo1.maven.org/maven2
    <http://repo1.maven.org/maven2>
    >>>>             <http://repo1.maven.org/maven2> back to my local
    maven
    >>>>             repo.  So I've made some assumptions about the repo
    >>>>             layout that should probably be verified.
    >>>>
    >>>>             Best wishes,
    >>>>             Paul
    >>>>
    >>>>             On Apr 5, 2007, at 6:22 PM, Jason Dillon wrote:
    >>>>
    >>>>>             Will it hurt anything to commit it now?  Or will it
    >>>>>             break things?
    >>>>>
    >>>>>             --jason
    >>>>>
    >>>>>
    >>>>>             On Apr 5, 2007, at 3:14 PM, Paul McMahan wrote:
    >>>>>
    >>>>>>
    >>>>>>             On Apr 5, 2007, at 2:11 PM, Joe Bohn wrote:
    >>>>>>
    >>>>>>>             I couldn't do much with the framework assembly
    as it
    >>>>>>>             requires a plugin repository with 1.2 plugins and
    >>>>>>>             AFAIK there is no such plugin repository available
    >>>>>>>             yet.  Will you be making the plugins available for
    >>>>>>>             1.2 as you make the release available?  If
    not, then
    >>>>>>>             perhaps we shouldn't include the framework
    assembly
    >>>>>>>             in the distribution.
    >>>>>>
    >>>>>>             I updated the plugin catalog stuff in
    >>>>>>             site/trunk/docs/plugins/geronimo- 1.2 locally
    and ran
    >>>>>>             some quick tests of plugin download & install from
    >>>>>>             maven repo.  I'm ready to commit if/when the 1.2
    >>>>>>             artifacts are published to central.
    >>>>>>
    >>>>>>             Best wishes,
    >>>>>>             Paul
    >>>>>>
    >>>>>
    >>>>
    >>>
    >>>
    >>>
    >>>
    >>>         --
    >>>         "I say never be complete, I say stop being perfect, I say
    >>>         let... lets evolve, let the chips fall where they may." -
    >>>         Tyler Durden
    >>
    >>
    >>
    >>
    >>     --
    >>     "I say never be complete, I say stop being perfect, I say
    let...
    >>     lets evolve, let the chips fall where they may." - Tyler Durden
    >>     <geronimo.log>
    >
    >
    >
    >
    > --
    > "I say never be complete, I say stop being perfect, I say let...
    lets
    > evolve, let the chips fall where they may." - Tyler Durden




--

"I say never be complete, I say stop being perfect, I say let... letsevolve, let the chips fall where they may." - Tyler Durden

Index: 
/usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/org/apache/geronimo/connector/outbound/connectiontracking/ConnectionTrackingCoordinator.java
===================================================================
--- 
/usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/org/apache/geronimo/connector/outbound/connectiontracking/ConnectionTrackingCoordinator.java
 (revision 526213)
+++ 
/usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/org/apache/geronimo/connector/outbound/connectiontracking/ConnectionTrackingCoordinator.java
 (working copy)
@@ -124,7 +124,23 @@
 
                 // if no connection remain clear context... we could support 
automatic commit, rollback or exception here
                 if (connections.isEmpty()) {
-                    i.remove();
+                    boolean retry = false;
+                    int numberOfRetries = 0;
+                    do {
+                        try {
+                            i.remove();
+                        } catch (java.util.ConcurrentModificationException ex) 
{
+                            if (numberOfRetries < 5) {
+                                retry = true;
+                            } else {
+                                retry = false;
+                            }
+                            numberOfRetries += 1;
+                        }
+                    } while (retry);
+                    if (numberOfRetries >= 5) {
+                        throw new 
ResourceException("ConcurrentModificationException - Unable to remove 
resource");
+                    }
                 }
             }
         } finally {

Re: [discuss] Release Geronimo 1.2

Reply via email to