Re: [discuss] Release Geronimo 1.2

Fri, 06 Apr 2007 19:29:22 -0700 

Hello all,

New attempt that doesn't just hide the problem.
I managed to find reference on how to make a linked list behave as synchronized (above and beyond simply trying to access them from within synchronized code blocks).
Attached is the patch - It is actually for OpenEJB...I'm going to wait until someone has a chance to confirm that it actually resolves the Geronimo/Daytrader issue then make the JIRA over on OpenEJB. 

Jay

David Jencks wrote:
I don't think this is acceptable. There should be only one thread working with the context at a time. Either this exception is caused by modifying the collection in the same thread in which case we can fix it easily or it is caused by more than one thread having access to a context at once. Since by my reading of the code this is a context attached to a stateless session bean instance, that would mean that more than one thread is using a stateless session bean instance at once, which is definitely cause to -1 the release.
I hope there's another possibility I haven't thought of..... but hiding the problem is not acceptable unless we really understand what is going on and are really convinced it's harmless. 

thanks
david jencks

On Apr 6, 2007, at 1:40 PM, Jay D. McHugh wrote:


Chris (do you go by Chris or Christopher?),
Here is a patch that I just wrote that allows the exit routine of ConnectionTrackingCoordinator to finish cleanly after a number (5) of attempts at removing the resource.
If it fails after five tries, then the routine exits and throws a ResourceException (that will hopefully be caught further up the stack). 

Would you like to try it to see if it solves your concurrency problem?

Jay

Christopher Blythe wrote:
Doubtful... everything tested fine under light browser based testing. As the exception suggests, this is a concurrency problem that you would only hit under load.
On 4/6/07, * Jay D. McHugh* <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote: 

    If Matt had no problem deploying and testing DT, could it be a Java
    version or classpath issue?
That could explain the difference in the exception during deployment 
    (and the problems during deployment could possibly explain the run
    time
    problems).

    Jay

    Christopher Blythe wrote:
> I use a commercial load driving tool... FYI, I'm fairly certain that 
    > G-2.0 has the same issue.
    >
    > On 4/6/07, *David Jencks* < [EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>
    > <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>>
    wrote:
    >
    >     I think we need to figure out why the
    >     concurrentModificationException is happening before we
    release.  I
    >     think that one possible reason is that we are multithreading
    >     stateless session bean instances.  I hope this isn't the
    cause....
    >     but IMO we need to find out.
    >
    >     Chris, how do you run the several clients?  manually or with
    a tool?
    >
    >     thanks
    >     david jencks
    >
    >
    >     On Apr 6, 2007, at 11:09 AM, Christopher Blythe wrote:
    >
>> Gave it a shot... no luck. As soon as I started 2 clients, the 
    >>     same exceptions started to pile up. I have attached the
    >>     geronimo.log. Also, noticed the following exception during
    startup.
    >>
    >>     14:05:00,640 ERROR [TransportConnector] Could not accept
    >>     connection from /127.0.0.1:28428: java.io.IOException: Wire
>> format negociation timeout: peer did not send his wire format. >> java.io.IOException: Wire format negociation timeout: peer did 
    >>     not send his wire format.
    >>         at
    org.apache.activemq.transport.WireFormatNegotiator.oneway
    >>     (WireFormatNegotiator.java :88)
    >>         at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:47) >> at org.apache.activemq.broker.TransportConnection.dispatch 
    >>     (TransportConnection.java :1138)
    >>         at
>> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:805) 
    >>         at
    >>     org.apache.activemq.broker.TransportConnection.start
    (TransportConnection.java
    >>     :885)
>> at org.apache.activemq.broker.TransportConnector$1.onAccept 
    >>     (TransportConnector.java:148)
    >>         at
    >>     org.apache.activemq.transport.tcp.TcpTransportServer.run
    (TcpTransportServer.java:167)
    >>         at java.lang.Thread.run (Thread.java:797)
    >>
    >>
    >>
    >>     On 4/6/07, *Matt Hogstrom* < [EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>
>> <mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>>> wrote: 
    >>
    >>         Only a very light load from a few browsers.  One thing
    to try
    >>         is to increase the number of SLSBs in the pool.
    >>
    >>         Can you add
    >>
    >>                         <session>
    >>                             <ejb-name>TradeJDBC</ejb-name>
    >>                             <jndi-name>ejb/TradeJDBC</jndi-name>
    >>                             <cache-size>100</cache-size>
    >>                         </session>
    >>
    >>         to your plan and redeploy.  I added some support for
    multiple
>> SLSBs in a pool for 1.2 which we did not have before. This 
    >>         will hopefully make it better and not worse :)
    >>
    >>         On Apr 6, 2007, at 11:32 AM, Christopher Blythe wrote:
    >>
    >>>         Matt...
    >>>
>>> You mentioned that you deployed DayTrader 1.2... did you >>> happen to run it under load? JDBC/Direct mode looks good; 
    >>>         however, I am still seeing
    ConcurrentModificationExceptions
    >>>         while attempting to run more than 1 client in Session
    Direct
    >>>         mode (
    https://issues.apache.org/jira/browse/GERONIMO-2708).
>>> These exceptions are thrown throughout the duration of the >>> run. FYI - I deployed the same ear on Geronimo 1.1.1 and 
    >>>         didn't have a problem scaling up the users for Session
    >>>         Direct mode.
    >>>
    >>>         java.util.ConcurrentModificationException
    >>>             at
    java.util.HashMap$HashIterator.remove(HashMap.java:861)
    >>>             at
>>> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionTrackingCoordinator.exit 
    >>>         ( ConnectionTrackingCoordinator.java :127)
    >>>             at
>>> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionTrackingCoordinator$$FastClassByCGLIB$$5d33aabf.invoke(<generated>) 

    >>>             at net.sf.cglib.reflect.FastMethod.invoke
    >>>         (FastMethod.java :53)
    >>>             at
>>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke 
    (FastMethodInvoker.java:38)
    >>>             at
>>> org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:122) 
    >>>             at
    >>>         org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
    >>>         (GBeanInstance.java:820)
    >>>             at
>>> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57) 
    >>>             at
>>> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35) 
    >>>             at
>>> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept 
    >>>         ( ProxyMethodInterceptor.java:96)
    >>>             at
>>> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionTracker$$EnhancerByCGLIB$$b6b1324a.exit(<generated>) 
    >>>             at
>>> org.apache.openejb.NoConnectionEnlistingInterceptor.invoke 
    >>>         (NoConnectionEnlistingInterceptor.java:70)
    >>>             at
    >>>         org.apache.openejb.SystemExceptionInterceptor.invoke
    (SystemExceptionInterceptor.java:35)
    >>>             at
>>> org.apache.openejb.security.DefaultSubjectInterceptor.invoke(DefaultSubjectInterceptor.java 
    >>>         :49)
    >>>             at
>>> org.apache.openejb.slsb.DefaultStatelessEjbContainer.invoke(DefaultStatelessEjbContainer.java:178) 
    >>>             at
>>> org.apache.openejb.slsb.DefaultStatelessEjbContainer$$FastClassByCGLIB$$7ad7a562.invoke 
    (<generated>)
    >>>
    >>>             at
>>> net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53) 
    >>>             at
>>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke 
    (FastMethodInvoker.java:38)
    >>>             at
    >>>         org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
    >>>         (GBeanOperation.java:122)
    >>>             at
>>> org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:820) 
    >>>             at
>>> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java 
    :57)
    >>>             at
>>> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke 
    >>>         (RawOperationInvoker.java:35)
    >>>             at
>>> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96) 
    >>>             at
>>> org.apache.openejb.StatelessEjbContainer$$EnhancerByCGLIB$$5c554f35.invoke 

    >>>         (<generated>)
    >>>             at
>>> org.apache.openejb.AbstractEjbDeployment.invoke(AbstractEjbDeployment.java:195) 
    >>>             at
>>> org.apache.openejb.proxy.EJBMethodInterceptor.intercept(EJBMethodInterceptor.java:145) 
    >>>             at
>>> org.apache.openejb.proxy.SessionEJBObject$$EnhancerByCGLIB$$f5a9c1b2.login 
    >>>         (<generated>)
    >>>             at
>>> org.apache.geronimo.samples.daytrader.TradeAction.login(TradeAction.java:449) 
    >>>             at org.apache.geronimo.samples.daytrader.
    >>>         web.TradeServletAction.doLogin
>>> <http://web.TradeServletAction.doLogin>(TradeServletAction.java:364) 
    >>>             at org.apache.geronimo.samples.daytrader .
    >>>         web.TradeAppServlet.performTask
>>> <http://web.TradeAppServlet.performTask>(TradeAppServlet.java:126) 
    >>>             at org.apache.geronimo.samples.daytrader.
    >>>         web.TradeAppServlet.doPost
>>> <http://web.TradeAppServlet.doPost>(TradeAppServlet.java :91) 
    >>>             at javax.servlet.http.HttpServlet.service
    >>>         (HttpServlet.java:617)
    >>>             at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java :690) 
    >>>             at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter 
    >>>         (ApplicationFilterChain.java:252)
    >>>             at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter 
    >>>         (ApplicationFilterChain.java:173)
    >>>             at org.apache.geronimo.samples.daytrader.
    >>>         web.OrdersAlertFilter.doFilter
>>> <http://web.OrdersAlertFilter.doFilter>(OrdersAlertFilter.java:91) 
    >>>             at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter 
    (ApplicationFilterChain.java
    >>>         :202)
    >>>             at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter 
    >>>         (ApplicationFilterChain.java :173)
    >>>             at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
    >>>             at
    org.apache.catalina.core.StandardContextValve.invoke
    >>>         (StandardContextValve.java:178)
    >>>             at
>>> org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke(DefaultSubjectValve.java:56) 
    >>>             at
>>> org.apache.geronimo.tomcat.GeronimoStandardContext$SystemMethodValve.invoke(GeronimoStandardContext.java 
    >>>         :328)
    >>>             at
>>> org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke 
    (GeronimoBeforeAfterValve.java:47)
    >>>             at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) >>> at org.apache.catalina.valves.ErrorReportValve.invoke 
    >>>         (ErrorReportValve.java:105)
    >>>             at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) 
    >>>             at
>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541) >>> at org.apache.catalina.connector.CoyoteAdapter.service 
    >>>         (CoyoteAdapter.java :148)
    >>>             at
>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) 
    >>>             at
>>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection 
    (Http11BaseProtocol.java
    >>>         :667)
    >>>             at
>>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) 
    >>>             at
>>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) 
    >>>             at
>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run 
    >>>         (ThreadPool.java:684)
    >>>             at java.lang.Thread.run(Thread.java:797)
    >>>
    >>>         On 4/5/07, *Jason Dillon* < [EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>
    >>>         <mailto:[EMAIL PROTECTED]
    <mailto:[EMAIL PROTECTED]>>> wrote:
    >>>
    >>>             Aight, no worries.  I still don't fully understand
    all
    >>>             that plugin stuff... yet ;-)
    >>>
    >>>             --jason
    >>>
    >>>
    >>>             On Apr 5, 2007, at 3:38 PM, Paul McMahan wrote:
    >>>
    >>>>             The change I have cued up replaces "
    1.2-SNAPSHOT" with
    >>>>             " 1.2" for all the catalog entries.  So it would
    break
    >>>>             anyone using the Geronimo plugin repo from a
>>>> 1.2-SNAPSHOT server (maybe not a huge deal). Also, 
    >>>>             I've tested the catalog updates by looping http
    >>>>             requests to repo1.maven.org/maven2
    <http://repo1.maven.org/maven2>
    >>>>             <http://repo1.maven.org/maven2> back to my local
    maven
>>>> repo. So I've made some assumptions about the repo 
    >>>>             layout that should probably be verified.
    >>>>
    >>>>             Best wishes,
    >>>>             Paul
    >>>>
    >>>>             On Apr 5, 2007, at 6:22 PM, Jason Dillon wrote:
    >>>>
>>>>> Will it hurt anything to commit it now? Or will it 
    >>>>>             break things?
    >>>>>
    >>>>>             --jason
    >>>>>
    >>>>>
    >>>>>             On Apr 5, 2007, at 3:14 PM, Paul McMahan wrote:
    >>>>>
    >>>>>>
    >>>>>>             On Apr 5, 2007, at 2:11 PM, Joe Bohn wrote:
    >>>>>>
    >>>>>>>             I couldn't do much with the framework assembly
    as it
>>>>>>> requires a plugin repository with 1.2 plugins and >>>>>>> AFAIK there is no such plugin repository available >>>>>>> yet. Will you be making the plugins available for 
    >>>>>>>             1.2 as you make the release available?  If
    not, then
    >>>>>>>             perhaps we shouldn't include the framework
    assembly
    >>>>>>>             in the distribution.
    >>>>>>
    >>>>>>             I updated the plugin catalog stuff in
    >>>>>>             site/trunk/docs/plugins/geronimo- 1.2 locally
    and ran
>>>>>> some quick tests of plugin download & install from 
    >>>>>>             maven repo.  I'm ready to commit if/when the 1.2
    >>>>>>             artifacts are published to central.
    >>>>>>
    >>>>>>             Best wishes,
    >>>>>>             Paul
    >>>>>>
    >>>>>
    >>>>
    >>>
    >>>
    >>>
    >>>
    >>>         --
>>> "I say never be complete, I say stop being perfect, I say >>> let... lets evolve, let the chips fall where they may." - 
    >>>         Tyler Durden
    >>
    >>
    >>
    >>
    >>     --
    >>     "I say never be complete, I say stop being perfect, I say
    let...
>> lets evolve, let the chips fall where they may." - Tyler Durden 
    >>     <geronimo.log>
    >
    >
    >
    >
    > --
    > "I say never be complete, I say stop being perfect, I say let...
    lets
    > evolve, let the chips fall where they may." - Tyler Durden
--"I say never be complete, I say stop being perfect, I say let... lets evolve, let the chips fall where they may." - Tyler Durden
Index: /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/org/apache/geronimo/connector/outbound/connectiontracking/ConnectionTrackingCoordinator.java 
===================================================================
--- /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/org/apache/geronimo/connector/outbound/connectiontracking/ConnectionTrackingCoordinator.java (revision 526213) +++ /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/org/apache/geronimo/connector/outbound/connectiontracking/ConnectionTrackingCoordinator.java (working copy) 
@@ -124,7 +124,23 @@
// if no connection remain clear context... we could support automatic commit, rollback or exception here 
                 if (connections.isEmpty()) {
-                    i.remove();
+                    boolean retry = false;
+                    int numberOfRetries = 0;
+                    do {
+                        try {
+                            i.remove();
+ } catch (java.util.ConcurrentModificationException ex) { 
+                            if (numberOfRetries < 5) {
+                                retry = true;
+                            } else {
+                                retry = false;
+                            }
+                            numberOfRetries += 1;
+                        }
+                    } while (retry);
+                    if (numberOfRetries >= 5) {
+ throw new ResourceException("ConcurrentModificationException - Unable to remove resource"); 
+                    }
                 }
             }
         } finally {






Index: 
modules/openejb-core/src/main/java/org/apache/openejb/util/SoftLimitedInstancePool.java
===================================================================
--- 
modules/openejb-core/src/main/java/org/apache/openejb/util/SoftLimitedInstancePool.java
     (revision 526318)
+++ 
modules/openejb-core/src/main/java/org/apache/openejb/util/SoftLimitedInstancePool.java
     (working copy)
@@ -16,6 +16,7 @@
  */
 package org.apache.openejb.util;
 
+import java.util.Collections;
 import java.util.LinkedList;
 import java.io.Serializable;
 
@@ -35,7 +36,7 @@
     public SoftLimitedInstancePool(final InstanceFactory factory, final int 
maxSize) {
         this.factory = factory;
         this.maxSize = maxSize;
-        pool = new LinkedList();
+        pool = (LinkedList) Collections.synchronizedList(new 
java.util.LinkedList());
     }
 
     public Object acquire() throws Exception {

Reply via email to