On 26 September 2011 09:32, Philippe Mouawad <philippe.moua...@gmail.com> wrote: > Hello I attached all infos on ISSUE: > > - Test case > - Log file in DEBUG > - Thread Dump > > Note that I reproduced it every time with what I described in issue.
I can also now reproduce the deadlock easily with HC4 and mirror server. I was using a simplified test case that had caused the deadlock prior to my fix, but was not causing the dealock afterwards. This was because it only had one sample - that had been sufficient originally, but was not sufficient later. Sorry, that's my fault; should have stayed with the original test case. The fix I made to HC4 - overriding the retryRequest() method - actually only works for a single sampler, because the "interrupted" variable is fetched from the class instance that was used to create the override. I introduced the "interrupted" variable, because the fix from https://issues.apache.org/jira/browse/HTTPCLIENT-1120 did not seem to work. I need to look at that again. This is needed to stop HC4 from retrying aborted requests, which I think is a separate issue from the deadlocks. I now think there's a subtle bug in JMeterThread. I had expected the interrupt to complete, and then the sample to complete, thus the shutdown code would run after the interrupt code. However, it looks as though the sample can complete before the interrupt method returns. This allows the thread to run the shutdown code whilst the interrupt is still in progress. In turn this triggers the deadlock issue in HC4, but could equally cause issues with other samplers. The code needs to ensure that shutdown cannot happen whilst an interrupt is in progress. I'll commit a fix to SVN ASAP. Sorry Milamber, but I think we need to cancel this RC vote. > I also attached a patch that works on my tests. > I let you check. > > Regards > Philippe Mouawad > > On Sun, Sep 25, 2011 at 1:46 PM, sebb <seb...@gmail.com> wrote: > >> On 25 September 2011 12:22, sebb <seb...@gmail.com> wrote: >> > On 25 September 2011 01:45, Milamber <milam...@apache.org> wrote: >> >> >> >> >> >> Le 25/09/2011 00:08, sebb a ecrit : >> >>> On 25 September 2011 01:03, Milamber <milam...@apache.org> wrote: >> >>> >> >>>> >> >>>> Le 23/09/2011 23:38, sebb a ecrit : >> >>>> >> >>>>> On 23 September 2011 23:17, sebb <seb...@gmail.com> wrote: >> >>>>> >> >>>>> >> >>>>>> On 23 September 2011 18:14, sebb <seb...@gmail.com> wrote: >> >>>>>> >> >>>>>> >> >>>>>>> On 23 September 2011 17:15, Milamber <milam...@apache.org> wrote: >> >>>>>>> >> >>>>>>> >> >>>>>>>> Hello, >> >>>>>>>> >> >>>>>>>> It's seems all open bugs with 2.5.1RC1 are closed. >> >>>>>>>> I will start the RC2 process tomorrow. >> >>>>>>>> >> >>>>>>>> >> >>>>>>> OK. >> >>>>>>> >> >>>>>>> >> >>>>>> Just found a problem with the HC4 retries - they prevent StopTest >> from working. >> >>>>>> >> >>>>>> There's a deadlock in the code that prevents the interrupt from >> >>>>>> working when there is a retry. >> >>>>>> [At the back of my mind, I thought there was another reason why I >> set >> >>>>>> the retry count to 0!] >> >>>>>> >> >>>>>> Shutdown works fine, because it does not try to interrupt the >> sample. >> >>>>>> >> >>>>>> I think there's a work-round I can use in the JMeter code, but I >> don't >> >>>>>> think we should release the code as is. >> >>>>>> >> >>>>>> Sorry. >> >>>>>> >> >>>>>> The Java and HC3.1 samplers work fine; it's only HC4 that has the >> problem. >> >>>>>> >> >>>>>> I'll let you know if there's a solution ASAP. >> >>>>>> >> >>>>>> >> >>>>> URL: http://svn.apache.org/viewvc?rev=1175069&view=rev >> >>>>> Log: >> >>>>> Temporary hack to work round >> >>>>> >> >>>>> >> >>>> This temporary hack don't always work for me. When I call Stop command >> >>>> at the beginning of a test (before end of ramp up), I have the same >> >>>> deadlock. >> >>>> (but sometimes the stop works fine.) >> >>>> >> >>> Can you do a thread dump when this happens? >> >>> >> >> >> >> >> >> Found one Java-level deadlock: >> >> ============================= >> >> "Thread-205": >> >> waiting to lock monitor 0x0000000000d0bf78 (object 0x00000000e2c89ba8, >> >> a org.apache.http.impl.conn.SingleClientConnManager), >> >> which is held by "Thread Group 1-1" >> >> "Thread Group 1-1": >> >> waiting to lock monitor 0x000000000205e638 (object 0x00000000e3425510, >> >> a org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter), >> >> which is held by "Thread-205" >> >> >> >> Java stack information for the threads listed above: >> >> =================================================== >> >> "Thread-205": >> >> at >> >> >> org.apache.http.impl.conn.SingleClientConnManager.releaseConnection(SingleClientConnManager.java:258) >> >> - waiting to lock <0x00000000e2c89ba8> (a >> >> org.apache.http.impl.conn.SingleClientConnManager) >> >> at >> >> >> org.apache.http.impl.conn.AbstractClientConnAdapter.abortConnection(AbstractClientConnAdapter.java:323) >> >> - locked <0x00000000e3425510> (a >> >> org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter) >> >> at >> >> >> org.apache.http.client.methods.HttpRequestBase.abort(HttpRequestBase.java:161) >> >> at >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.interrupt(HTTPHC4Impl.java:1087) >> >> at >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.interrupt(HTTPSamplerProxy.java:77) >> >> at >> >> org.apache.jmeter.threads.JMeterThread.interrupt(JMeterThread.java:580) >> >> at >> >> >> org.apache.jmeter.engine.StandardJMeterEngine.tellThreadsToStop(StandardJMeterEngine.java:552) >> >> at >> >> >> org.apache.jmeter.engine.StandardJMeterEngine.access$300(StandardJMeterEngine.java:58) >> >> at >> >> >> org.apache.jmeter.engine.StandardJMeterEngine$StopTest.run(StandardJMeterEngine.java:284) >> >> at java.lang.Thread.run(Thread.java:722) >> >> "Thread Group 1-1": >> >> at >> >> >> org.apache.http.impl.conn.AbstractPooledConnAdapter.detach(AbstractPooledConnAdapter.java:106) >> >> - waiting to lock <0x00000000e3425510> (a >> >> org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter) >> >> at >> >> >> org.apache.http.impl.conn.SingleClientConnManager.shutdown(SingleClientConnManager.java:342) >> >> - locked <0x00000000e2c89ba8> (a >> >> org.apache.http.impl.conn.SingleClientConnManager) >> >> at >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.closeThreadLocalConnections(HTTPHC4Impl.java:1072) >> >> at >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.threadFinished(HTTPHC4Impl.java:1061) >> >> at >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.threadFinished(HTTPSamplerProxy.java:71) >> >> at >> >> >> org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:553) >> >> at >> >> org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:986) >> >> at >> org.apache.jorphan.collections.HashTree.traverse(HashTree.java:969) >> >> at >> >> >> org.apache.jmeter.threads.JMeterThread.threadFinished(JMeterThread.java:528) >> >> at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:308) >> >> at java.lang.Thread.run(Thread.java:722) >> >> >> >> Found 1 deadlock. >> >> >> >> >> >> >> >> Full thread dump in attachment. >> > >> > Thanks! >> > >> >> Test case is the SimpleTest.jmx, stop after 6-7 VU. >> >> For this thread dump, I wait the end of ramp up for 100 VU, but only 99 >> >> up. (the 'last' is the TG 1-1) >> >> >> >> >> >>> >> >>> >> >>>> I thinks we must release the version 2.5.1 to correct the others bugs >> >>>> already fixed, and add a known problem in documentation for this >> deadlook? >> >>>> >> >>> Yes, that sounds reasonable. It's not clear yet whether this is a >> >>> JMeter or HC4 problem; nor is it clear what to do to fix it. >> >>> Anyway, it only occurs sometimes, and it only occurs when trying to >> >>> stop the test - so the GUI can just be exitted. >> >>> >> >> >> >> Ok, I will prepare the RC2 tomorrow. >> >> (Perhaps open a jmeter bugzilla for this bug?) >> > >> > I've opened https://issues.apache.org/jira/browse/HTTPCLIENT-1127 in >> > case the bug is in HC4. >> > Looks like that may have a lock ordering problem. >> > >> > But it may be caused by a bug in the way JMeter processes shutdown; >> > I'm a bit surprised that threadFinished occurs during interrupt >> > processing. >> > I'll raise a JMeter bug and take a further look later today. >> >> Raised https://issues.apache.org/bugzilla/show_bug.cgi?id=51888 >> >> I don't suppose you have the corresponding JMeter log file? >> I think we need both dump and log to debug the problem. >> >> If not, never mind, I'll see if I can trigger the fault again. >> >> > >> >> Milamber >> >> >> >>> >> >>>> Milamber >> >>>> >> >>>> >> >>>> >> >>>> >> >>>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1120 >> >>>>> Note: copying the code from the patch did not seem to work; it looks >> >>>>> like isAborted() was not set. >> >>>>> Hopefully that is fixed in 4.1.3 >> >>>>> >> >>>>> That seems to have fixed it for me, or at least improved matters. >> >>>>> >> >>>>> Still needs more testing to see if the deadlock I found - reported in >> >>>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1127 - can still >> >>>>> occur. >> >>>>> >> >>>>> BTW, I found the deadlock testing against the mirror server. >> >>>>> >> >>>>> --------------------------------------------------------------------- >> >>>>> To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org >> >>>>> For additional commands, e-mail: dev-h...@jakarta.apache.org >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org >> >>>> For additional commands, e-mail: dev-h...@jakarta.apache.org >> >>>> >> >>>> >> >>>> >> >>> --------------------------------------------------------------------- >> >>> To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org >> >>> For additional commands, e-mail: dev-h...@jakarta.apache.org >> >>> >> >>> >> >>> >> >> >> >> >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org >> For additional commands, e-mail: dev-h...@jakarta.apache.org >> >> > > > -- > Cordialement. > Philippe Mouawad. > Ubik-Ingénierie > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@jakarta.apache.org For additional commands, e-mail: dev-h...@jakarta.apache.org
