Yes true, the bug happens also with the DummySampler of our plugins ( http://code.google.com/p/jmeter-plugins/) and appeared from 2.5.1 only.
Stef On Mon, Sep 26, 2011 at 9:18 PM, sebb <[email protected]> wrote: > On 26 September 2011 09:32, Philippe Mouawad <[email protected]> > wrote: > > Hello I attached all infos on ISSUE: > > > > - Test case > > - Log file in DEBUG > > - Thread Dump > > > > Note that I reproduced it every time with what I described in issue. > > I can also now reproduce the deadlock easily with HC4 and mirror server. > I was using a simplified test case that had caused the deadlock prior > to my fix, but was not causing the dealock afterwards. > This was because it only had one sample - that had been sufficient > originally, but was not sufficient later. > Sorry, that's my fault; should have stayed with the original test case. > > The fix I made to HC4 - overriding the retryRequest() method - > actually only works for a single sampler, because the "interrupted" > variable is fetched from the class instance that was used to create > the override. I introduced the "interrupted" variable, because the fix > from https://issues.apache.org/jira/browse/HTTPCLIENT-1120 did not > seem to work. I need to look at that again. This is needed to stop HC4 > from retrying aborted requests, which I think is a separate issue from > the deadlocks. > > I now think there's a subtle bug in JMeterThread. > I had expected the interrupt to complete, and then the sample to > complete, thus the shutdown code would run after the interrupt code. > However, it looks as though the sample can complete before the > interrupt method returns. > This allows the thread to run the shutdown code whilst the interrupt > is still in progress. > > In turn this triggers the deadlock issue in HC4, but could equally > cause issues with other samplers. > The code needs to ensure that shutdown cannot happen whilst an > interrupt is in progress. > > I'll commit a fix to SVN ASAP. > > Sorry Milamber, but I think we need to cancel this RC vote. > > > I also attached a patch that works on my tests. > > I let you check. > > > > Regards > > Philippe Mouawad > > > > On Sun, Sep 25, 2011 at 1:46 PM, sebb <[email protected]> wrote: > > > >> On 25 September 2011 12:22, sebb <[email protected]> wrote: > >> > On 25 September 2011 01:45, Milamber <[email protected]> wrote: > >> >> > >> >> > >> >> Le 25/09/2011 00:08, sebb a ecrit : > >> >>> On 25 September 2011 01:03, Milamber <[email protected]> wrote: > >> >>> > >> >>>> > >> >>>> Le 23/09/2011 23:38, sebb a ecrit : > >> >>>> > >> >>>>> On 23 September 2011 23:17, sebb <[email protected]> wrote: > >> >>>>> > >> >>>>> > >> >>>>>> On 23 September 2011 18:14, sebb <[email protected]> wrote: > >> >>>>>> > >> >>>>>> > >> >>>>>>> On 23 September 2011 17:15, Milamber <[email protected]> > wrote: > >> >>>>>>> > >> >>>>>>> > >> >>>>>>>> Hello, > >> >>>>>>>> > >> >>>>>>>> It's seems all open bugs with 2.5.1RC1 are closed. > >> >>>>>>>> I will start the RC2 process tomorrow. > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>> OK. > >> >>>>>>> > >> >>>>>>> > >> >>>>>> Just found a problem with the HC4 retries - they prevent StopTest > >> from working. > >> >>>>>> > >> >>>>>> There's a deadlock in the code that prevents the interrupt from > >> >>>>>> working when there is a retry. > >> >>>>>> [At the back of my mind, I thought there was another reason why I > >> set > >> >>>>>> the retry count to 0!] > >> >>>>>> > >> >>>>>> Shutdown works fine, because it does not try to interrupt the > >> sample. > >> >>>>>> > >> >>>>>> I think there's a work-round I can use in the JMeter code, but I > >> don't > >> >>>>>> think we should release the code as is. > >> >>>>>> > >> >>>>>> Sorry. > >> >>>>>> > >> >>>>>> The Java and HC3.1 samplers work fine; it's only HC4 that has the > >> problem. > >> >>>>>> > >> >>>>>> I'll let you know if there's a solution ASAP. > >> >>>>>> > >> >>>>>> > >> >>>>> URL: http://svn.apache.org/viewvc?rev=1175069&view=rev > >> >>>>> Log: > >> >>>>> Temporary hack to work round > >> >>>>> > >> >>>>> > >> >>>> This temporary hack don't always work for me. When I call Stop > command > >> >>>> at the beginning of a test (before end of ramp up), I have the same > >> >>>> deadlock. > >> >>>> (but sometimes the stop works fine.) > >> >>>> > >> >>> Can you do a thread dump when this happens? > >> >>> > >> >> > >> >> > >> >> Found one Java-level deadlock: > >> >> ============================= > >> >> "Thread-205": > >> >> waiting to lock monitor 0x0000000000d0bf78 (object > 0x00000000e2c89ba8, > >> >> a org.apache.http.impl.conn.SingleClientConnManager), > >> >> which is held by "Thread Group 1-1" > >> >> "Thread Group 1-1": > >> >> waiting to lock monitor 0x000000000205e638 (object > 0x00000000e3425510, > >> >> a org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter), > >> >> which is held by "Thread-205" > >> >> > >> >> Java stack information for the threads listed above: > >> >> =================================================== > >> >> "Thread-205": > >> >> at > >> >> > >> > org.apache.http.impl.conn.SingleClientConnManager.releaseConnection(SingleClientConnManager.java:258) > >> >> - waiting to lock <0x00000000e2c89ba8> (a > >> >> org.apache.http.impl.conn.SingleClientConnManager) > >> >> at > >> >> > >> > org.apache.http.impl.conn.AbstractClientConnAdapter.abortConnection(AbstractClientConnAdapter.java:323) > >> >> - locked <0x00000000e3425510> (a > >> >> org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter) > >> >> at > >> >> > >> > org.apache.http.client.methods.HttpRequestBase.abort(HttpRequestBase.java:161) > >> >> at > >> >> > >> > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.interrupt(HTTPHC4Impl.java:1087) > >> >> at > >> >> > >> > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.interrupt(HTTPSamplerProxy.java:77) > >> >> at > >> >> > org.apache.jmeter.threads.JMeterThread.interrupt(JMeterThread.java:580) > >> >> at > >> >> > >> > org.apache.jmeter.engine.StandardJMeterEngine.tellThreadsToStop(StandardJMeterEngine.java:552) > >> >> at > >> >> > >> > org.apache.jmeter.engine.StandardJMeterEngine.access$300(StandardJMeterEngine.java:58) > >> >> at > >> >> > >> > org.apache.jmeter.engine.StandardJMeterEngine$StopTest.run(StandardJMeterEngine.java:284) > >> >> at java.lang.Thread.run(Thread.java:722) > >> >> "Thread Group 1-1": > >> >> at > >> >> > >> > org.apache.http.impl.conn.AbstractPooledConnAdapter.detach(AbstractPooledConnAdapter.java:106) > >> >> - waiting to lock <0x00000000e3425510> (a > >> >> org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter) > >> >> at > >> >> > >> > org.apache.http.impl.conn.SingleClientConnManager.shutdown(SingleClientConnManager.java:342) > >> >> - locked <0x00000000e2c89ba8> (a > >> >> org.apache.http.impl.conn.SingleClientConnManager) > >> >> at > >> >> > >> > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.closeThreadLocalConnections(HTTPHC4Impl.java:1072) > >> >> at > >> >> > >> > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.threadFinished(HTTPHC4Impl.java:1061) > >> >> at > >> >> > >> > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.threadFinished(HTTPSamplerProxy.java:71) > >> >> at > >> >> > >> > org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:553) > >> >> at > >> >> > org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:986) > >> >> at > >> org.apache.jorphan.collections.HashTree.traverse(HashTree.java:969) > >> >> at > >> >> > >> > org.apache.jmeter.threads.JMeterThread.threadFinished(JMeterThread.java:528) > >> >> at > org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:308) > >> >> at java.lang.Thread.run(Thread.java:722) > >> >> > >> >> Found 1 deadlock. > >> >> > >> >> > >> >> > >> >> Full thread dump in attachment. > >> > > >> > Thanks! > >> > > >> >> Test case is the SimpleTest.jmx, stop after 6-7 VU. > >> >> For this thread dump, I wait the end of ramp up for 100 VU, but only > 99 > >> >> up. (the 'last' is the TG 1-1) > >> >> > >> >> > >> >>> > >> >>> > >> >>>> I thinks we must release the version 2.5.1 to correct the others > bugs > >> >>>> already fixed, and add a known problem in documentation for this > >> deadlook? > >> >>>> > >> >>> Yes, that sounds reasonable. It's not clear yet whether this is a > >> >>> JMeter or HC4 problem; nor is it clear what to do to fix it. > >> >>> Anyway, it only occurs sometimes, and it only occurs when trying to > >> >>> stop the test - so the GUI can just be exitted. > >> >>> > >> >> > >> >> Ok, I will prepare the RC2 tomorrow. > >> >> (Perhaps open a jmeter bugzilla for this bug?) > >> > > >> > I've opened https://issues.apache.org/jira/browse/HTTPCLIENT-1127 in > >> > case the bug is in HC4. > >> > Looks like that may have a lock ordering problem. > >> > > >> > But it may be caused by a bug in the way JMeter processes shutdown; > >> > I'm a bit surprised that threadFinished occurs during interrupt > >> > processing. > >> > I'll raise a JMeter bug and take a further look later today. > >> > >> Raised https://issues.apache.org/bugzilla/show_bug.cgi?id=51888 > >> > >> I don't suppose you have the corresponding JMeter log file? > >> I think we need both dump and log to debug the problem. > >> > >> If not, never mind, I'll see if I can trigger the fault again. > >> > >> > > >> >> Milamber > >> >> > >> >>> > >> >>>> Milamber > >> >>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1120 > >> >>>>> Note: copying the code from the patch did not seem to work; it > looks > >> >>>>> like isAborted() was not set. > >> >>>>> Hopefully that is fixed in 4.1.3 > >> >>>>> > >> >>>>> That seems to have fixed it for me, or at least improved matters. > >> >>>>> > >> >>>>> Still needs more testing to see if the deadlock I found - reported > in > >> >>>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1127 - can still > >> >>>>> occur. > >> >>>>> > >> >>>>> BTW, I found the deadlock testing against the mirror server. > >> >>>>> > >> >>>>> > --------------------------------------------------------------------- > >> >>>>> To unsubscribe, e-mail: [email protected] > >> >>>>> For additional commands, e-mail: [email protected] > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >> >>>> > >> >>>> > --------------------------------------------------------------------- > >> >>>> To unsubscribe, e-mail: [email protected] > >> >>>> For additional commands, e-mail: [email protected] > >> >>>> > >> >>>> > >> >>>> > >> >>> > --------------------------------------------------------------------- > >> >>> To unsubscribe, e-mail: [email protected] > >> >>> For additional commands, e-mail: [email protected] > >> >>> > >> >>> > >> >>> > >> >> > >> >> > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > > > > -- > > Cordialement. > > Philippe Mouawad. > > Ubik-Ingénierie > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
