2011/9/26 Stéphane Hoblingre <[email protected]>: > Yes true, the bug happens also with the DummySampler of our plugins ( > http://code.google.com/p/jmeter-plugins/) and appeared from 2.5.1 only.
Not sure that has the same cause; as far as I can tell DummySampler does not support interrupt(), which is the origin of the deadlocks in HC4. Please could you attach a thread dump and jmeter log file to the bug? > Stef > > On Mon, Sep 26, 2011 at 9:18 PM, sebb <[email protected]> wrote: > >> On 26 September 2011 09:32, Philippe Mouawad <[email protected]> >> wrote: >> > Hello I attached all infos on ISSUE: >> > >> > - Test case >> > - Log file in DEBUG >> > - Thread Dump >> > >> > Note that I reproduced it every time with what I described in issue. >> >> I can also now reproduce the deadlock easily with HC4 and mirror server. >> I was using a simplified test case that had caused the deadlock prior >> to my fix, but was not causing the dealock afterwards. >> This was because it only had one sample - that had been sufficient >> originally, but was not sufficient later. >> Sorry, that's my fault; should have stayed with the original test case. >> >> The fix I made to HC4 - overriding the retryRequest() method - >> actually only works for a single sampler, because the "interrupted" >> variable is fetched from the class instance that was used to create >> the override. I introduced the "interrupted" variable, because the fix >> from https://issues.apache.org/jira/browse/HTTPCLIENT-1120 did not >> seem to work. I need to look at that again. This is needed to stop HC4 >> from retrying aborted requests, which I think is a separate issue from >> the deadlocks. >> >> I now think there's a subtle bug in JMeterThread. >> I had expected the interrupt to complete, and then the sample to >> complete, thus the shutdown code would run after the interrupt code. >> However, it looks as though the sample can complete before the >> interrupt method returns. >> This allows the thread to run the shutdown code whilst the interrupt >> is still in progress. >> >> In turn this triggers the deadlock issue in HC4, but could equally >> cause issues with other samplers. >> The code needs to ensure that shutdown cannot happen whilst an >> interrupt is in progress. >> >> I'll commit a fix to SVN ASAP. >> >> Sorry Milamber, but I think we need to cancel this RC vote. >> >> > I also attached a patch that works on my tests. >> > I let you check. >> > >> > Regards >> > Philippe Mouawad >> > >> > On Sun, Sep 25, 2011 at 1:46 PM, sebb <[email protected]> wrote: >> > >> >> On 25 September 2011 12:22, sebb <[email protected]> wrote: >> >> > On 25 September 2011 01:45, Milamber <[email protected]> wrote: >> >> >> >> >> >> >> >> >> Le 25/09/2011 00:08, sebb a ecrit : >> >> >>> On 25 September 2011 01:03, Milamber <[email protected]> wrote: >> >> >>> >> >> >>>> >> >> >>>> Le 23/09/2011 23:38, sebb a ecrit : >> >> >>>> >> >> >>>>> On 23 September 2011 23:17, sebb <[email protected]> wrote: >> >> >>>>> >> >> >>>>> >> >> >>>>>> On 23 September 2011 18:14, sebb <[email protected]> wrote: >> >> >>>>>> >> >> >>>>>> >> >> >>>>>>> On 23 September 2011 17:15, Milamber <[email protected]> >> wrote: >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>>> Hello, >> >> >>>>>>>> >> >> >>>>>>>> It's seems all open bugs with 2.5.1RC1 are closed. >> >> >>>>>>>> I will start the RC2 process tomorrow. >> >> >>>>>>>> >> >> >>>>>>>> >> >> >>>>>>> OK. >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>> Just found a problem with the HC4 retries - they prevent StopTest >> >> from working. >> >> >>>>>> >> >> >>>>>> There's a deadlock in the code that prevents the interrupt from >> >> >>>>>> working when there is a retry. >> >> >>>>>> [At the back of my mind, I thought there was another reason why I >> >> set >> >> >>>>>> the retry count to 0!] >> >> >>>>>> >> >> >>>>>> Shutdown works fine, because it does not try to interrupt the >> >> sample. >> >> >>>>>> >> >> >>>>>> I think there's a work-round I can use in the JMeter code, but I >> >> don't >> >> >>>>>> think we should release the code as is. >> >> >>>>>> >> >> >>>>>> Sorry. >> >> >>>>>> >> >> >>>>>> The Java and HC3.1 samplers work fine; it's only HC4 that has the >> >> problem. >> >> >>>>>> >> >> >>>>>> I'll let you know if there's a solution ASAP. >> >> >>>>>> >> >> >>>>>> >> >> >>>>> URL: http://svn.apache.org/viewvc?rev=1175069&view=rev >> >> >>>>> Log: >> >> >>>>> Temporary hack to work round >> >> >>>>> >> >> >>>>> >> >> >>>> This temporary hack don't always work for me. When I call Stop >> command >> >> >>>> at the beginning of a test (before end of ramp up), I have the same >> >> >>>> deadlock. >> >> >>>> (but sometimes the stop works fine.) >> >> >>>> >> >> >>> Can you do a thread dump when this happens? >> >> >>> >> >> >> >> >> >> >> >> >> Found one Java-level deadlock: >> >> >> ============================= >> >> >> "Thread-205": >> >> >> waiting to lock monitor 0x0000000000d0bf78 (object >> 0x00000000e2c89ba8, >> >> >> a org.apache.http.impl.conn.SingleClientConnManager), >> >> >> which is held by "Thread Group 1-1" >> >> >> "Thread Group 1-1": >> >> >> waiting to lock monitor 0x000000000205e638 (object >> 0x00000000e3425510, >> >> >> a org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter), >> >> >> which is held by "Thread-205" >> >> >> >> >> >> Java stack information for the threads listed above: >> >> >> =================================================== >> >> >> "Thread-205": >> >> >> at >> >> >> >> >> >> org.apache.http.impl.conn.SingleClientConnManager.releaseConnection(SingleClientConnManager.java:258) >> >> >> - waiting to lock <0x00000000e2c89ba8> (a >> >> >> org.apache.http.impl.conn.SingleClientConnManager) >> >> >> at >> >> >> >> >> >> org.apache.http.impl.conn.AbstractClientConnAdapter.abortConnection(AbstractClientConnAdapter.java:323) >> >> >> - locked <0x00000000e3425510> (a >> >> >> org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter) >> >> >> at >> >> >> >> >> >> org.apache.http.client.methods.HttpRequestBase.abort(HttpRequestBase.java:161) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.interrupt(HTTPHC4Impl.java:1087) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.interrupt(HTTPSamplerProxy.java:77) >> >> >> at >> >> >> >> org.apache.jmeter.threads.JMeterThread.interrupt(JMeterThread.java:580) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.engine.StandardJMeterEngine.tellThreadsToStop(StandardJMeterEngine.java:552) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.engine.StandardJMeterEngine.access$300(StandardJMeterEngine.java:58) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.engine.StandardJMeterEngine$StopTest.run(StandardJMeterEngine.java:284) >> >> >> at java.lang.Thread.run(Thread.java:722) >> >> >> "Thread Group 1-1": >> >> >> at >> >> >> >> >> >> org.apache.http.impl.conn.AbstractPooledConnAdapter.detach(AbstractPooledConnAdapter.java:106) >> >> >> - waiting to lock <0x00000000e3425510> (a >> >> >> org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter) >> >> >> at >> >> >> >> >> >> org.apache.http.impl.conn.SingleClientConnManager.shutdown(SingleClientConnManager.java:342) >> >> >> - locked <0x00000000e2c89ba8> (a >> >> >> org.apache.http.impl.conn.SingleClientConnManager) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.closeThreadLocalConnections(HTTPHC4Impl.java:1072) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.threadFinished(HTTPHC4Impl.java:1061) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.threadFinished(HTTPSamplerProxy.java:71) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:553) >> >> >> at >> >> >> >> org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:986) >> >> >> at >> >> org.apache.jorphan.collections.HashTree.traverse(HashTree.java:969) >> >> >> at >> >> >> >> >> >> org.apache.jmeter.threads.JMeterThread.threadFinished(JMeterThread.java:528) >> >> >> at >> org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:308) >> >> >> at java.lang.Thread.run(Thread.java:722) >> >> >> >> >> >> Found 1 deadlock. >> >> >> >> >> >> >> >> >> >> >> >> Full thread dump in attachment. >> >> > >> >> > Thanks! >> >> > >> >> >> Test case is the SimpleTest.jmx, stop after 6-7 VU. >> >> >> For this thread dump, I wait the end of ramp up for 100 VU, but only >> 99 >> >> >> up. (the 'last' is the TG 1-1) >> >> >> >> >> >> >> >> >>> >> >> >>> >> >> >>>> I thinks we must release the version 2.5.1 to correct the others >> bugs >> >> >>>> already fixed, and add a known problem in documentation for this >> >> deadlook? >> >> >>>> >> >> >>> Yes, that sounds reasonable. It's not clear yet whether this is a >> >> >>> JMeter or HC4 problem; nor is it clear what to do to fix it. >> >> >>> Anyway, it only occurs sometimes, and it only occurs when trying to >> >> >>> stop the test - so the GUI can just be exitted. >> >> >>> >> >> >> >> >> >> Ok, I will prepare the RC2 tomorrow. >> >> >> (Perhaps open a jmeter bugzilla for this bug?) >> >> > >> >> > I've opened https://issues.apache.org/jira/browse/HTTPCLIENT-1127 in >> >> > case the bug is in HC4. >> >> > Looks like that may have a lock ordering problem. >> >> > >> >> > But it may be caused by a bug in the way JMeter processes shutdown; >> >> > I'm a bit surprised that threadFinished occurs during interrupt >> >> > processing. >> >> > I'll raise a JMeter bug and take a further look later today. >> >> >> >> Raised https://issues.apache.org/bugzilla/show_bug.cgi?id=51888 >> >> >> >> I don't suppose you have the corresponding JMeter log file? >> >> I think we need both dump and log to debug the problem. >> >> >> >> If not, never mind, I'll see if I can trigger the fault again. >> >> >> >> > >> >> >> Milamber >> >> >> >> >> >>> >> >> >>>> Milamber >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1120 >> >> >>>>> Note: copying the code from the patch did not seem to work; it >> looks >> >> >>>>> like isAborted() was not set. >> >> >>>>> Hopefully that is fixed in 4.1.3 >> >> >>>>> >> >> >>>>> That seems to have fixed it for me, or at least improved matters. >> >> >>>>> >> >> >>>>> Still needs more testing to see if the deadlock I found - reported >> in >> >> >>>>> https://issues.apache.org/jira/browse/HTTPCLIENT-1127 - can still >> >> >>>>> occur. >> >> >>>>> >> >> >>>>> BTW, I found the deadlock testing against the mirror server. >> >> >>>>> >> >> >>>>> >> --------------------------------------------------------------------- >> >> >>>>> To unsubscribe, e-mail: [email protected] >> >> >>>>> For additional commands, e-mail: [email protected] >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>> >> >> >>>> >> --------------------------------------------------------------------- >> >> >>>> To unsubscribe, e-mail: [email protected] >> >> >>>> For additional commands, e-mail: [email protected] >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>> >> --------------------------------------------------------------------- >> >> >>> To unsubscribe, e-mail: [email protected] >> >> >>> For additional commands, e-mail: [email protected] >> >> >>> >> >> >>> >> >> >>> >> >> >> >> >> >> >> >> > >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> >> >> >> >> > >> > >> > -- >> > Cordialement. >> > Philippe Mouawad. >> > Ubik-Ingénierie >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
