Hi Tuukka, Thanks for your feedback. This feature has now been added cleanly and documented to trunk within bug: https://bz.apache.org/bugzilla/show_bug.cgi?id=60888
# true if it's OK to retry requests that have been sent # This will retry Idempotent and non Idempotent requests # This should usually be false, but it can be useful # when testing against some Load Balancers like Amazon ELB #httpclient4.request_sent_retry_enabled=false See: - https://github.com/apache/httpclient/blob/4.5.x/httpclient/src/main/java/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.java#L160 As you can guess, settings this to true means that a request could be received twice as we retry requests already sent. So this means that even Non Idempotent requests are retried. Regards Philippe On Wed, Mar 15, 2017 at 4:45 PM, Tuukka Mustonen <[email protected]> wrote: > Ok, I added test that uses POST/PATCH/DELETE/GET methods. > > With #6020 build of JMeter > and httpclient4.request_sent_retry_enabled=false, there were some retries, > but mostly not. I think it retried GET (and maybe DELETE) methods but not > anything else. > > With httpclient4.request_sent_retry_enabled=true, all failed requests were > retried and this resulted in zero errors. > > So yeah, this new option is something I would need/want with ALB (until > they fix the issue, I still haven't received reply if AWS considers it a > bug or a feature). > > Though I am still not sure what it actually does: > > *"However, can you crystalize what that new property is supposed to do? As > I wrote above I thought it was to retry also non-idempotent methods, but I > guess it's actually supposed to do something else?"* > > Tuukka > > > On Fri, Mar 10, 2017 at 12:08 PM, Tuukka Mustonen < > [email protected] > > wrote: > > > > You confirm it's this night nightly or yesterday's one ? > > > Did you use the property: > > > httpclient4.request_sent_retry_enabled=true > > > > It was this build: https://builds.apache.org/job/JMeter-trunk/6020/ > > > > No, I didn't enable that property - I thought that property was to also > > retry non-idempotent methods. > > > > > If you used httpclient4.request_sent_retry_enabled=true and it works > > while > > > it doesn't with httpclient4.request_sent_retry_enabled=false > > > It's a good reason to add it. > > > BUt if it already works with : > > > httpclient4.request_sent_retry_enabled=false > > > > > > Then it might not be useful. Can you give feedback on this ? > > > > Yeah, I didn't declare it at so I guess it's not needed then... > > > > However, can you crystalize what that new property is supposed to do? As > I > > wrote above I thought it was to retry also non-idempotent methods, but I > > guess it's actually supposed to do something else? > > > > > And we'll wait a week for your test on non idempotent methods. > > > > Ok. > > > > > Yes but it's not a good setting, in this case better use stalecheck. > > > > Ok. > > > > Tuukka > > > > > > On Fri, Mar 10, 2017 at 11:27 AM, Philippe Mouawad < > > [email protected]> wrote: > > > >> On Fri, Mar 10, 2017 at 8:08 AM, Tuukka Mustonen < > >> [email protected]> > >> wrote: > >> > >> > > > +DELETE also like sebb said. > >> > > > > >> > > True but iIt was not concerned by bug. > >> > > >> > Maybe I got something wrong, but you did paste DELETE originally as > >> > non-idemponent in list of non-retryable methods: > >> > > >> > Delete not being implementation of HttpEntityEnclosingRequest, it was > >> not > >> concerned by issue. > >> Yes I thought wrongly Delete was not idempotent > >> > >> > > - Idempotent HTTP methods which are by default those that do not > >> > implement > >> > > > HttpEntityEnclosingRequest, so not POST, PUT,DELETE, PATCH, GET > With > >> > body > >> > > >> > > 1/ To diagnose set those classes in debug mode: > >> > > org.apache.http.impl.client.DefaultRequestDirector > >> > > > >> > > You should see: > >> > > Retrying connect to > >> > > Retrying request to > >> > > > >> > > Log level can be set in log4j2.xml > >> > > >> > Ok, with latest nightly build, retrying works just fine, I got: > >> > > >> > 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: I/O exception > >> > (org.apache.http.NoHttpResponseException) caught when processing > >> request > >> > to > >> > {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80: The target > >> server > >> > failed to respond > >> > 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: Retrying > >> request to > >> > {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80 > >> > > >> > However, with exact same testplan/configuration/scenario, retrying > just > >> > does not work with HC4 on JMeter 3.1. Unfortunately, I wasn't able to > >> > configure logging on JMeter 3.1 so that it would print similar lines. > I > >> > tried adding to user.properties: > >> > > >> > log_level.org.apache.http.impl.client.DefaultRequestDirector=DEBUG > >> > > >> > But that didn't do anything. Also tried tuning bin/log4j.conf. Maybe > >> that > >> > class does not exist in JMeter 3.1? > >> > > >> > Anyway, I can confirm that retying with HC4 does not work on JMeter > 3.1 > >> but > >> > does work on latest nightly. > >> > >> > >> You confirm it's this night nightly or yesterday's one ? > >> Did you use the property: > >> httpclient4.request_sent_retry_enabled=true > >> > >> > >> > Not for me at least (I don't believe I have > >> > anything special in my JMeter 3.1 installation, just same plugins > that I > >> > downloaded into nightly build also). > >> > > >> > > I have added a property that you can set and try in user.properties: > >> > > httpclient4.request_sent_retry_enabled=true > >> > > Please give your feedback rapidly, so that we decide what to do if > it > >> > has an > >> > effect. > >> > > >> > Unfortunately, I don't have a test case with non-idempotent method at > >> hand > >> > just now. I can give it a try at the start of next week. > >> > > >> > >> If you used httpclient4.request_sent_retry_enabled=true and it works > >> while > >> it doesn't with httpclient4.request_sent_retry_enabled=false > >> It's a good reason to add it. > >> BUt if it already works with : > >> httpclient4.request_sent_retry_enabled=false > >> > >> Then it might not be useful. Can you give feedback on this ? > >> And we'll wait a week for your test on non idempotent methods. > >> > >> > > >> > > But setting httpclient4.validate_after_inactivity=0 means you > >> disable > >> > the > >> > > validation which is not a good idea as you'll end up having broken > >> > > connection. > >> > > >> > Maybe httpclient4.validate_after_inactivity=1 then? > >> > >> > >> Yes but it's not a good setting, in this case better use stalecheck. > >> > >> > >> > I assume that's pretty > >> > much the same as the old stale check? > >> > > >> > >> > >> > >> > > >> > Tuukka > >> > > >> > > >> > On Thu, Mar 9, 2017 at 10:38 PM, Philippe Mouawad < > >> > [email protected]> wrote: > >> > > >> > > On Thu, Mar 9, 2017 at 5:28 PM, Tuukka Mustonen < > >> > [email protected] > >> > > > > >> > > wrote: > >> > > > >> > > > Sorry for the late reply, > >> > > > > >> > > > About customizing retry method whitelist: > >> > > > > >> > > > > You case is a bit soecific no ? > >> > > > > >> > > > True, it's very undesirable scenario, but that's what you get with > >> AWS > >> > > ALB > >> > > > for now - I believe there are lots and lots of other users that > will > >> > face > >> > > > the same problem. > >> > > > > >> > > > I have added a property that you can set and try in > user.properties: > >> > > httpclient4.request_sent_retry_enabled=true > >> > > > >> > > Please give your feedback rapidly, so that we decide what to do if > it > >> has > >> > > an effect. > >> > > > >> > > Having said that, I am in discussion with AWS about if they will fix > >> it > >> > and > >> > > > with what timeline - current behavior is complete nonsense... > >> > > > > >> > > > > In this case, the bug is larger than what I thought as for now > we > >> > > > consider > >> > > > > PUT as non idempotent. > >> > > > > >> > > > +DELETE also like sebb said. > >> > > > > >> > > True but iIt was not concerned by bug. > >> > > > >> > > > > >> > > > > 1/ To diagnose set those classes in debug mode: > >> > > > > org.apache.http.impl.client.DefaultRequestDirector > >> > > > > > >> > > > > You should see: > >> > > > > Retrying connect to > >> > > > > Retrying request to > >> > > > > > >> > > > > Log level can be set in log4j2.xml > >> > > > > >> > > > Thanks! I'll try to find time to check that tomorrow... > >> > > > > >> > > > > 2/ I confirm stalecheck is still available. > >> > > > > >> > > > Can you clarify what is the difference to having > >> > > > httpclient4.validate_after_inactivity=0? > >> > > > > >> > > > >> > > It was an optimisation added by HC4 to replace stalecheck which was > >> very > >> > > costly. > >> > > But setting httpclient4.validate_after_inactivity=0 means you > >> disable > >> > the > >> > > validation which is not a good idea as you'll end up having broken > >> > > connection. > >> > > This value should be set to a value lower than the keepalive set on > >> > server > >> > > side. > >> > > > >> > > > >> > > > Tuukka > >> > > > > >> > > > > >> > > > On Thu, Mar 9, 2017 at 2:23 PM, Philippe Mouawad < > >> > > > [email protected] > >> > > > > wrote: > >> > > > > >> > > > > Thanks for link sebb. > >> > > > > > >> > > > > In this case, the bug is larger than what I thought as for now > we > >> > > > consider > >> > > > > PUT as non idempotent. > >> > > > > > >> > > > > I'll commit a new fix. > >> > > > > Regards > >> > > > > > >> > > > > On Thu, Mar 9, 2017 at 12:20 PM, sebb <[email protected]> wrote: > >> > > > > > >> > > > > > On 9 March 2017 at 06:42, Philippe Mouawad < > >> > > [email protected] > >> > > > > > >> > > > > > wrote: > >> > > > > > > On Thursday, March 9, 2017, Tuukka Mustonen < > >> > > > [email protected] > >> > > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > >> Hi Philippe and thanks for speedy actions! > >> > > > > > >> > >> > > > > > >> 1) Retrying only idempotent requests makes sense, as usual. > >> But > >> > > this > >> > > > > > (retry > >> > > > > > >> on idempotent) seems to be only documented on wiki page ( > >> > > > > > >> https://wiki.apache.org/jmeter/JMeterSocketClosed). You > >> should > >> > > add > >> > > > it > >> > > > > > also > >> > > > > > >> to http://jmeter.apache.org/userm > >> anual/component_reference.html > >> > . > >> > > > > > >> > >> > > > > > >> 2) In this case, the requests are GETs (and without body) > so > >> > they > >> > > > > > should be > >> > > > > > >> retried. > >> > > > > > > > >> > > > > > > Yes. > >> > > > > > > I debugged it and I confirm they are unless you face the > >> > > exceptions I > >> > > > > > > mentionned. > >> > > > > > > > >> > > > > > >> > >> > > > > > >> So the bug you fixed shouldn't affect this scenario. > >> > > > > > > > >> > > > > > > Yes > >> > > > > > > > >> > > > > > >> > >> > > > > > >> There must be > >> > > > > > >> something else - any pointers to what logging module/level > I > >> > > should > >> > > > > > enable > >> > > > > > >> to inspect the (failing) retry logic? > >> > > > > > > > >> > > > > > > As you can see there is no logging in this HC4 class > >> > > > > > > Try the calling class. I' ll provide its name later > >> > > > > > > > >> > > > > > >> > >> > > > > > >> 3) AFAIK PUT is idempotent - shouldn't you retry also that? > >> > > > > > > > >> > > > > > > No it's not IMU, it changes state of server. > >> > > > > > > >> > > > > > PUT *is* idempotent. > >> > > > > > > >> > > > > > It may change the state of the server the first time it is > sent, > >> > but > >> > > > > > subsequent PUTs should not change the state further. > >> > > > > > So it can be repeated as required. > >> > > > > > > >> > > > > > Similarly for DELETE > >> > > > > > > >> > > > > > https://tools.ietf.org/html/rfc2616#section-9.1.2 > >> > > > > > > >> > > > > > >> > >> > > > > > >> 4) In this case, once I add tests for POST/PATCH/DELETE > etc. > >> I > >> > > also > >> > > > > > want to > >> > > > > > >> retry these non-idempotent requests (to tolerate ALBs ugly > >> > > > behavior). > >> > > > > > These > >> > > > > > >> are "just" performance tests so I don't care if I'm > creating > >> > > > duplicate > >> > > > > > >> data. Is there way to specify custom method whitelist to > >> retry > >> > any > >> > > > > HTTP > >> > > > > > >> method? I see requestSentRetryEnabled in HC4 source code - > >> can I > >> > > > > enable > >> > > > > > >> that somehow? > >> > > > > > > > >> > > > > > > Not for now. > >> > > > > > > You case is a bit soecific no ? > >> > > > > > > > >> > > > > > >> > >> > > > > > >> 5) *Related question: > >> > > > > > >> is http.connection.stalecheck$Boolean=false (in > >> hc.parameters > >> > > file) > >> > > > > > valid > >> > > > > > >> anymore on JMeter 3.x with improved retry/stale logic > >> > > > > > >> (and httpclient4.validate_after_inactivity)? The line is > >> still > >> > > > there > >> > > > > in > >> > > > > > >> bundled hc.parameters file...* > >> > > > > > > > >> > > > > > > i'll double check > >> > > > > > > > >> > > > > > >> > >> > > > > > >> Tuukka > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> On Thu, Mar 9, 2017 at 12:25 AM, Philippe Mouawad < > >> > > > > > >> [email protected] <javascript:;>> wrote: > >> > > > > > >> > >> > > > > > >> > Hello, > >> > > > > > >> > The issue with Get with body should be fixed now: > >> > > > > > >> > - https://bz.apache.org/bugzilla/show_bug.cgi?id=60837 > >> > > > > > >> > > >> > > > > > >> > Regards > >> > > > > > >> > Philippe > >> > > > > > >> > > >> > > > > > >> > On Wed, Mar 8, 2017 at 8:53 PM, Philippe Mouawad < > >> > > > > > >> > [email protected] <javascript:;> > >> > > > > > >> > > wrote: > >> > > > > > >> > > >> > > > > > >> > > Hello Tuukka, > >> > > > > > >> > > > >> > > > > > >> > > In my recent tests I didn't face any issue with > >> > > > > > httpclient4.retrycount. > >> > > > > > >> > > For me it works but be aware that JMeter (HC4) does > not > >> > retry > >> > > > all > >> > > > > > >> > > requests, it only retries those it is allowed to : > >> > > > > > >> > > - Idempotent HTTP methods which are by default those > >> that do > >> > > not > >> > > > > > >> > implement > >> > > > > > >> > > HttpEntityEnclosingRequest, so not POST, PUT,DELETE, > >> PATCH, > >> > > GET > >> > > > > With > >> > > > > > >> body > >> > > > > > >> > > (<= That might be a bug thinking more about it) > >> > > > > > >> > > - Non retriable exceptions > (InterruptedIOException.class, > >> > > > > > >> > > UnknownHostException.class, ConnectException.class, > >> > > > > > >> SSLException.class) > >> > > > > > >> > > - + Some other reasons > >> > > > > > >> > > > >> > > > > > >> > > See: > >> > > > > > >> > > https://github.com/apache/httpclient/blob/4.5.x/ > >> > > > > > >> > > httpclient/src/main/java/org/apache/http/impl/client/ > >> > > > > > >> > > DefaultHttpRequestRetryHandler.java#L129 > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > Regards > >> > > > > > >> > > Philippe > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > On Wed, Mar 8, 2017 at 2:14 PM, Tuukka Mustonen < > >> > > > > > >> > [email protected] <javascript:;> > >> > > > > > >> > > > wrote: > >> > > > > > >> > > > >> > > > > > >> > >> My problem is that AWS Application Load Balancer (ALB) > >> > > > terminates > >> > > > > > all > >> > > > > > >> > >> existing connections during configuration changes > >> > (including > >> > > > > > >> > >> auto-scaling). > >> > > > > > >> > >> As my perf tests trigger auto-scaling, I want to retry > >> > failed > >> > > > > > requests > >> > > > > > >> > >> that > >> > > > > > >> > >> ALB connection termination causes. > >> > > > > > >> > >> > >> > > > > > >> > >> This results in a bunch of NoHttpResponseException > >> whenever > >> > > > > scaling > >> > > > > > >> > occurs > >> > > > > > >> > >> (=when ALB terminates existing connections). > >> > > > > > >> > >> > >> > > > > > >> > >> (And yeah, this load balancer behavior is weird and > ugly > >> > but > >> > > > it's > >> > > > > > what > >> > > > > > >> > >> they > >> > > > > > >> > >> confirmed). > >> > > > > > >> > >> > >> > > > > > >> > >> Using HttpClient 4, In user.properties I have set: > >> > > > > > >> > >> > >> > > > > > >> > >> httpclient4.retrycount=1 > >> > > > > > >> > >> > >> > > > > > >> > >> But that does nothing. I even tried: > >> > > > > > >> > >> > >> > > > > > >> > >> httpclient4.retrycount=100000000 > >> > > > > > >> > >> > >> > > > > > >> > >> But zero effect. > >> > > > > > >> > >> > >> > > > > > >> > >> Switching to HttpClient 3.1 reproduces the problem. > >> > However, > >> > > > with > >> > > > > > >> > >> HttpClient 3 and: > >> > > > > > >> > >> > >> > > > > > >> > >> httpclient3.retrycount=1 > >> > > > > > >> > >> > >> > > > > > >> > >> The problem vanishes so I assume retrying then works. > >> > > > > > >> > >> > >> > > > > > >> > >> I couldn't find where retry-attempts are logged so I > >> have > >> > no > >> > > > > > "proof" > >> > > > > > >> > that > >> > > > > > >> > >> HttpClient 4 wouldn't actually retry, but a) > retrycount > >> > makes > >> > > > > > >> difference > >> > > > > > >> > >> on > >> > > > > > >> > >> HttpClient 3.1 but not on 4 b) I would assume my whole > >> > > process > >> > > > > > should > >> > > > > > >> > >> crash > >> > > > > > >> > >> with retrycount=100000000 if it was really applied. > >> > > > > > >> > >> > >> > > > > > >> > >> Related question: is http.connection.stalecheck$ > >> > > Boolean=false > >> > > > > (in > >> > > > > > >> > >> hc.parameters file) valid anymore on JMeter 3.x with > >> > improved > >> > > > > > >> > retry/stale > >> > > > > > >> > >> logic (and httpclient4.validate_after_inactivity)? > The > >> > line > >> > > is > >> > > > > > still > >> > > > > > >> > >> there > >> > > > > > >> > >> in bundled hc.parameters file... > >> > > > > > >> > >> > >> > > > > > >> > >> Tested on JMeter 3.1. > >> > > > > > >> > >> > >> > > > > > >> > >> Any open (or already fixed) tickets about this? > Couldn't > >> > find > >> > > > > > any... > >> > > > > > >> > >> > >> > > > > > >> > >> FYI: to describe better what happens during ALB > >> connection > >> > > > > > >> termination: > >> > > > > > >> > >> > >> > > > > > >> > >> ...Successful communication with keep-alive... > >> > > > > > >> > >> - ALB responds to a request, and sends Connection: > >> > keep-alive > >> > > > so > >> > > > > > >> JMeter > >> > > > > > >> > >> leaves the connection open > >> > > > > > >> > >> - JMeter sends a new request > >> > > > > > >> > >> - ALB may might or might not ack it (not sure if there > >> was > >> > > > > pattern) > >> > > > > > >> > >> - ALB closes connection on TCP level (FIN) > >> > > > > > >> > >> - Connection gets closed and so sent request failed > and > >> > needs > >> > > > to > >> > > > > be > >> > > > > > >> > >> retried > >> > > > > > >> > >> > >> > > > > > >> > >> I think this generates NoHttpResponseException in > >> JMeter. > >> > > > > > >> > >> > >> > > > > > >> > >> Tuukka > >> > > > > > >> > >> > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > -- > >> > > > > > >> > > Cordialement. > >> > > > > > >> > > Philippe Mouawad. > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > -- > >> > > > > > >> > Cordialement. > >> > > > > > >> > Philippe Mouawad. > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > > >> > > > > > > > >> > > > > > > -- > >> > > > > > > Cordialement. > >> > > > > > > Philippe Mouawad. > >> > > > > > > >> > > > > > ------------------------------------------------------------ > >> > > --------- > >> > > > > > To unsubscribe, e-mail: [email protected] > >> > > > > > For additional commands, e-mail: [email protected] > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > Cordialement. > >> > > > > Philippe Mouawad. > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Cordialement. > >> > > Philippe Mouawad. > >> > > > >> > > >> > >> > >> > >> -- > >> Cordialement. > >> Philippe Mouawad. > >> > > > > > -- Cordialement. Philippe Mouawad.
