On Fri, Mar 10, 2017 at 8:08 AM, Tuukka Mustonen <[email protected]>
wrote:
> > > +DELETE also like sebb said.
> > >
> > True but iIt was not concerned by bug.
>
> Maybe I got something wrong, but you did paste DELETE originally as
> non-idemponent in list of non-retryable methods:
>
> Delete not being implementation of HttpEntityEnclosingRequest, it was not
concerned by issue.
Yes I thought wrongly Delete was not idempotent
> > - Idempotent HTTP methods which are by default those that do not
> implement
> > > HttpEntityEnclosingRequest, so not POST, PUT,DELETE, PATCH, GET With
> body
>
> > 1/ To diagnose set those classes in debug mode:
> > org.apache.http.impl.client.DefaultRequestDirector
> >
> > You should see:
> > Retrying connect to
> > Retrying request to
> >
> > Log level can be set in log4j2.xml
>
> Ok, with latest nightly build, retrying works just fine, I got:
>
> 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: I/O exception
> (org.apache.http.NoHttpResponseException) caught when processing request
> to
> {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80: The target server
> failed to respond
> 2017-03-10 08:50:57,856 INFO o.a.j.p.h.s.HTTPHC4Impl$3: Retrying request to
> {}->http://client-api.perf.ew1.cosmos-ci.fsapi.com:80
>
> However, with exact same testplan/configuration/scenario, retrying just
> does not work with HC4 on JMeter 3.1. Unfortunately, I wasn't able to
> configure logging on JMeter 3.1 so that it would print similar lines. I
> tried adding to user.properties:
>
> log_level.org.apache.http.impl.client.DefaultRequestDirector=DEBUG
>
> But that didn't do anything. Also tried tuning bin/log4j.conf. Maybe that
> class does not exist in JMeter 3.1?
>
> Anyway, I can confirm that retying with HC4 does not work on JMeter 3.1 but
> does work on latest nightly.
You confirm it's this night nightly or yesterday's one ?
Did you use the property:
httpclient4.request_sent_retry_enabled=true
> Not for me at least (I don't believe I have
> anything special in my JMeter 3.1 installation, just same plugins that I
> downloaded into nightly build also).
>
> > I have added a property that you can set and try in user.properties:
> > httpclient4.request_sent_retry_enabled=true
> > Please give your feedback rapidly, so that we decide what to do if it
> has an
> effect.
>
> Unfortunately, I don't have a test case with non-idempotent method at hand
> just now. I can give it a try at the start of next week.
>
If you used httpclient4.request_sent_retry_enabled=true and it works while
it doesn't with httpclient4.request_sent_retry_enabled=false
It's a good reason to add it.
BUt if it already works with :
httpclient4.request_sent_retry_enabled=false
Then it might not be useful. Can you give feedback on this ?
And we'll wait a week for your test on non idempotent methods.
>
> > But setting httpclient4.validate_after_inactivity=0 means you disable
> the
> > validation which is not a good idea as you'll end up having broken
> > connection.
>
> Maybe httpclient4.validate_after_inactivity=1 then?
Yes but it's not a good setting, in this case better use stalecheck.
> I assume that's pretty
> much the same as the old stale check?
>
>
> Tuukka
>
>
> On Thu, Mar 9, 2017 at 10:38 PM, Philippe Mouawad <
> [email protected]> wrote:
>
> > On Thu, Mar 9, 2017 at 5:28 PM, Tuukka Mustonen <
> [email protected]
> > >
> > wrote:
> >
> > > Sorry for the late reply,
> > >
> > > About customizing retry method whitelist:
> > >
> > > > You case is a bit soecific no ?
> > >
> > > True, it's very undesirable scenario, but that's what you get with AWS
> > ALB
> > > for now - I believe there are lots and lots of other users that will
> face
> > > the same problem.
> > >
> > > I have added a property that you can set and try in user.properties:
> > httpclient4.request_sent_retry_enabled=true
> >
> > Please give your feedback rapidly, so that we decide what to do if it has
> > an effect.
> >
> > Having said that, I am in discussion with AWS about if they will fix it
> and
> > > with what timeline - current behavior is complete nonsense...
> > >
> > > > In this case, the bug is larger than what I thought as for now we
> > > consider
> > > > PUT as non idempotent.
> > >
> > > +DELETE also like sebb said.
> > >
> > True but iIt was not concerned by bug.
> >
> > >
> > > > 1/ To diagnose set those classes in debug mode:
> > > > org.apache.http.impl.client.DefaultRequestDirector
> > > >
> > > > You should see:
> > > > Retrying connect to
> > > > Retrying request to
> > > >
> > > > Log level can be set in log4j2.xml
> > >
> > > Thanks! I'll try to find time to check that tomorrow...
> > >
> > > > 2/ I confirm stalecheck is still available.
> > >
> > > Can you clarify what is the difference to having
> > > httpclient4.validate_after_inactivity=0?
> > >
> >
> > It was an optimisation added by HC4 to replace stalecheck which was very
> > costly.
> > But setting httpclient4.validate_after_inactivity=0 means you disable
> the
> > validation which is not a good idea as you'll end up having broken
> > connection.
> > This value should be set to a value lower than the keepalive set on
> server
> > side.
> >
> >
> > > Tuukka
> > >
> > >
> > > On Thu, Mar 9, 2017 at 2:23 PM, Philippe Mouawad <
> > > [email protected]
> > > > wrote:
> > >
> > > > Thanks for link sebb.
> > > >
> > > > In this case, the bug is larger than what I thought as for now we
> > > consider
> > > > PUT as non idempotent.
> > > >
> > > > I'll commit a new fix.
> > > > Regards
> > > >
> > > > On Thu, Mar 9, 2017 at 12:20 PM, sebb <[email protected]> wrote:
> > > >
> > > > > On 9 March 2017 at 06:42, Philippe Mouawad <
> > [email protected]
> > > >
> > > > > wrote:
> > > > > > On Thursday, March 9, 2017, Tuukka Mustonen <
> > > [email protected]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Philippe and thanks for speedy actions!
> > > > > >>
> > > > > >> 1) Retrying only idempotent requests makes sense, as usual. But
> > this
> > > > > (retry
> > > > > >> on idempotent) seems to be only documented on wiki page (
> > > > > >> https://wiki.apache.org/jmeter/JMeterSocketClosed). You should
> > add
> > > it
> > > > > also
> > > > > >> to http://jmeter.apache.org/usermanual/component_reference.html
> .
> > > > > >>
> > > > > >> 2) In this case, the requests are GETs (and without body) so
> they
> > > > > should be
> > > > > >> retried.
> > > > > >
> > > > > > Yes.
> > > > > > I debugged it and I confirm they are unless you face the
> > exceptions I
> > > > > > mentionned.
> > > > > >
> > > > > >>
> > > > > >> So the bug you fixed shouldn't affect this scenario.
> > > > > >
> > > > > > Yes
> > > > > >
> > > > > >>
> > > > > >> There must be
> > > > > >> something else - any pointers to what logging module/level I
> > should
> > > > > enable
> > > > > >> to inspect the (failing) retry logic?
> > > > > >
> > > > > > As you can see there is no logging in this HC4 class
> > > > > > Try the calling class. I' ll provide its name later
> > > > > >
> > > > > >>
> > > > > >> 3) AFAIK PUT is idempotent - shouldn't you retry also that?
> > > > > >
> > > > > > No it's not IMU, it changes state of server.
> > > > >
> > > > > PUT *is* idempotent.
> > > > >
> > > > > It may change the state of the server the first time it is sent,
> but
> > > > > subsequent PUTs should not change the state further.
> > > > > So it can be repeated as required.
> > > > >
> > > > > Similarly for DELETE
> > > > >
> > > > > https://tools.ietf.org/html/rfc2616#section-9.1.2
> > > > >
> > > > > >>
> > > > > >> 4) In this case, once I add tests for POST/PATCH/DELETE etc. I
> > also
> > > > > want to
> > > > > >> retry these non-idempotent requests (to tolerate ALBs ugly
> > > behavior).
> > > > > These
> > > > > >> are "just" performance tests so I don't care if I'm creating
> > > duplicate
> > > > > >> data. Is there way to specify custom method whitelist to retry
> any
> > > > HTTP
> > > > > >> method? I see requestSentRetryEnabled in HC4 source code - can I
> > > > enable
> > > > > >> that somehow?
> > > > > >
> > > > > > Not for now.
> > > > > > You case is a bit soecific no ?
> > > > > >
> > > > > >>
> > > > > >> 5) *Related question:
> > > > > >> is http.connection.stalecheck$Boolean=false (in hc.parameters
> > file)
> > > > > valid
> > > > > >> anymore on JMeter 3.x with improved retry/stale logic
> > > > > >> (and httpclient4.validate_after_inactivity)? The line is still
> > > there
> > > > in
> > > > > >> bundled hc.parameters file...*
> > > > > >
> > > > > > i'll double check
> > > > > >
> > > > > >>
> > > > > >> Tuukka
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Mar 9, 2017 at 12:25 AM, Philippe Mouawad <
> > > > > >> [email protected] <javascript:;>> wrote:
> > > > > >>
> > > > > >> > Hello,
> > > > > >> > The issue with Get with body should be fixed now:
> > > > > >> > - https://bz.apache.org/bugzilla/show_bug.cgi?id=60837
> > > > > >> >
> > > > > >> > Regards
> > > > > >> > Philippe
> > > > > >> >
> > > > > >> > On Wed, Mar 8, 2017 at 8:53 PM, Philippe Mouawad <
> > > > > >> > [email protected] <javascript:;>
> > > > > >> > > wrote:
> > > > > >> >
> > > > > >> > > Hello Tuukka,
> > > > > >> > >
> > > > > >> > > In my recent tests I didn't face any issue with
> > > > > httpclient4.retrycount.
> > > > > >> > > For me it works but be aware that JMeter (HC4) does not
> retry
> > > all
> > > > > >> > > requests, it only retries those it is allowed to :
> > > > > >> > > - Idempotent HTTP methods which are by default those that do
> > not
> > > > > >> > implement
> > > > > >> > > HttpEntityEnclosingRequest, so not POST, PUT,DELETE, PATCH,
> > GET
> > > > With
> > > > > >> body
> > > > > >> > > (<= That might be a bug thinking more about it)
> > > > > >> > > - Non retriable exceptions (InterruptedIOException.class,
> > > > > >> > > UnknownHostException.class, ConnectException.class,
> > > > > >> SSLException.class)
> > > > > >> > > - + Some other reasons
> > > > > >> > >
> > > > > >> > > See:
> > > > > >> > > https://github.com/apache/httpclient/blob/4.5.x/
> > > > > >> > > httpclient/src/main/java/org/apache/http/impl/client/
> > > > > >> > > DefaultHttpRequestRetryHandler.java#L129
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Regards
> > > > > >> > > Philippe
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, Mar 8, 2017 at 2:14 PM, Tuukka Mustonen <
> > > > > >> > [email protected] <javascript:;>
> > > > > >> > > > wrote:
> > > > > >> > >
> > > > > >> > >> My problem is that AWS Application Load Balancer (ALB)
> > > terminates
> > > > > all
> > > > > >> > >> existing connections during configuration changes
> (including
> > > > > >> > >> auto-scaling).
> > > > > >> > >> As my perf tests trigger auto-scaling, I want to retry
> failed
> > > > > requests
> > > > > >> > >> that
> > > > > >> > >> ALB connection termination causes.
> > > > > >> > >>
> > > > > >> > >> This results in a bunch of NoHttpResponseException whenever
> > > > scaling
> > > > > >> > occurs
> > > > > >> > >> (=when ALB terminates existing connections).
> > > > > >> > >>
> > > > > >> > >> (And yeah, this load balancer behavior is weird and ugly
> but
> > > it's
> > > > > what
> > > > > >> > >> they
> > > > > >> > >> confirmed).
> > > > > >> > >>
> > > > > >> > >> Using HttpClient 4, In user.properties I have set:
> > > > > >> > >>
> > > > > >> > >> httpclient4.retrycount=1
> > > > > >> > >>
> > > > > >> > >> But that does nothing. I even tried:
> > > > > >> > >>
> > > > > >> > >> httpclient4.retrycount=100000000
> > > > > >> > >>
> > > > > >> > >> But zero effect.
> > > > > >> > >>
> > > > > >> > >> Switching to HttpClient 3.1 reproduces the problem.
> However,
> > > with
> > > > > >> > >> HttpClient 3 and:
> > > > > >> > >>
> > > > > >> > >> httpclient3.retrycount=1
> > > > > >> > >>
> > > > > >> > >> The problem vanishes so I assume retrying then works.
> > > > > >> > >>
> > > > > >> > >> I couldn't find where retry-attempts are logged so I have
> no
> > > > > "proof"
> > > > > >> > that
> > > > > >> > >> HttpClient 4 wouldn't actually retry, but a) retrycount
> makes
> > > > > >> difference
> > > > > >> > >> on
> > > > > >> > >> HttpClient 3.1 but not on 4 b) I would assume my whole
> > process
> > > > > should
> > > > > >> > >> crash
> > > > > >> > >> with retrycount=100000000 if it was really applied.
> > > > > >> > >>
> > > > > >> > >> Related question: is http.connection.stalecheck$
> > Boolean=false
> > > > (in
> > > > > >> > >> hc.parameters file) valid anymore on JMeter 3.x with
> improved
> > > > > >> > retry/stale
> > > > > >> > >> logic (and httpclient4.validate_after_inactivity)? The
> line
> > is
> > > > > still
> > > > > >> > >> there
> > > > > >> > >> in bundled hc.parameters file...
> > > > > >> > >>
> > > > > >> > >> Tested on JMeter 3.1.
> > > > > >> > >>
> > > > > >> > >> Any open (or already fixed) tickets about this? Couldn't
> find
> > > > > any...
> > > > > >> > >>
> > > > > >> > >> FYI: to describe better what happens during ALB connection
> > > > > >> termination:
> > > > > >> > >>
> > > > > >> > >> ...Successful communication with keep-alive...
> > > > > >> > >> - ALB responds to a request, and sends Connection:
> keep-alive
> > > so
> > > > > >> JMeter
> > > > > >> > >> leaves the connection open
> > > > > >> > >> - JMeter sends a new request
> > > > > >> > >> - ALB may might or might not ack it (not sure if there was
> > > > pattern)
> > > > > >> > >> - ALB closes connection on TCP level (FIN)
> > > > > >> > >> - Connection gets closed and so sent request failed and
> needs
> > > to
> > > > be
> > > > > >> > >> retried
> > > > > >> > >>
> > > > > >> > >> I think this generates NoHttpResponseException in JMeter.
> > > > > >> > >>
> > > > > >> > >> Tuukka
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > --
> > > > > >> > > Cordialement.
> > > > > >> > > Philippe Mouawad.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Cordialement.
> > > > > >> > Philippe Mouawad.
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Cordialement.
> > > > > > Philippe Mouawad.
> > > > >
> > > > > ------------------------------------------------------------
> > ---------
> > > > > To unsubscribe, e-mail: [email protected]
> > > > > For additional commands, e-mail: [email protected]
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Cordialement.
> > > > Philippe Mouawad.
> > > >
> > >
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>
--
Cordialement.
Philippe Mouawad.
