While others digest that (seems more than fine to me except for the hang), will try to replicate in the next couple of days.
Wasn't going to be doing our upgrade for a month or so, but seems worth dipping toe in water to see if this will be an issue for us also. We also use "selective" chunking but _do_ use one connect. Any reason why you have one connect off ? Oh, the request header X-Forwarded-SSL. I thought Jetty looks for X-Forwarded-Proto with value "https" to enable secure session cookies. Oh, and totally irrelevant, why are your session cookies so huge and containing a repeat of the first part. Feel free to ignore this :) On 29 January 2014 00:30, Stefan Magnus Landrø <[email protected]>wrote: > Hi there, > > I've been looking further into the issue, and it seems clear what is going > on. > > This is what happens when running requests through our big ip (not using > one connect): > > jetty 9.1: > > whenever a request contains a connection:keep-alive header (the default > for all browsers I believe, but not for curl), curl hangs until the keep > alive timeout is reached. Also, jetty replies with a connection: keep-alive > header in the response. > > whenever a request does not contain a connection:keep-alive header, curl > does not hang, and jetty doesn't add a connection:keep-alive to the response > > jetty 8: > > it doesn't matter if the request contains a connection:keep-alive header > or not, curl does not hang under any circumstance and jetty will never add > a connection:keep-alive header in the response. > > In addition, it turns out that big ip, removes the > transfer-encoding:chunked header that jetty generated (see local debug > block) from the response (we're using the recommended selective response > chunking mode in the big ip http profile). In my opinion that is wrong, > since it doesn't add the Content-Length header, something which would > require the connection to be closed in order for the client to know when > there is no more content. > > BTW, the servlet is spitting out the request headers as-is. > > > Comments, anyone? > > > ************* > Jetty 9.1 (through big ip) > ************* > > > ➜ tmp curl -i https://whereever.no > HTTP/1.0 200 Connection established > > HTTP/1.1 200 OK > X-MiniProfiler-Ids: ["c2600ed9-5e37-4d86-bfd0-f6330c818961"] > Set-Cookie: > SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098g74r8cd0loxh65ry63fq5mle.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly > Expires: Thu, 01 Jan 1970 00:00:00 GMT > X-Frame-Options: SAMEORIGIN > X-XSS-Protection: 1; mode=block > X-Permitted-Cross-Domain-Policies: master-only > X-Content-Type-Options: nosniff > Content-Security-Policy-Report-Only: default-src 'none'; report-uri > /post/cspreport/ > Cache-Control: no-cache, no-store, no-transform > Content-Type: text/plain > Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/ > Vary: Accept-Encoding > > Header name X-Forwarded-SSL > - Header true > Header name X-Forwarded-For > - Header 139.112.144.209 > Header name User-Agent > - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 > OpenSSL/0.9.8y zlib/1.2.5 > Header name Accept > - Header */* > Header name Host > - Header whereever.no > > Does not hang > > > +++++++++++++++++++ > > curl -i -H "*Connection: Keep-Alive*" https://whereever.no > HTTP/1.0 200 Connection established > > HTTP/1.1 200 OK > X-MiniProfiler-Ids: ["7549c3f2-c396-4b26-802d-40350aa64d4c"] > Set-Cookie: > SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee0981q2s047k9wh6aspppnrzwaxbe.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly > Expires: Thu, 01 Jan 1970 00:00:00 GMT > X-Frame-Options: SAMEORIGIN > X-XSS-Protection: 1; mode=block > X-Permitted-Cross-Domain-Policies: master-only > X-Content-Type-Options: nosniff > Content-Security-Policy-Report-Only: default-src 'none'; report-uri > /post/cspreport/ > Cache-Control: no-cache, no-store, no-transform > Content-Type: text/plain > *Connection: keep-alive* > Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/ > Vary: Accept-Encoding > > Header name X-Forwarded-SSL > - Header true > Header name X-Forwarded-For > - Header 139.112.144.209 > Header name User-Agent > - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 > OpenSSL/0.9.8y zlib/1.2.5 > *Header name Connection* > * - Header keep-alive* > Header name Accept > - Header */* > Header name Host > - Header whereever.no > > * This one hangs* > > > ************* > Jetty 8 (through big ip) > ************* > > ➜ tmp curl -i https://whereever.no > HTTP/1.0 200 Connection established > > HTTP/1.1 200 OK > X-MiniProfiler-Ids: ["edba831e-bee6-46e2-bb05-74ecf4dd211f"] > Set-Cookie: > SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pqsvn1ha8o6ve3zjba0wq42s.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly > Expires: Thu, 01 Jan 1970 00:00:00 GMT > X-Frame-Options: SAMEORIGIN > X-XSS-Protection: 1; mode=block > X-Permitted-Cross-Domain-Policies: master-only > X-Content-Type-Options: nosniff > Content-Security-Policy-Report-Only: default-src 'none'; report-uri > /post/cspreport/ > Cache-Control: no-cache, no-store, no-transform > Content-Type: text/plain > Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/ > Vary: Accept-Encoding > > Header name Host > - Header whereever.no > Header name Accept > - Header */* > Header name X-Forwarded-For > - Header 139.112.144.209 > Header name X-Forwarded-SSL > - Header true > Header name User-Agent > - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 > OpenSSL/0.9.8y zlib/1.2.5 > > Does not hang > > +++++++++++++++++++ > > > tmp curl -i -H *"Connection: Keep-Alive"* https://whereever.no > HTTP/1.0 200 Connection established > > HTTP/1.1 200 OK > X-MiniProfiler-Ids: ["509b4ffb-cd28-4dfc-9f29-6d05571162c5"] > Set-Cookie: > SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pul9pgxgmzmt14loahxebxjab.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly > Expires: Thu, 01 Jan 1970 00:00:00 GMT > X-Frame-Options: SAMEORIGIN > X-XSS-Protection: 1; mode=block > X-Permitted-Cross-Domain-Policies: master-only > X-Content-Type-Options: nosniff > Content-Security-Policy-Report-Only: default-src 'none'; report-uri > /post/cspreport/ > Cache-Control: no-cache, no-store, no-transform > Content-Type: text/plain > Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/ > Vary: Accept-Encoding > > Header name Host > - Header whereever.no > Header name Accept > - Header */* > Header name X-Forwarded-For > - Header 139.112.144.209 > Header name X-Forwarded-SSL > - Header true > *Header name Connection* > * - Header keep-alive* > Header name User-Agent > - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 > OpenSSL/0.9.8y zlib/1.2.5 > > Does not hang > > > > ************* > Local test (not through big ip) > ************* > > this is jetty 8, but it looks similar for jetty 9 with regards to the > cunked transfer-encoding > > $ curl -i -H "Connection: keep-alive" http://localhost:12345/whereever > HTTP/1.1 200 OK > X-MiniProfiler-Ids: ["c9172169-755c-4bc2-a03d-9049e2b77e95"] > Set-Cookie: > SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098r6navchnwslj1xn8vwlkrqoq2.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly > Expires: Thu, 01 Jan 1970 00:00:00 GMT > X-Frame-Options: SAMEORIGIN > X-XSS-Protection: 1; mode=block > X-Permitted-Cross-Domain-Policies: master-only > Vary: Accept-Encoding > X-Content-Type-Options: nosniff > Content-Security-Policy-Report-Only: default-src 'none'; report-uri > /post/cspreport/ > Cache-Control: no-cache, no-store, no-transform > Content-Type: text/plain > *Transfer-Encoding: chunked* > > Header name Host > - Header localhost:12345 > Header name Accept > - Header */* > Header name Connection > - Header keep-alive > Header name User-Agent > - Header curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 > OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 > > > > > > 2014-01-28 Stefan Magnus Landrø <[email protected]> > > Now new infrastructure, so this is definitely triggered by upgrading >> jetty. However, the actual issue might be caused by new behaviour in jetty >> triggering a bug in big ip. >> >> Stefan >> >> >> 2014-01-28 "" <[email protected]> >> >> Stefan, >>> >>> You said this was an 8.x to 9.x upgrade, so if the F5 was existing, it >>> shouldn't be an issue unless there are infrastructure changes at the same >>> time for upgrade. We use F5 and the only time we have seen something >>> similar was where a perimeter f/w was dropping an established connection >>> (due to inactivity) within the keep alive time. When the f/w does that >>> the browser does not know the connection it has is stuffed. >>> >>> tcpdump is the best approach. >>> >>> >>> >>> >>> On 28 January 2014 08:45, Stefan Magnus Landrø >>> <[email protected]>wrote: >>> >>>> Thanks, Simone! I'm afraid I have to agree with you - this probably >>>> won't fix my issue. I'll do some tcpdumping in the next few days - I'll >>>> keep you posted. This might as well turn out being a f5 big ip issue. >>>> >>>> Stefan >>>> >>>> >>>> 2014-01-27 Simone Bordet <[email protected]> >>>> >>>> Hi, >>>>> >>>>> On Fri, Jan 24, 2014 at 9:36 PM, "" <[email protected]> wrote: >>>>> > Good find and a _critical_ bug in my view. >>>>> > >>>>> > A change that came with 9.0 as it was this in 8.x :- >>>>> > >>>>> > _header.put(CONNECTION_KEEP_ALIVE); >>>>> > if (connection!=null) >>>>> > { >>>>> > _header.setPutIndex(_header.putIndex()-2); >>>>> > _header.put((byte)','); >>>>> > >>>>> _header.put(connection.toString().getBytes()); >>>>> > _header.put(CRLF); >>>>> > } >>>>> > >>>>> > which had less logic, shorter lines... Yes, it copied 2 more bytes >>>>> than it >>>>> > needed to but I would prefer less logic/shorter code over saving 2 >>>>> bytes. >>>>> > >>>>> > To fix this, you could _try_ turning off one connect on the f5 if >>>>> you don't >>>>> > need it for other reasons. >>>>> > >>>>> > We are also behind an F5, and are just about to start looking at >>>>> moving from >>>>> > 8.x to 9.x. ... so thanks ... will raise on support. >>>>> >>>>> I filed https://bugs.eclipse.org/bugs/show_bug.cgi?id=426739 and >>>>> fixed it. >>>>> >>>>> Note that this is a corner case that should happen very rarely, only >>>>> when the "Connection" header gets some custom value that is not >>>>> "close" or "upgrade" or "keep-alive". >>>>> >>>>> Therefore, I am dubious that the original problem (curl hanging for >>>>> chunked responses) is fixed by this fix. >>>>> >>>>> Let us know if that is the case, though. >>>>> >>>>> -- >>>>> Simone Bordet >>>>> ---- >>>>> http://cometd.org >>>>> http://webtide.com >>>>> http://intalio.com >>>>> Developer advice, training, services and support >>>>> from the Jetty & CometD experts. >>>>> Intalio, the modern way to build business applications. >>>>> _______________________________________________ >>>>> jetty-users mailing list >>>>> [email protected] >>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>> >>>> >>>> >>>> >>>> -- >>>> BEKK Open >>>> http://open.bekk.no >>>> >>>> TesTcl - a unit test framework for iRules >>>> http://testcl.com >>>> >>>> _______________________________________________ >>>> jetty-users mailing list >>>> [email protected] >>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>>> >>> >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >>> >> >> >> >> > > > -- > BEKK Open > http://open.bekk.no > > TesTcl - a unit test framework for iRules > http://testcl.com > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > >
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
