Hi there, I've been looking further into the issue, and it seems clear what is going on.
This is what happens when running requests through our big ip (not using one connect): jetty 9.1: whenever a request contains a connection:keep-alive header (the default for all browsers I believe, but not for curl), curl hangs until the keep alive timeout is reached. Also, jetty replies with a connection: keep-alive header in the response. whenever a request does not contain a connection:keep-alive header, curl does not hang, and jetty doesn't add a connection:keep-alive to the response jetty 8: it doesn't matter if the request contains a connection:keep-alive header or not, curl does not hang under any circumstance and jetty will never add a connection:keep-alive header in the response. In addition, it turns out that big ip, removes the transfer-encoding:chunked header that jetty generated (see local debug block) from the response (we're using the recommended selective response chunking mode in the big ip http profile). In my opinion that is wrong, since it doesn't add the Content-Length header, something which would require the connection to be closed in order for the client to know when there is no more content. BTW, the servlet is spitting out the request headers as-is. Comments, anyone? ************* Jetty 9.1 (through big ip) ************* ➜ tmp curl -i https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: ["c2600ed9-5e37-4d86-bfd0-f6330c818961"] Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098g74r8cd0loxh65ry63fq5mle.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/ Vary: Accept-Encoding Header name X-Forwarded-SSL - Header true Header name X-Forwarded-For - Header 139.112.144.209 Header name User-Agent - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 Header name Accept - Header */* Header name Host - Header whereever.no Does not hang +++++++++++++++++++ curl -i -H "*Connection: Keep-Alive*" https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: ["7549c3f2-c396-4b26-802d-40350aa64d4c"] Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee0981q2s047k9wh6aspppnrzwaxbe.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain *Connection: keep-alive* Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/ Vary: Accept-Encoding Header name X-Forwarded-SSL - Header true Header name X-Forwarded-For - Header 139.112.144.209 Header name User-Agent - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 *Header name Connection* * - Header keep-alive* Header name Accept - Header */* Header name Host - Header whereever.no * This one hangs* ************* Jetty 8 (through big ip) ************* ➜ tmp curl -i https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: ["edba831e-bee6-46e2-bb05-74ecf4dd211f"] Set-Cookie: SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pqsvn1ha8o6ve3zjba0wq42s.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/ Vary: Accept-Encoding Header name Host - Header whereever.no Header name Accept - Header */* Header name X-Forwarded-For - Header 139.112.144.209 Header name X-Forwarded-SSL - Header true Header name User-Agent - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 Does not hang +++++++++++++++++++ tmp curl -i -H *"Connection: Keep-Alive"* https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: ["509b4ffb-cd28-4dfc-9f29-6d05571162c5"] Set-Cookie: SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pul9pgxgmzmt14loahxebxjab.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/ Vary: Accept-Encoding Header name Host - Header whereever.no Header name Accept - Header */* Header name X-Forwarded-For - Header 139.112.144.209 Header name X-Forwarded-SSL - Header true *Header name Connection* * - Header keep-alive* Header name User-Agent - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 Does not hang ************* Local test (not through big ip) ************* this is jetty 8, but it looks similar for jetty 9 with regards to the cunked transfer-encoding $ curl -i -H "Connection: keep-alive" http://localhost:12345/whereever HTTP/1.1 200 OK X-MiniProfiler-Ids: ["c9172169-755c-4bc2-a03d-9049e2b77e95"] Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098r6navchnwslj1xn8vwlkrqoq2.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only Vary: Accept-Encoding X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain *Transfer-Encoding: chunked* Header name Host - Header localhost:12345 Header name Accept - Header */* Header name Connection - Header keep-alive Header name User-Agent - Header curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 2014-01-28 Stefan Magnus Landrø <stefan.lan...@gmail.com> > Now new infrastructure, so this is definitely triggered by upgrading > jetty. However, the actual issue might be caused by new behaviour in jetty > triggering a bug in big ip. > > Stefan > > > 2014-01-28 "" <lord.bud...@gmail.com> > > Stefan, >> >> You said this was an 8.x to 9.x upgrade, so if the F5 was existing, it >> shouldn't be an issue unless there are infrastructure changes at the same >> time for upgrade. We use F5 and the only time we have seen something >> similar was where a perimeter f/w was dropping an established connection >> (due to inactivity) within the keep alive time. When the f/w does that >> the browser does not know the connection it has is stuffed. >> >> tcpdump is the best approach. >> >> >> >> >> On 28 January 2014 08:45, Stefan Magnus Landrø >> <stefan.lan...@gmail.com>wrote: >> >>> Thanks, Simone! I'm afraid I have to agree with you - this probably >>> won't fix my issue. I'll do some tcpdumping in the next few days - I'll >>> keep you posted. This might as well turn out being a f5 big ip issue. >>> >>> Stefan >>> >>> >>> 2014-01-27 Simone Bordet <sbor...@intalio.com> >>> >>> Hi, >>>> >>>> On Fri, Jan 24, 2014 at 9:36 PM, "" <lord.bud...@gmail.com> wrote: >>>> > Good find and a _critical_ bug in my view. >>>> > >>>> > A change that came with 9.0 as it was this in 8.x :- >>>> > >>>> > _header.put(CONNECTION_KEEP_ALIVE); >>>> > if (connection!=null) >>>> > { >>>> > _header.setPutIndex(_header.putIndex()-2); >>>> > _header.put((byte)','); >>>> > >>>> _header.put(connection.toString().getBytes()); >>>> > _header.put(CRLF); >>>> > } >>>> > >>>> > which had less logic, shorter lines... Yes, it copied 2 more bytes >>>> than it >>>> > needed to but I would prefer less logic/shorter code over saving 2 >>>> bytes. >>>> > >>>> > To fix this, you could _try_ turning off one connect on the f5 if you >>>> don't >>>> > need it for other reasons. >>>> > >>>> > We are also behind an F5, and are just about to start looking at >>>> moving from >>>> > 8.x to 9.x. ... so thanks ... will raise on support. >>>> >>>> I filed https://bugs.eclipse.org/bugs/show_bug.cgi?id=426739 and fixed >>>> it. >>>> >>>> Note that this is a corner case that should happen very rarely, only >>>> when the "Connection" header gets some custom value that is not >>>> "close" or "upgrade" or "keep-alive". >>>> >>>> Therefore, I am dubious that the original problem (curl hanging for >>>> chunked responses) is fixed by this fix. >>>> >>>> Let us know if that is the case, though. >>>> >>>> -- >>>> Simone Bordet >>>> ---- >>>> http://cometd.org >>>> http://webtide.com >>>> http://intalio.com >>>> Developer advice, training, services and support >>>> from the Jetty & CometD experts. >>>> Intalio, the modern way to build business applications. >>>> _______________________________________________ >>>> jetty-users mailing list >>>> jetty-users@eclipse.org >>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>> >>> >>> >>> >>> -- >>> BEKK Open >>> http://open.bekk.no >>> >>> TesTcl - a unit test framework for iRules >>> http://testcl.com >>> >>> _______________________________________________ >>> jetty-users mailing list >>> jetty-users@eclipse.org >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >>> >> >> _______________________________________________ >> jetty-users mailing list >> jetty-users@eclipse.org >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> >> > > > > -- BEKK Open http://open.bekk.no TesTcl - a unit test framework for iRules http://testcl.com
_______________________________________________ jetty-users mailing list jetty-users@eclipse.org https://dev.eclipse.org/mailman/listinfo/jetty-users
