Hmmm, a Hazelcast session manager would be something nice to contribute to Jetty ?
We use sticky sessions with the F5 with no replication..... So not for us. I got one of our apps running locally on Jetty 9 today and will try and get it promoted to a test env with an F5 tomorrow. It was a tad more difficult for us as we were not using any of the jetty etc/xml files on 8.1.12 or their sh script. We also had a custom request logger that had to be updated. So _much_ changed with 9.x. We do use iRules within the company, just not in the apps in our dept. On 29 January 2014 21:19, Stefan Magnus Landrø <[email protected]>wrote: > > > > 2014-01-28 "" <[email protected]> > >> >> We also use "selective" chunking but _do_ use one connect. Any reason >> why you have one connect off ? >> > Not really. Historic reasons. Will switch it on soon. > > Oh, the request header X-Forwarded-SSL. I thought Jetty looks for >> X-Forwarded-Proto with value "https" to enable secure session cookies. >> > Indeed, this is historic too. We actually don't use it any more. Have to > do some cleaning up in the iRules... > BTW, if you're into irules, you should give my irule testing framework a > go: testcl.com > >> >> Oh, and totally irrelevant, why are your session cookies so huge and >> containing a repeat of the first part. Feel free to ignore this :) >> > > We're using the jdbc session id manager - it allows us to perform deploys > without downtime. The first/last parts identify a hashed version of the > hostname. Btw, performance-wise it sucks (there is lots of synchronization > code in the AbstractSession(Id)Manager) - that's why we're in the process > of writing our own Hazelcast-based session manager from scratch (without > all the synchro code). Using jdbc, we can't do more than 10-20 logins per > second per server, and that will give us problems down the road. > > >> >> On 29 January 2014 00:30, Stefan Magnus Landrø >> <[email protected]>wrote: >> >>> Hi there, >>> >>> I've been looking further into the issue, and it seems clear what is >>> going on. >>> >>> This is what happens when running requests through our big ip (not using >>> one connect): >>> >>> jetty 9.1: >>> >>> whenever a request contains a connection:keep-alive header (the default >>> for all browsers I believe, but not for curl), curl hangs until the keep >>> alive timeout is reached. Also, jetty replies with a connection: keep-alive >>> header in the response. >>> >>> whenever a request does not contain a connection:keep-alive header, curl >>> does not hang, and jetty doesn't add a connection:keep-alive to the response >>> >>> jetty 8: >>> >>> it doesn't matter if the request contains a connection:keep-alive header >>> or not, curl does not hang under any circumstance and jetty will never add >>> a connection:keep-alive header in the response. >>> >>> In addition, it turns out that big ip, removes the >>> transfer-encoding:chunked header that jetty generated (see local debug >>> block) from the response (we're using the recommended selective response >>> chunking mode in the big ip http profile). In my opinion that is wrong, >>> since it doesn't add the Content-Length header, something which would >>> require the connection to be closed in order for the client to know when >>> there is no more content. >>> >>> BTW, the servlet is spitting out the request headers as-is. >>> >>> >>> Comments, anyone? >>> >>> >>> ************* >>> Jetty 9.1 (through big ip) >>> ************* >>> >>> >>> ➜ tmp curl -i https://whereever.no >>> HTTP/1.0 200 Connection established >>> >>> HTTP/1.1 200 OK >>> X-MiniProfiler-Ids: ["c2600ed9-5e37-4d86-bfd0-f6330c818961"] >>> Set-Cookie: >>> SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098g74r8cd0loxh65ry63fq5mle.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT >>> X-Frame-Options: SAMEORIGIN >>> X-XSS-Protection: 1; mode=block >>> X-Permitted-Cross-Domain-Policies: master-only >>> X-Content-Type-Options: nosniff >>> Content-Security-Policy-Report-Only: default-src 'none'; report-uri >>> /post/cspreport/ >>> Cache-Control: no-cache, no-store, no-transform >>> Content-Type: text/plain >>> Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/ >>> Vary: Accept-Encoding >>> >>> Header name X-Forwarded-SSL >>> - Header true >>> Header name X-Forwarded-For >>> - Header 139.112.144.209 >>> Header name User-Agent >>> - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 >>> OpenSSL/0.9.8y zlib/1.2.5 >>> Header name Accept >>> - Header */* >>> Header name Host >>> - Header whereever.no >>> >>> Does not hang >>> >>> >>> +++++++++++++++++++ >>> >>> curl -i -H "*Connection: Keep-Alive*" https://whereever.no >>> HTTP/1.0 200 Connection established >>> >>> HTTP/1.1 200 OK >>> X-MiniProfiler-Ids: ["7549c3f2-c396-4b26-802d-40350aa64d4c"] >>> Set-Cookie: >>> SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee0981q2s047k9wh6aspppnrzwaxbe.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT >>> X-Frame-Options: SAMEORIGIN >>> X-XSS-Protection: 1; mode=block >>> X-Permitted-Cross-Domain-Policies: master-only >>> X-Content-Type-Options: nosniff >>> Content-Security-Policy-Report-Only: default-src 'none'; report-uri >>> /post/cspreport/ >>> Cache-Control: no-cache, no-store, no-transform >>> Content-Type: text/plain >>> *Connection: keep-alive* >>> Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/ >>> Vary: Accept-Encoding >>> >>> Header name X-Forwarded-SSL >>> - Header true >>> Header name X-Forwarded-For >>> - Header 139.112.144.209 >>> Header name User-Agent >>> - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 >>> OpenSSL/0.9.8y zlib/1.2.5 >>> *Header name Connection* >>> * - Header keep-alive* >>> Header name Accept >>> - Header */* >>> Header name Host >>> - Header whereever.no >>> >>> * This one hangs* >>> >>> >>> ************* >>> Jetty 8 (through big ip) >>> ************* >>> >>> ➜ tmp curl -i https://whereever.no >>> HTTP/1.0 200 Connection established >>> >>> HTTP/1.1 200 OK >>> X-MiniProfiler-Ids: ["edba831e-bee6-46e2-bb05-74ecf4dd211f"] >>> Set-Cookie: >>> SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pqsvn1ha8o6ve3zjba0wq42s.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT >>> X-Frame-Options: SAMEORIGIN >>> X-XSS-Protection: 1; mode=block >>> X-Permitted-Cross-Domain-Policies: master-only >>> X-Content-Type-Options: nosniff >>> Content-Security-Policy-Report-Only: default-src 'none'; report-uri >>> /post/cspreport/ >>> Cache-Control: no-cache, no-store, no-transform >>> Content-Type: text/plain >>> Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/ >>> Vary: Accept-Encoding >>> >>> Header name Host >>> - Header whereever.no >>> Header name Accept >>> - Header */* >>> Header name X-Forwarded-For >>> - Header 139.112.144.209 >>> Header name X-Forwarded-SSL >>> - Header true >>> Header name User-Agent >>> - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 >>> OpenSSL/0.9.8y zlib/1.2.5 >>> >>> Does not hang >>> >>> +++++++++++++++++++ >>> >>> >>> tmp curl -i -H *"Connection: Keep-Alive"* https://whereever.no >>> HTTP/1.0 200 Connection established >>> >>> HTTP/1.1 200 OK >>> X-MiniProfiler-Ids: ["509b4ffb-cd28-4dfc-9f29-6d05571162c5"] >>> Set-Cookie: >>> SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pul9pgxgmzmt14loahxebxjab.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT >>> X-Frame-Options: SAMEORIGIN >>> X-XSS-Protection: 1; mode=block >>> X-Permitted-Cross-Domain-Policies: master-only >>> X-Content-Type-Options: nosniff >>> Content-Security-Policy-Report-Only: default-src 'none'; report-uri >>> /post/cspreport/ >>> Cache-Control: no-cache, no-store, no-transform >>> Content-Type: text/plain >>> Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/ >>> Vary: Accept-Encoding >>> >>> Header name Host >>> - Header whereever.no >>> Header name Accept >>> - Header */* >>> Header name X-Forwarded-For >>> - Header 139.112.144.209 >>> Header name X-Forwarded-SSL >>> - Header true >>> *Header name Connection* >>> * - Header keep-alive* >>> Header name User-Agent >>> - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 >>> OpenSSL/0.9.8y zlib/1.2.5 >>> >>> Does not hang >>> >>> >>> >>> ************* >>> Local test (not through big ip) >>> ************* >>> >>> this is jetty 8, but it looks similar for jetty 9 with regards to the >>> cunked transfer-encoding >>> >>> $ curl -i -H "Connection: keep-alive" http://localhost:12345/whereever >>> HTTP/1.1 200 OK >>> X-MiniProfiler-Ids: ["c9172169-755c-4bc2-a03d-9049e2b77e95"] >>> Set-Cookie: >>> SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098r6navchnwslj1xn8vwlkrqoq2.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT >>> X-Frame-Options: SAMEORIGIN >>> X-XSS-Protection: 1; mode=block >>> X-Permitted-Cross-Domain-Policies: master-only >>> Vary: Accept-Encoding >>> X-Content-Type-Options: nosniff >>> Content-Security-Policy-Report-Only: default-src 'none'; report-uri >>> /post/cspreport/ >>> Cache-Control: no-cache, no-store, no-transform >>> Content-Type: text/plain >>> *Transfer-Encoding: chunked* >>> >>> Header name Host >>> - Header localhost:12345 >>> Header name Accept >>> - Header */* >>> Header name Connection >>> - Header keep-alive >>> Header name User-Agent >>> - Header curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 >>> OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 >>> >>> >>> >>> >>> >>> 2014-01-28 Stefan Magnus Landrø <[email protected]> >>> >>> Now new infrastructure, so this is definitely triggered by upgrading >>>> jetty. However, the actual issue might be caused by new behaviour in jetty >>>> triggering a bug in big ip. >>>> >>>> Stefan >>>> >>>> >>>> 2014-01-28 "" <[email protected]> >>>> >>>> Stefan, >>>>> >>>>> You said this was an 8.x to 9.x upgrade, so if the F5 was existing, it >>>>> shouldn't be an issue unless there are infrastructure changes at the same >>>>> time for upgrade. We use F5 and the only time we have seen something >>>>> similar was where a perimeter f/w was dropping an established connection >>>>> (due to inactivity) within the keep alive time. When the f/w does that >>>>> the browser does not know the connection it has is stuffed. >>>>> >>>>> tcpdump is the best approach. >>>>> >>>>> >>>>> >>>>> >>>>> On 28 January 2014 08:45, Stefan Magnus Landrø < >>>>> [email protected]> wrote: >>>>> >>>>>> Thanks, Simone! I'm afraid I have to agree with you - this probably >>>>>> won't fix my issue. I'll do some tcpdumping in the next few days - I'll >>>>>> keep you posted. This might as well turn out being a f5 big ip issue. >>>>>> >>>>>> Stefan >>>>>> >>>>>> >>>>>> 2014-01-27 Simone Bordet <[email protected]> >>>>>> >>>>>> Hi, >>>>>>> >>>>>>> On Fri, Jan 24, 2014 at 9:36 PM, "" <[email protected]> wrote: >>>>>>> > Good find and a _critical_ bug in my view. >>>>>>> > >>>>>>> > A change that came with 9.0 as it was this in 8.x :- >>>>>>> > >>>>>>> > _header.put(CONNECTION_KEEP_ALIVE); >>>>>>> > if (connection!=null) >>>>>>> > { >>>>>>> > _header.setPutIndex(_header.putIndex()-2); >>>>>>> > _header.put((byte)','); >>>>>>> > >>>>>>> _header.put(connection.toString().getBytes()); >>>>>>> > _header.put(CRLF); >>>>>>> > } >>>>>>> > >>>>>>> > which had less logic, shorter lines... Yes, it copied 2 more bytes >>>>>>> than it >>>>>>> > needed to but I would prefer less logic/shorter code over saving 2 >>>>>>> bytes. >>>>>>> > >>>>>>> > To fix this, you could _try_ turning off one connect on the f5 if >>>>>>> you don't >>>>>>> > need it for other reasons. >>>>>>> > >>>>>>> > We are also behind an F5, and are just about to start looking at >>>>>>> moving from >>>>>>> > 8.x to 9.x. ... so thanks ... will raise on support. >>>>>>> >>>>>>> I filed https://bugs.eclipse.org/bugs/show_bug.cgi?id=426739 and >>>>>>> fixed it. >>>>>>> >>>>>>> Note that this is a corner case that should happen very rarely, only >>>>>>> when the "Connection" header gets some custom value that is not >>>>>>> "close" or "upgrade" or "keep-alive". >>>>>>> >>>>>>> Therefore, I am dubious that the original problem (curl hanging for >>>>>>> chunked responses) is fixed by this fix. >>>>>>> >>>>>>> Let us know if that is the case, though. >>>>>>> >>>>>>> -- >>>>>>> Simone Bordet >>>>>>> ---- >>>>>>> http://cometd.org >>>>>>> http://webtide.com >>>>>>> http://intalio.com >>>>>>> Developer advice, training, services and support >>>>>>> from the Jetty & CometD experts. >>>>>>> Intalio, the modern way to build business applications. >>>>>>> _______________________________________________ >>>>>>> jetty-users mailing list >>>>>>> [email protected] >>>>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> BEKK Open >>>>>> http://open.bekk.no >>>>>> >>>>>> TesTcl - a unit test framework for iRules >>>>>> http://testcl.com >>>>>> >>>>>> _______________________________________________ >>>>>> jetty-users mailing list >>>>>> [email protected] >>>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> jetty-users mailing list >>>>> [email protected] >>>>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> BEKK Open >>> http://open.bekk.no >>> >>> TesTcl - a unit test framework for iRules >>> http://testcl.com >>> >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >>> >>> >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> >> > > > -- > BEKK Open > http://open.bekk.no > > TesTcl - a unit test framework for iRules > http://testcl.com > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > >
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
