Re: [jetty-users] Upgrading to Jetty 9.1
2014-01-28 lord.bud...@gmail.com We also use selective chunking but _do_ use one connect. Any reason why you have one connect off ? Not really. Historic reasons. Will switch it on soon. Oh, the request header X-Forwarded-SSL.I thought Jetty looks for X-Forwarded-Proto with value https to enable secure session cookies. Indeed, this is historic too. We actually don't use it any more. Have to do some cleaning up in the iRules... BTW, if you're into irules, you should give my irule testing framework a go: testcl.com Oh, and totally irrelevant, why are your session cookies so huge and containing a repeat of the first part. Feel free to ignore this :) We're using the jdbc session id manager - it allows us to perform deploys without downtime. The first/last parts identify a hashed version of the hostname. Btw, performance-wise it sucks (there is lots of synchronization code in the AbstractSession(Id)Manager) - that's why we're in the process of writing our own Hazelcast-based session manager from scratch (without all the synchro code). Using jdbc, we can't do more than 10-20 logins per second per server, and that will give us problems down the road. On 29 January 2014 00:30, Stefan Magnus Landrø stefan.lan...@gmail.comwrote: Hi there, I've been looking further into the issue, and it seems clear what is going on. This is what happens when running requests through our big ip (not using one connect): jetty 9.1: whenever a request contains a connection:keep-alive header (the default for all browsers I believe, but not for curl), curl hangs until the keep alive timeout is reached. Also, jetty replies with a connection: keep-alive header in the response. whenever a request does not contain a connection:keep-alive header, curl does not hang, and jetty doesn't add a connection:keep-alive to the response jetty 8: it doesn't matter if the request contains a connection:keep-alive header or not, curl does not hang under any circumstance and jetty will never add a connection:keep-alive header in the response. In addition, it turns out that big ip, removes the transfer-encoding:chunked header that jetty generated (see local debug block) from the response (we're using the recommended selective response chunking mode in the big ip http profile). In my opinion that is wrong, since it doesn't add the Content-Length header, something which would require the connection to be closed in order for the client to know when there is no more content. BTW, the servlet is spitting out the request headers as-is. Comments, anyone? * Jetty 9.1 (through big ip) * ➜ tmp curl -i https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: [c2600ed9-5e37-4d86-bfd0-f6330c818961] Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098g74r8cd0loxh65ry63fq5mle.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain Set-Cookie: BIGipServerpool_sticky=111834251.3879.; path=/ Vary: Accept-Encoding Header name X-Forwarded-SSL - Header true Header name X-Forwarded-For - Header 139.112.144.209 Header name User-Agent - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 Header name Accept - Header */* Header name Host - Header whereever.no Does not hang +++ curl -i -H *Connection: Keep-Alive* https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: [7549c3f2-c396-4b26-802d-40350aa64d4c] Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee0981q2s047k9wh6aspppnrzwaxbe.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain *Connection: keep-alive* Set-Cookie: BIGipServerpool_sticky=111834251.3879.; path=/ Vary: Accept-Encoding Header name X-Forwarded-SSL - Header true Header name X-Forwarded-For - Header 139.112.144.209 Header name User-Agent - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 *Header name Connection* * - Header keep-alive* Header name Accept - Header */* Header name Host - Header whereever.no * This one hangs* * Jetty 8 (through big ip) * ➜ tmp curl -i https://whereever.no HTTP/1.0 200 Connection
Re: [jetty-users] [jetty-9.1.1.v20140108] Jetty client idle timeout
Hi, On Tue, Jan 28, 2014 at 10:47 PM, Vijay Panghal vijay.pang...@gmail.com wrote: Hi, We had some issue on server side that did cause the client idle timeout. Thanks for the help! All right, thank you for reporting back. -- Simone Bordet http://cometd.org http://webtide.com http://intalio.com Developer advice, training, services and support from the Jetty CometD experts. Intalio, the modern way to build business applications. ___ jetty-users mailing list jetty-users@eclipse.org https://dev.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Upgrading to Jetty 9.1
Hmmm, a Hazelcast session manager would be something nice to contribute to Jetty ? We use sticky sessions with the F5 with no replication. So not for us. I got one of our apps running locally on Jetty 9 today and will try and get it promoted to a test env with an F5 tomorrow. It was a tad more difficult for us as we were not using any of the jetty etc/xml files on 8.1.12 or their sh script. We also had a custom request logger that had to be updated. So _much_ changed with 9.x. We do use iRules within the company, just not in the apps in our dept. On 29 January 2014 21:19, Stefan Magnus Landrø stefan.lan...@gmail.comwrote: 2014-01-28 lord.bud...@gmail.com We also use selective chunking but _do_ use one connect. Any reason why you have one connect off ? Not really. Historic reasons. Will switch it on soon. Oh, the request header X-Forwarded-SSL.I thought Jetty looks for X-Forwarded-Proto with value https to enable secure session cookies. Indeed, this is historic too. We actually don't use it any more. Have to do some cleaning up in the iRules... BTW, if you're into irules, you should give my irule testing framework a go: testcl.com Oh, and totally irrelevant, why are your session cookies so huge and containing a repeat of the first part. Feel free to ignore this :) We're using the jdbc session id manager - it allows us to perform deploys without downtime. The first/last parts identify a hashed version of the hostname. Btw, performance-wise it sucks (there is lots of synchronization code in the AbstractSession(Id)Manager) - that's why we're in the process of writing our own Hazelcast-based session manager from scratch (without all the synchro code). Using jdbc, we can't do more than 10-20 logins per second per server, and that will give us problems down the road. On 29 January 2014 00:30, Stefan Magnus Landrø stefan.lan...@gmail.comwrote: Hi there, I've been looking further into the issue, and it seems clear what is going on. This is what happens when running requests through our big ip (not using one connect): jetty 9.1: whenever a request contains a connection:keep-alive header (the default for all browsers I believe, but not for curl), curl hangs until the keep alive timeout is reached. Also, jetty replies with a connection: keep-alive header in the response. whenever a request does not contain a connection:keep-alive header, curl does not hang, and jetty doesn't add a connection:keep-alive to the response jetty 8: it doesn't matter if the request contains a connection:keep-alive header or not, curl does not hang under any circumstance and jetty will never add a connection:keep-alive header in the response. In addition, it turns out that big ip, removes the transfer-encoding:chunked header that jetty generated (see local debug block) from the response (we're using the recommended selective response chunking mode in the big ip http profile). In my opinion that is wrong, since it doesn't add the Content-Length header, something which would require the connection to be closed in order for the client to know when there is no more content. BTW, the servlet is spitting out the request headers as-is. Comments, anyone? * Jetty 9.1 (through big ip) * ➜ tmp curl -i https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: [c2600ed9-5e37-4d86-bfd0-f6330c818961] Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098g74r8cd0loxh65ry63fq5mle.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control: no-cache, no-store, no-transform Content-Type: text/plain Set-Cookie: BIGipServerpool_sticky=111834251.3879.; path=/ Vary: Accept-Encoding Header name X-Forwarded-SSL - Header true Header name X-Forwarded-For - Header 139.112.144.209 Header name User-Agent - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5 Header name Accept - Header */* Header name Host - Header whereever.no Does not hang +++ curl -i -H *Connection: Keep-Alive* https://whereever.no HTTP/1.0 200 Connection established HTTP/1.1 200 OK X-MiniProfiler-Ids: [7549c3f2-c396-4b26-802d-40350aa64d4c] Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee0981q2s047k9wh6aspppnrzwaxbe.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block X-Permitted-Cross-Domain-Policies: master-only X-Content-Type-Options: nosniff Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/ Cache-Control:
Re: [jetty-users] Upgrading to Jetty 9.1
Hi, I tracked this issue with https://bugs.eclipse.org/bugs/show_bug.cgi?id=426870. On Tue, Jan 28, 2014 at 12:30 PM, Stefan Magnus Landrø stefan.lan...@gmail.com wrote: Hi there, I've been looking further into the issue, and it seems clear what is going on. This is what happens when running requests through our big ip (not using one connect): jetty 9.1: whenever a request contains a connection:keep-alive header (the default for all browsers I believe, but not for curl), curl hangs until the keep alive timeout is reached. Also, jetty replies with a connection: keep-alive header in the response. This is a bug in 9.1.1, that I reported above. For a HTTP 1.0 request with Connection: keep-alive and unknown response content length, Jetty should have closed the connection. whenever a request does not contain a connection:keep-alive header, curl does not hang, and jetty doesn't add a connection:keep-alive to the response jetty 8: it doesn't matter if the request contains a connection:keep-alive header or not, curl does not hang under any circumstance and jetty will never add a connection:keep-alive header in the response. This is a bug (not tracked, not fixed) in Jetty 8. The Connection: keep-alive header should be added in the response if the server knows the response content length. In addition, it turns out that big ip, removes the transfer-encoding:chunked header that jetty generated (see local debug block) from the response (we're using the recommended selective response chunking mode in the big ip http profile). In my opinion that is wrong, since it doesn't add the Content-Length header, something which would require the connection to be closed in order for the client to know when there is no more content. This would require more investigation. Curl to F5 is HTTP 1.0, so F5 cannot send chunked content to a HTTP 1.0 client because it won't be able to parse it (chunked content is only a HTTP 1.1 feature). If F5 speaks HTTP 1.1 with Jetty, then Jetty is right in replying with chunked content, and F5 has to perform a transformation by removing the Transfer-Encoding header, removing the chunked encoding and then by closing the connection to the client. If F5 speaks HTTP 1.0 with Jetty, then Jetty is mistaken because it cannot send chunked content to a HTTP 1.0 client. In my tests, Jetty never sent chunked content to a HTTP 1.0 client, but you may be hitting a corner case that I did not. I just pushed a fix for #426870, can you try it out ? -- Simone Bordet http://cometd.org http://webtide.com http://intalio.com Developer advice, training, services and support from the Jetty CometD experts. Intalio, the modern way to build business applications. ___ jetty-users mailing list jetty-users@eclipse.org https://dev.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Upgrading to Jetty 9.1
I just pushed a fix for #426870, can you try it out ? Thanks, Simone! Just gave it a shot after building from source. It works perfectly now. I tried using the snapshot repo first, but apparently that is only built once a day. There is something I cant really understand, though - why the heck does jetty interpret the request coming from big ip as a http 1.0 request? My clients, both my browser and curl use http 1.1 - what could make jetty believe it's a 1.0 request? Actually, I'm using a proxy (which only supports http 1.0 it turns out ...), but this is https (using Connect), so the 1.1 request reaches the big ip. In addition the response from big ip clearly states it's a 1.1 request. BTW - I attached to the actual jvm behind the load balancer, and looked at the ServletRequest in ServletHandler.java. It tells me its a 1.0 request. Where is the jetty code that chooses to interpret the request as 1.0, 1.1 etc Unfortunately I'm struggeling to get access to run a tcpdump - but should get that in the next few days Stefan ___ jetty-users mailing list jetty-users@eclipse.org https://dev.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Upgrading to Jetty 9.1
I managed to capture the bytes byte array in HttpVersion.lookAheadGet(...) It clearly says version 1.0. I'll have to check the big ip config. Stefan 2014-01-29 Stefan Magnus Landrø stefan.lan...@gmail.com I just pushed a fix for #426870, can you try it out ? Thanks, Simone! Just gave it a shot after building from source. It works perfectly now. I tried using the snapshot repo first, but apparently that is only built once a day. There is something I cant really understand, though - why the heck does jetty interpret the request coming from big ip as a http 1.0 request? My clients, both my browser and curl use http 1.1 - what could make jetty believe it's a 1.0 request? Actually, I'm using a proxy (which only supports http 1.0 it turns out ...), but this is https (using Connect), so the 1.1 request reaches the big ip. In addition the response from big ip clearly states it's a 1.1 request. BTW - I attached to the actual jvm behind the load balancer, and looked at the ServletRequest in ServletHandler.java. It tells me its a 1.0 request. Where is the jetty code that chooses to interpret the request as 1.0, 1.1 etc Unfortunately I'm struggeling to get access to run a tcpdump - but should get that in the next few days Stefan -- BEKK Open http://open.bekk.no TesTcl - a unit test framework for iRules http://testcl.com ___ jetty-users mailing list jetty-users@eclipse.org https://dev.eclipse.org/mailman/listinfo/jetty-users
Re: [jetty-users] Upgrading to Jetty 9.1
Stefan, On 29 January 2014 19:19, Stefan Magnus Landrø stefan.lan...@gmail.com wrote: We're using the jdbc session id manager - it allows us to perform deploys without downtime. The first/last parts identify a hashed version of the hostname. Btw, performance-wise it sucks (there is lots of synchronization code in the AbstractSession(Id)Manager) - that's why we're in the process of writing our own Hazelcast-based session manager from scratch (without all the synchro code). Using jdbc, we can't do more than 10-20 logins per second per server, and that will give us problems down the road. Is this referring to jetty-8 or jetty-9? Recently in jetty-9 there was some unnecessary synchronization removed. However, I'm not sure that the synchronization is necessarily the limiting factor here database access tends to be the culprit speedwise, so it would be interesting to know about your environment and what you are observing. You may want to try the jetty mongodb session clustering solution? It would be great to have a Hazelcast-based session implementation - might be a good opportunity to collaborate between your company and Webtide? Jan -- Jan Bartel j...@intalio.com www.webtide.com 'Expert Jetty/CometD developer,production,operations advice' ___ jetty-users mailing list jetty-users@eclipse.org https://dev.eclipse.org/mailman/listinfo/jetty-users
