Re: mod_jk: sessions connecting although worker is oset to DIS
Am 23.03.2018 um 11:19 schrieb Martin Knoblauch:
Hi Rainer,
so basically I took the Apache path and ended up with the following brute
force method:
RewriteCond "%{REQUEST_METHOD}" "GET"
RewriteRule ^/xxx/facelets/logon.xhtml$ - [E=login_jsid:%{HTTP_COOKIE}]
CustomLog "/opt/xxx/apache2/logs/login_log" xxxlogheader env=login_jsid
RequestHeader unset Cookie env=login_jsid
GET requests on the login page from the same client/browser now end up on
different nodes.
Looking at JK_STICKY_IGNORE, this seems also to work as well:
RewriteCond "%{REQUEST_METHOD}" "GET"
RewriteRule ^/xxx/facelets/logon.xhtml$ - [E=JK_STICKY_IGNORE]
CustomLog "/opt/xxx/apache2/logs/login_log" xxxlogheader
env=JK_STICKY_IGNORE
I like this actually better, as it does not need to mess with the Cookie.
This is better when components like SiteMinder (for SSO) are involved. I
will give that into our testing/integration environments to see whether we
have any bad side effects.
So thanks again for the valuable input. I will also have a look at the
filter/valve suggestions. But they look more intrusive and getting approval
[did I say big professional organisation somewhere :-)] for that might be
more difficult.
Thanks for the feedback, let us know if you run into surprises :)
As I understand Chris, he plans to backport the tomcat valve so that it
will be a configurable standard part of all supported Tomcat versions.
That at least would lower the future cost of getting it used in
enterprise environments (from code addition down to config addition).
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk: sessions connecting although worker is oset to DIS
Hi Rainer,
so basically I took the Apache path and ended up with the following brute
force method:
RewriteCond "%{REQUEST_METHOD}" "GET"
RewriteRule ^/xxx/facelets/logon.xhtml$ - [E=login_jsid:%{HTTP_COOKIE}]
CustomLog "/opt/xxx/apache2/logs/login_log" xxxlogheader env=login_jsid
RequestHeader unset Cookie env=login_jsid
GET requests on the login page from the same client/browser now end up on
different nodes.
Looking at JK_STICKY_IGNORE, this seems also to work as well:
RewriteCond "%{REQUEST_METHOD}" "GET"
RewriteRule ^/xxx/facelets/logon.xhtml$ - [E=JK_STICKY_IGNORE]
CustomLog "/opt/xxx/apache2/logs/login_log" xxxlogheader
env=JK_STICKY_IGNORE
I like this actually better, as it does not need to mess with the Cookie.
This is better when components like SiteMinder (for SSO) are involved. I
will give that into our testing/integration environments to see whether we
have any bad side effects.
So thanks again for the valuable input. I will also have a look at the
filter/valve suggestions. But they look more intrusive and getting approval
[did I say big professional organisation somewhere :-)] for that might be
more difficult.
Cheers
Martin
On Thu, Mar 22, 2018 at 3:07 PM, Rainer Jung
wrote:
> Am 22.03.2018 um 11:58 schrieb Martin Knoblauch:
>
>> Hi,
>>
>> we have this annoying problem that although Tomcat workers are set to
>> DIS
>> state in mod_jk, there are new connections being opened. This prevents us
>> from "idle down" Tomcats in a timely fashion. This hurts when a restart is
>> needed. While our empathy for human clients is pretty limited, we care a
>> lot about our automated workloads :-)
>>
>> Our setup: Apache 2.4.x frontend -> mod_jk(1.2.42) balancer, sticky
>> sessions -> a bunch of Tomcat 7 (yes I know. Oldie but Goldie :-) workers.
>>
>> As far as I understand, the problem arises with HTTP(S) clients that
>> cache
>> the session cookies. If such a client tries to open a new connection to
>> the
>> backend, the old cookie passes to a DIS worker even if it is no longer
>> valid on the Tomcat side. Tomcat/Application then sees that the cookie is
>> not valid and doles out a new cookie which then allows the new connection
>> to be established.
>>
>> My idea is now to delete existing session cookies for certain request
>> pattern on the Apache frontend. If it works, it should force mod_jk to a
>> new ACT Tomcat. But I am nor really fond of messing with the Apache
>> configuration, as the reason lies on the mod_jk/Tomcat side.
>>
>> So if anybody has another suggestion I would love to hear it. Before I
>> forget, changing the application itself is not an option.
>>
>
> Either your idea, or - if those unwanted new request flows start with a
> specific URL, e.g. a login - mark those URLs specifically as not using the
> cookie. If you are using a uriworkermap.properties, look for
> "sticky_ignore" below http://tomcat.apache.org/conne
> ctors-doc/reference/uriworkermap.html#Rule%20extensions. If you are
> instead using JkMount style maps, look for JK_STICKY_IGNORE in
> http://tomcat.apache.org/connectors-doc/reference/apache.
> html#Advanced%20Environment%20Variables.
>
> Whenever you set the Apache environment variable JK_STICKY_IGNORE during a
> request execution, mod_jk will ignore the cookie and distribute like if it
> didn't see one. Such environment variables (different from Unix environment
> variables) can be set e.g. with mod_setenvif based on request properties
> and also with mod_rewrite as a [E=] side effect.
>
> But that would only help if you can describe request properties that alow
> you to decide - based on the request alone - whether the cookie needs to be
> ignored or not. You can use any request info including all headers for that
> decision. What is not possible, is to first check, whether the session is
> valid.
>
> You could also write a filter that checks the cookie for validity in
> Tomcat and if it is not valid, reply with a self-referential redirect
> including a Set-Cookie-Header that deletes the cookie. This filter should
> be written, such that it does not itself create a new session.
>
> HTH,
>
> Rainer
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
--
--
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
Re: mod_jk: sessions connecting although worker is oset to DIS
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Rainer, On 3/22/18 10:11 AM, Rainer Jung wrote: > Am 22.03.2018 um 15:07 schrieb Rainer Jung: >> Am 22.03.2018 um 11:58 schrieb Martin Knoblauch: >>> Hi, >>> >>> we have this annoying problem that although Tomcat workers are >>> set to DIS state in mod_jk, there are new connections being >>> opened. This prevents us from "idle down" Tomcats in a timely >>> fashion. This hurts when a restart is needed. While our empathy >>> for human clients is pretty limited, we care a lot about our >>> automated workloads :-) >>> >>> Our setup: Apache 2.4.x frontend -> mod_jk(1.2.42) balancer, >>> sticky sessions -> a bunch of Tomcat 7 (yes I know. Oldie but >>> Goldie :-) workers. >>> >>> As far as I understand, the problem arises with HTTP(S) >>> clients that cache the session cookies. If such a client tries >>> to open a new connection to the backend, the old cookie passes >>> to a DIS worker even if it is no longer valid on the Tomcat >>> side. Tomcat/Application then sees that the cookie is not valid >>> and doles out a new cookie which then allows the new >>> connection to be established. >>> >>> My idea is now to delete existing session cookies for certain >>> request pattern on the Apache frontend. If it works, it should >>> force mod_jk to a new ACT Tomcat. But I am nor really fond of >>> messing with the Apache configuration, as the reason lies on >>> the mod_jk/Tomcat side. >>> >>> So if anybody has another suggestion I would love to hear it. >>> Before I forget, changing the application itself is not an >>> option. >> >> Either your idea, or - if those unwanted new request flows start >> with a specific URL, e.g. a login - mark those URLs specifically >> as not using the cookie. If you are using a >> uriworkermap.properties, look for "sticky_ignore" below >> http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#R ule%20extensions. >> >> If you are instead using JkMount style maps, look for JK_STICKY_IGNORE >> in >> http://tomcat.apache.org/connectors-doc/reference/apache.html#Advance d%20Environment%20Variables. >> >> >> >> Whenever you set the Apache environment variable JK_STICKY_IGNORE >> during a request execution, mod_jk will ignore the cookie and >> distribute like if it didn't see one. Such environment variables >> (different from Unix environment variables) can be set e.g. with >> mod_setenvif based on request properties and also with >> mod_rewrite as a [E=] side effect. >> >> But that would only help if you can describe request properties >> that alow you to decide - based on the request alone - whether >> the cookie needs to be ignored or not. You can use any request >> info including all headers for that decision. What is not >> possible, is to first check, whether the session is valid. >> >> You could also write a filter that checks the cookie for validity >> in Tomcat and if it is not valid, reply with a self-referential >> redirect including a Set-Cookie-Header that deletes the cookie. >> This filter should be written, such that it does not itself >> create a new session. > > Plus I vaguely remember Chris (Schultz) has written such code that > yould be used pretty generic. Chris, am I right? If I am not > mistaken, it is actually here: > > http://home.apache.org/~schultz/lbdf/ Yup, that's it. I think the Valve might be better because it can intercept the request before any potential authentication occurs. The Filter may be easier to get into a Tomcat 7.0 environment/application, though. There's coverage of the theory and pretty pictures starting on slide 41 here: http://home.apache.org/~schultz/ApacheCon%20NA%202015/Load-balancing%20T omcat%20with%20mod_jk.pdf I'm using the Filter in production and WOW does it make a difference. We used to have to wait for almost a day for users to drain from a node and now it's maybe 30-90 minutes. I have not used the Valve in production because we are on Tomcat 8 (moving to 8.5) and neither version has the Valve in it, yet. Martin, please take the valve for a test-drive if you can and let me know how it goes. It it works for you I'll absolutely back-port it to Tomcat 8.5.x and possibly 8.0.x and 7.0.x. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlqzvZkdHGNocmlzQGNo cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFj/5xAAtIb5XbXQj5YGMtVl ZUjq0m2gfZ9C/p5Xtyumd1u5gXLrKLVwhxb9Zx4QF6pSOjvKY2mrbbPiLnILqn/w B2izrvi8Xr8nsIInAcrKV6jwWuQT9rJVLwG7PfkdK6UXmXSxq+HcqW78zpD9y11+ O1QM0PijYDvCbDFY9JME6PJ2EuxNFNLllPsmrJ/Ad/UTOLnWBLCp0IgrE0EP6uUg FAJVS+6EsItcuOL+OmgFnqVkJHkWFeRHC4mL4l9P/RCvgA6g2Hs2vW5uanXKORI4 6fiikmWrDdvoH9oEr01kOFaWGZuIjK6ASqenTsRY3uVIoLAkcrYMR/QCM0dyjq79 eVbhsGAWYACYo9Ah+2qa8kfeW2SzB/t2FOcrEn+VwfQlec7V8sOwvlxhsBTzxy9c 0eyCNi3g65y+oolSoYbap06mr6/8xIevoufhCDuRRFWsCOiDaTDcq5C1+nomWKts
Re: mod_jk: sessions connecting although worker is oset to DIS
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Martin, On 3/22/18 6:58 AM, Martin Knoblauch wrote: > we have this annoying problem that although Tomcat workers are set > to DIS state in mod_jk, there are new connections being opened. > This prevents us from "idle down" Tomcats in a timely fashion. This > hurts when a restart is needed. While our empathy for human clients > is pretty limited, we care a lot about our automated workloads :-) > > Our setup: Apache 2.4.x frontend -> mod_jk(1.2.42) balancer, > sticky sessions -> a bunch of Tomcat 7 (yes I know. Oldie but > Goldie :-) workers. > > As far as I understand, the problem arises with HTTP(S) clients > that cache the session cookies. If such a client tries to open a > new connection to the backend, the old cookie passes to a DIS > worker even if it is no longer valid on the Tomcat side. > Tomcat/Application then sees that the cookie is not valid and doles > out a new cookie which then allows the new connection to be > established. > > My idea is now to delete existing session cookies for certain > request pattern on the Apache frontend. If it works, it should > force mod_jk to a new ACT Tomcat. But I am nor really fond of > messing with the Apache configuration, as the reason lies on the > mod_jk/Tomcat side. > > So if anybody has another suggestion I would love to hear it. > Before I forget, changing the application itself is not an option. My suggestion is to use the LoadBalancerDrainingValve[1][2]. It was written for this exact purpose. It's currently only available in Tomcat 9.0.x, but you should be able to use it for previous Tomcat versions as the Valve/Request/Response interfaces shouldn't have changed in any incompatible ways between those versions. Hope that helps, - -chris [1] https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Load_Balancer _Draining_Valve [2] https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/catalina/valves/ LoadBalancerDrainingValve.html -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlqzurodHGNocmlzQGNo cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFjpfBAAyuM5kfF/ukIcnJjd NRoxyMIZtuKrVOht8tXn3or5ONye71Q/iZ3vd8UVlbrGurHWdi3ADH2n7Cvss8oX +02XSRzkW4EAmtu0Ailpc5PBDW0j3ASZTqnkymtNyBjQ8MDQkYtizPql51SU6tOp CaG4RP8TlEP91qI+nldCWMT+YbnAmAlEHvGOAJ1MzEKjmi14AfT8e4guBu2dfRCt QXderPV9imVveBmP53Z3WklPDCRMfEmGCdUf25Be/5ZnUGDZ5wmAZl3l/HLnUBAD tOwEBBCsm2OeVpvbbWj815ngnB/NkCC6gleePGN8J3HysJY+fnmPUl6Cw0t8jV3T CbqUlXJfH54QRvzie+1eGXGK1iwX3++icfecusC2Rea/9of+s65OVCFoJVkHj1qY 3T/tWZm0dbqLqAbadE7/2ihrOPBg3VnW/vdD+/lKXBEv/ECH/9nYNY3OkY6mJ3KS ptaKCdojTh1KCqxqilxfYEBMc9YFh1AoW/GYoF41mqFKRWc0CMLakCuNrm2vT8L0 u+KiFo8bPnuJ9Sw4jjKS2a52yp1jcJCjXnsDyO3vn7t3MNvyfygcbQR6Jftdq441 imwYVdwJTwULiR45cL6/AtGDSgKr1L4LhtcyhCzfxBRJh24W8u3dataH0loEVCup uAZ1f50984WWHJMJYmI6UrA7lKI= =8fI4 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk: sessions connecting although worker is oset to DIS
Am 22.03.2018 um 15:07 schrieb Rainer Jung: Am 22.03.2018 um 11:58 schrieb Martin Knoblauch: Hi, we have this annoying problem that although Tomcat workers are set to DIS state in mod_jk, there are new connections being opened. This prevents us from "idle down" Tomcats in a timely fashion. This hurts when a restart is needed. While our empathy for human clients is pretty limited, we care a lot about our automated workloads :-) Our setup: Apache 2.4.x frontend -> mod_jk(1.2.42) balancer, sticky sessions -> a bunch of Tomcat 7 (yes I know. Oldie but Goldie :-) workers. As far as I understand, the problem arises with HTTP(S) clients that cache the session cookies. If such a client tries to open a new connection to the backend, the old cookie passes to a DIS worker even if it is no longer valid on the Tomcat side. Tomcat/Application then sees that the cookie is not valid and doles out a new cookie which then allows the new connection to be established. My idea is now to delete existing session cookies for certain request pattern on the Apache frontend. If it works, it should force mod_jk to a new ACT Tomcat. But I am nor really fond of messing with the Apache configuration, as the reason lies on the mod_jk/Tomcat side. So if anybody has another suggestion I would love to hear it. Before I forget, changing the application itself is not an option. Either your idea, or - if those unwanted new request flows start with a specific URL, e.g. a login - mark those URLs specifically as not using the cookie. If you are using a uriworkermap.properties, look for "sticky_ignore" below http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#Rule%20extensions. If you are instead using JkMount style maps, look for JK_STICKY_IGNORE in http://tomcat.apache.org/connectors-doc/reference/apache.html#Advanced%20Environment%20Variables. Whenever you set the Apache environment variable JK_STICKY_IGNORE during a request execution, mod_jk will ignore the cookie and distribute like if it didn't see one. Such environment variables (different from Unix environment variables) can be set e.g. with mod_setenvif based on request properties and also with mod_rewrite as a [E=] side effect. But that would only help if you can describe request properties that alow you to decide - based on the request alone - whether the cookie needs to be ignored or not. You can use any request info including all headers for that decision. What is not possible, is to first check, whether the session is valid. You could also write a filter that checks the cookie for validity in Tomcat and if it is not valid, reply with a self-referential redirect including a Set-Cookie-Header that deletes the cookie. This filter should be written, such that it does not itself create a new session. Plus I vaguely remember Chris (Schultz) has written such code that yould be used pretty generic. Chris, am I right? If I am not mistaken, it is actually here: http://home.apache.org/~schultz/lbdf/ Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk: sessions connecting although worker is oset to DIS
Am 22.03.2018 um 11:58 schrieb Martin Knoblauch: Hi, we have this annoying problem that although Tomcat workers are set to DIS state in mod_jk, there are new connections being opened. This prevents us from "idle down" Tomcats in a timely fashion. This hurts when a restart is needed. While our empathy for human clients is pretty limited, we care a lot about our automated workloads :-) Our setup: Apache 2.4.x frontend -> mod_jk(1.2.42) balancer, sticky sessions -> a bunch of Tomcat 7 (yes I know. Oldie but Goldie :-) workers. As far as I understand, the problem arises with HTTP(S) clients that cache the session cookies. If such a client tries to open a new connection to the backend, the old cookie passes to a DIS worker even if it is no longer valid on the Tomcat side. Tomcat/Application then sees that the cookie is not valid and doles out a new cookie which then allows the new connection to be established. My idea is now to delete existing session cookies for certain request pattern on the Apache frontend. If it works, it should force mod_jk to a new ACT Tomcat. But I am nor really fond of messing with the Apache configuration, as the reason lies on the mod_jk/Tomcat side. So if anybody has another suggestion I would love to hear it. Before I forget, changing the application itself is not an option. Either your idea, or - if those unwanted new request flows start with a specific URL, e.g. a login - mark those URLs specifically as not using the cookie. If you are using a uriworkermap.properties, look for "sticky_ignore" below http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#Rule%20extensions. If you are instead using JkMount style maps, look for JK_STICKY_IGNORE in http://tomcat.apache.org/connectors-doc/reference/apache.html#Advanced%20Environment%20Variables. Whenever you set the Apache environment variable JK_STICKY_IGNORE during a request execution, mod_jk will ignore the cookie and distribute like if it didn't see one. Such environment variables (different from Unix environment variables) can be set e.g. with mod_setenvif based on request properties and also with mod_rewrite as a [E=] side effect. But that would only help if you can describe request properties that alow you to decide - based on the request alone - whether the cookie needs to be ignored or not. You can use any request info including all headers for that decision. What is not possible, is to first check, whether the session is valid. You could also write a filter that checks the cookie for validity in Tomcat and if it is not valid, reply with a self-referential redirect including a Set-Cookie-Header that deletes the cookie. This filter should be written, such that it does not itself create a new session. HTH, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
mod_jk: sessions connecting although worker is oset to DIS
Hi, we have this annoying problem that although Tomcat workers are set to DIS state in mod_jk, there are new connections being opened. This prevents us from "idle down" Tomcats in a timely fashion. This hurts when a restart is needed. While our empathy for human clients is pretty limited, we care a lot about our automated workloads :-) Our setup: Apache 2.4.x frontend -> mod_jk(1.2.42) balancer, sticky sessions -> a bunch of Tomcat 7 (yes I know. Oldie but Goldie :-) workers. As far as I understand, the problem arises with HTTP(S) clients that cache the session cookies. If such a client tries to open a new connection to the backend, the old cookie passes to a DIS worker even if it is no longer valid on the Tomcat side. Tomcat/Application then sees that the cookie is not valid and doles out a new cookie which then allows the new connection to be established. My idea is now to delete existing session cookies for certain request pattern on the Apache frontend. If it works, it should force mod_jk to a new ACT Tomcat. But I am nor really fond of messing with the Apache configuration, as the reason lies on the mod_jk/Tomcat side. So if anybody has another suggestion I would love to hear it. Before I forget, changing the application itself is not an option. TIA Martin -- -- Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
