Re: Tomcat 6.0.26 on Windows server - sessions are not expired
2014-02-03 Daniel Mikusa dmik...@gopivotal.com: On Feb 3, 2014, at 9:00 AM, Maor Yosef maoryo...@gmail.com wrote: On Mon, Feb 3, 2014 at 2:10 PM, Daniel Mikusa dmik...@gopivotal.com wrote: Also, please realize that a JSP page, even one that simply prints out OK will create a session. This is by design and if you don't want it to create a session you need to explicitly indicate that in your JSP. Ex: %@ page session=false % This is important in scenarios where you're doing load testing and using custom HTTP clients, because these client's may not be handling sessions correctly and thus end up creating a new session every time they access the page. Another way to handle misbehaving clients is, what Konstantin mentioned in his earlier message about web bots, the CrawlerSessionManagerValve. I added the above text to the FAQ page here: https://wiki.apache.org/tomcat/OutOfMemory Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
Hi. 1. We are aware that 6.0.26 is old, but since there is a large operational impact, we wont upgrade the tomcat until we will know its definetly an issue in this specific version 2. You are right, it was my mistake, it causes OOM and not stack overflow, when we see high sessions count we get exceptions saying unable to create new native thread 3. Looking at the sessions manager, we see a lot of sessions with a negative TTL - meaning they werent expired, if I manually expire them then the sessions count decreases. 4. Can you point me to an article on how to configure different background thread for each container? is it configured in tomcat or should be implemented in the application? Thanks, On Sun, Feb 2, 2014 at 7:38 PM, Konstantin Kolinko knst.koli...@gmail.comwrote: 2014-02-02 Maor Yosef maoryo...@gmail.com: Hi, 1. 6.0.26 is old. We are facing issues where the sessions are not being expired and eventually causing a stack overflow. 2. Non-expiring sessions may cause OOM, but they cannot cause a stack overflow. What are your evidences? We see that at some point the sessions get pilled up and we need to manually expire those sessions through the manager application, or until we will restart tomcat but after a few hours / days, sessions will start to get stuck again 3. Increasing count of session does not mean that sessions do not expire. Your evidence = ? We see it in all the applications that are deployed on tomcat, including the manager application - this rules out (in my opinion) the possibility that its an issue with our application. 4. Sessions are expired by a background thread. If the thread is stuck somewhere (e.g. doing auto-deployment work) it will affect expiration of sessions. Your thread dump = ? By default there is one background thread that is shared by all container levels in Tomcat, but you can configure a separate one in each container (container = Context, Host or Engine). 5. Web bots that do not support cookies may generate multiple sessions. See CrawlerSessionManagerValve Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
On 3 February 2014 09:30, Maor Yosef maoryo...@gmail.com wrote: 2. You are right, it was my mistake, it causes OOM and not stack overflow, when we see high sessions count we get exceptions saying unable to create new native thread do you use a 32 bit vm? Because most of the time when i see this i ask the same question, and most of the time that is then true and they have specified quite a high heap size. If you do that then the native part of the memory that a 32 bit process can have is getting small -- Johan Compagner Servoy
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
using 64 bit On Mon, Feb 3, 2014 at 10:51 AM, Johan Compagner jcompag...@servoy.comwrote: On 3 February 2014 09:30, Maor Yosef maoryo...@gmail.com wrote: 2. You are right, it was my mistake, it causes OOM and not stack overflow, when we see high sessions count we get exceptions saying unable to create new native thread do you use a 32 bit vm? Because most of the time when i see this i ask the same question, and most of the time that is then true and they have specified quite a high heap size. If you do that then the native part of the memory that a 32 bit process can have is getting small -- Johan Compagner Servoy
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
On Feb 3, 2014, at 3:30 AM, Maor Yosef maoryo...@gmail.com wrote: Hi. 1. We are aware that 6.0.26 is old, but since there is a large operational impact, we wont upgrade the tomcat until we will know its definetly an issue in this specific version While I understand what you’re saying, I disagree. If you need to sell the change to management, you should take a look at the security problems that have been fixed since 6.0.26 and weigh the cost of upgrading versus a security breach or manually applying mitigations, if that’s even possible. http://tomcat.apache.org/security-6.html 2. You are right, it was my mistake, it causes OOM and not stack overflow, when we see high sessions count we get exceptions saying unable to create new native thread” This is telling you that there’s not enough memory to allocate any more threads. 1.) Have you tried setting “-Xss”? This will set the thread stack size, i.e. how much memory each thread has available for its stack. Often times you don’t need nearly as much as the default allocated by the JVM, so you can lower it and get more threads out of the same available memory. Maybe try 256k or even lower. You’ll know you went too low if you see StackOverflowErrors. 2.) How many threads are being created / used? Perhaps you’re creating too many threads? You’ll probably want to do some monitoring and see how many the Tomcat is creating / using and how many your application is creating / using. Perhaps you’ve got a problem where too many threads are being created or where threads are being created and not properly cleaned up. Tools that could help here: jconsole / jvisualvm or thread dumps 3. Looking at the sessions manager, we see a lot of sessions with a negative TTL - meaning they werent expired, if I manually expire them then the sessions count decreases. This doesn’t sound related, although it’s hard to say. Might be helpful if you can include your configuration, minus comments. 4. Can you point me to an article on how to configure different background thread for each container? is it configured in tomcat or should be implemented in the application? What exactly do you mean here? You can control Tomcat’s thread usage with an Executor [1] or directly on the connector [2]. Dan [1] - http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html [2] - http://tomcat.apache.org/tomcat-6.0-doc/config/http.html Thanks, On Sun, Feb 2, 2014 at 7:38 PM, Konstantin Kolinko knst.koli...@gmail.comwrote: 2014-02-02 Maor Yosef maoryo...@gmail.com: Hi, 1. 6.0.26 is old. We are facing issues where the sessions are not being expired and eventually causing a stack overflow. 2. Non-expiring sessions may cause OOM, but they cannot cause a stack overflow. What are your evidences? We see that at some point the sessions get pilled up and we need to manually expire those sessions through the manager application, or until we will restart tomcat but after a few hours / days, sessions will start to get stuck again 3. Increasing count of session does not mean that sessions do not expire. Your evidence = ? We see it in all the applications that are deployed on tomcat, including the manager application - this rules out (in my opinion) the possibility that its an issue with our application. 4. Sessions are expired by a background thread. If the thread is stuck somewhere (e.g. doing auto-deployment work) it will affect expiration of sessions. Your thread dump = ? By default there is one background thread that is shared by all container levels in Tomcat, but you can configure a separate one in each container (container = Context, Host or Engine). 5. Web bots that do not support cookies may generate multiple sessions. See CrawlerSessionManagerValve Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
2014-02-03 Maor Yosef maoryo...@gmail.com: Hi. Please read the rules and do not top-post, as it is hard to follow. http://tomcat.apache.org/lists.html#tomcat-users 4. Can you point me to an article on how to configure different background thread for each container? is it configured in tomcat or should be implemented in the application? By setting a positive value for backgroundProcessorDelay attribute. http://tomcat.apache.org/tomcat-6.0-doc/config/engine.html#Common_Attributes The rest was answered by Daniel. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
On Mon, Feb 3, 2014 at 2:10 PM, Daniel Mikusa dmik...@gopivotal.com wrote: On Feb 3, 2014, at 3:30 AM, Maor Yosef maoryo...@gmail.com wrote: Hi. 1. We are aware that 6.0.26 is old, but since there is a large operational impact, we wont upgrade the tomcat until we will know its definetly an issue in this specific version While I understand what you're saying, I disagree. If you need to sell the change to management, you should take a look at the security problems that have been fixed since 6.0.26 and weigh the cost of upgrading versus a security breach or manually applying mitigations, if that's even possible. http://tomcat.apache.org/security-6.html I agree that an upgrade is needed, but still before we will determine to which version to upgrade, we need to solve this issue 2. You are right, it was my mistake, it causes OOM and not stack overflow, when we see high sessions count we get exceptions saying unable to create new native thread This is telling you that there's not enough memory to allocate any more threads. 1.) Have you tried setting -Xss? This will set the thread stack size, i.e. how much memory each thread has available for its stack. Often times you don't need nearly as much as the default allocated by the JVM, so you can lower it and get more threads out of the same available memory. Maybe try 256k or even lower. You'll know you went too low if you see StackOverflowErrors. 2.) How many threads are being created / used? Perhaps you're creating too many threads? You'll probably want to do some monitoring and see how many the Tomcat is creating / using and how many your application is creating / using. Perhaps you've got a problem where too many threads are being created or where threads are being created and not properly cleaned up. Tools that could help here: jconsole / jvisualvm or thread dumps The memory configuration doesn't seems to be the issue, as mentioned before we see a direct correlation between the amount of opened sessions (which can go as high as 100k+) even for a simple application that is just running a simple jsp file, which only prints back to the screen OK, we still see a large number of sessions for it (also after adding a web.xml with session expiration time = 1) 3. Looking at the sessions manager, we see a lot of sessions with a negative TTL - meaning they werent expired, if I manually expire them then the sessions count decreases. This doesn't sound related, although it's hard to say. Might be helpful if you can include your configuration, minus comments. 4. Can you point me to an article on how to configure different background thread for each container? is it configured in tomcat or should be implemented in the application? What exactly do you mean here? You can control Tomcat's thread usage with an Executor [1] or directly on the connector [2]. Dan [1] - http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html [2] - http://tomcat.apache.org/tomcat-6.0-doc/config/http.html Thanks, On Sun, Feb 2, 2014 at 7:38 PM, Konstantin Kolinko knst.koli...@gmail.comwrote: 2014-02-02 Maor Yosef maoryo...@gmail.com: Hi, 1. 6.0.26 is old. We are facing issues where the sessions are not being expired and eventually causing a stack overflow. 2. Non-expiring sessions may cause OOM, but they cannot cause a stack overflow. What are your evidences? We see that at some point the sessions get pilled up and we need to manually expire those sessions through the manager application, or until we will restart tomcat but after a few hours / days, sessions will start to get stuck again 3. Increasing count of session does not mean that sessions do not expire. Your evidence = ? We see it in all the applications that are deployed on tomcat, including the manager application - this rules out (in my opinion) the possibility that its an issue with our application. 4. Sessions are expired by a background thread. If the thread is stuck somewhere (e.g. doing auto-deployment work) it will affect expiration of sessions. Your thread dump = ? By default there is one background thread that is shared by all container levels in Tomcat, but you can configure a separate one in each container (container = Context, Host or Engine). 5. Web bots that do not support cookies may generate multiple sessions. See CrawlerSessionManagerValve Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
On Feb 3, 2014, at 9:00 AM, Maor Yosef maoryo...@gmail.com wrote: On Mon, Feb 3, 2014 at 2:10 PM, Daniel Mikusa dmik...@gopivotal.com wrote: On Feb 3, 2014, at 3:30 AM, Maor Yosef maoryo...@gmail.com wrote: Hi. 1. We are aware that 6.0.26 is old, but since there is a large operational impact, we wont upgrade the tomcat until we will know its definetly an issue in this specific version While I understand what you're saying, I disagree. If you need to sell the change to management, you should take a look at the security problems that have been fixed since 6.0.26 and weigh the cost of upgrading versus a security breach or manually applying mitigations, if that's even possible. http://tomcat.apache.org/security-6.html I agree that an upgrade is needed, but still before we will determine to which version to upgrade, we need to solve this issue 2. You are right, it was my mistake, it causes OOM and not stack overflow, when we see high sessions count we get exceptions saying unable to create new native thread This is telling you that there's not enough memory to allocate any more threads. 1.) Have you tried setting -Xss? This will set the thread stack size, i.e. how much memory each thread has available for its stack. Often times you don't need nearly as much as the default allocated by the JVM, so you can lower it and get more threads out of the same available memory. Maybe try 256k or even lower. You'll know you went too low if you see StackOverflowErrors. 2.) How many threads are being created / used? Perhaps you're creating too many threads? You'll probably want to do some monitoring and see how many the Tomcat is creating / using and how many your application is creating / using. Perhaps you've got a problem where too many threads are being created or where threads are being created and not properly cleaned up. Tools that could help here: jconsole / jvisualvm or thread dumps The memory configuration doesn't seems to be the issue, I would beg to differ. It’s certainly one issue. You seem to have multiple. As I said previously, I don’t think your OOME is related to your session problem. At least it doesn’t seem to be based on the information you’ve given to date. as mentioned before we see a direct correlation between the amount of opened sessions (which can go as high as 100k+) even for a simple application that is just running a simple jsp file, which only prints back to the screen OK, we still see a large number of sessions for it (also after adding a web.xml with session expiration time = 1) Here’s what I’d suggest. 1.) In a test environment, install Tomcat 6.0.39. It’s the latest. 2.) Deploy your “simple application”, if it’s really as simple as you say it shouldn’t be a problem to install. 3.) See if you can replicate the problem. If so then report your findings here, just include a test app your steps to reproduce. If you’re unable to reproduce the issue, then consider upgrading or compare your configuration’s to see if your current config is doing something wrong. Also, please realize that a JSP page, even one that simply prints out “OK” will create a session. This is by design and if you don’t want it to create a session you need to explicitly indicate that in your JSP. Ex: %@ page session=false % This is important in scenarios where you’re doing load testing and using custom HTTP clients, because these client’s may not be handling sessions correctly and thus end up creating a new session every time they access the page. Another way to handle misbehaving clients is, what Konstantin mentioned in his earlier message about web bots, the CrawlerSessionManagerValve. 3. Looking at the sessions manager, we see a lot of sessions with a negative TTL - meaning they werent expired, if I manually expire them then the sessions count decreases. This doesn't sound related, although it's hard to say. Might be helpful if you can include your configuration, minus comments. Are you going to provide your configuration? There’s only so much we can tell you without knowing how you have your environment configured. Dan 4. Can you point me to an article on how to configure different background thread for each container? is it configured in tomcat or should be implemented in the application? What exactly do you mean here? You can control Tomcat's thread usage with an Executor [1] or directly on the connector [2]. Dan [1] - http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html [2] - http://tomcat.apache.org/tomcat-6.0-doc/config/http.html Thanks, On Sun, Feb 2, 2014 at 7:38 PM, Konstantin Kolinko knst.koli...@gmail.comwrote: 2014-02-02 Maor Yosef maoryo...@gmail.com: Hi, 1. 6.0.26 is old. We are facing issues where the sessions are not being expired and eventually causing a stack overflow. 2. Non-expiring sessions may cause OOM, but
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Daniel, On 2/3/14, 7:10 AM, Daniel Mikusa wrote: On Feb 3, 2014, at 3:30 AM, Maor Yosef maoryo...@gmail.com wrote: Hi. 1. We are aware that 6.0.26 is old, but since there is a large operational impact, we wont upgrade the tomcat until we will know its definetly an issue in this specific version While I understand what you’re saying, I disagree. If you need to sell the change to management, you should take a look at the security problems that have been fixed since 6.0.26 and weigh the cost of upgrading versus a security breach or manually applying mitigations, if that’s even possible. http://tomcat.apache.org/security-6.html +1 2. You are right, it was my mistake, it causes OOM and not stack overflow, when we see high sessions count we get exceptions saying unable to create new native thread” This is telling you that there’s not enough memory to allocate any more threads. 1.) Have you tried setting “-Xss”? This will set the thread stack size, i.e. how much memory each thread has available for its stack. Often times you don’t need nearly as much as the default allocated by the JVM, so you can lower it and get more threads out of the same available memory. Maybe try 256k or even lower. You’ll know you went too low if you see StackOverflowErrors. 2.) How many threads are being created / used? Perhaps you’re creating too many threads? You’ll probably want to do some monitoring and see how many the Tomcat is creating / using and how many your application is creating / using. Perhaps you’ve got a problem where too many threads are being created or where threads are being created and not properly cleaned up. Tools that could help here: jconsole / jvisualvm or thread dumps I'd be interested in knowing why threads are being created at all. Thread dumps will reveal a lot of good information. 3. Looking at the sessions manager, we see a lot of sessions with a negative TTL - meaning they werent expired, if I manually expire them then the sessions count decreases. This doesn’t sound related, although it’s hard to say. Might be helpful if you can include your configuration, minus comments. 4. Can you point me to an article on how to configure different background thread for each container? is it configured in tomcat or should be implemented in the application? If your background thread is becoming stuck, you should fix *that* instead of trying to work-around it. Thread dump(s)? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJS7+VNAAoJEBzwKT+lPKRYcjkP/2A0gQ3HnJNOA2724kxHiYO/ q4ZLnqJUAnepPttDYu4eL8/sehnmLm1kNQ0q/vby+5LLXLeU3QldmJsZSHwDft7M +sph2hXy0Ed6a/3sS4nYEHLWYcIs9rEi13EkMTgvewE7jEp4QldTisfHi4I3XgDq ZlraHHQjvPgbYFwzQxmwg2F7+ag69GqR52q9zECC97tXctTPQHxd8hJ40298y40w 2HIyDV6l9EuPVkan1/g7xuWxRbWoAiwhawkiGA606r1IhtO7cB7C6ulAyDyoLKqj NEe1EHfVeDvmiavw7evIcknTVyK1hcuQC0NPV5bSMEQnQf6ZTWw67FQfosQUmqA2 L+kYtPKDzsnF9slUfgI7YokEjzApZx/dElsZUdgatIvb5yz8IFCXKaiFxkcHGffx TzHMe6EAiDZglM5fMQIPmvuS5p5/iaJ5mMTZzamcOZ2VOD1/RDtqQm5MLljd4M/0 cVpGb/xEEZLGoj8mnXTfQq+NFYbjkCA3PcglvoBi4VtgOS7pBykccEFEv+1HavHC h4ROzGJ8u7uHhGbUx2WbxHfkTtk6HGLon1bIyQkP1vraAdsOClAfiEto/C+bv9jw y5iLOfEEHlZTPCTv6lbDtYmTBaOO1r/3LQ12kc3eZfzjQaOuGUo7jwYc4A0yTDDJ 8V4q1aiF7dn26chh/BsS =R+5r -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Konstantin, On 2/3/14, 7:40 AM, Konstantin Kolinko wrote: 2014-02-03 Maor Yosef maoryo...@gmail.com: Hi. Please read the rules and do not top-post, as it is hard to follow. http://tomcat.apache.org/lists.html#tomcat-users 4. Can you point me to an article on how to configure different background thread for each container? is it configured in tomcat or should be implemented in the application? By setting a positive value for backgroundProcessorDelay attribute. http://tomcat.apache.org/tomcat-6.0-doc/config/engine.html#Common_Attributes The rest was answered by Daniel. I think taking any action before posting some thread dumps would be a mistake. I would be willing to bet that the background thread is getting stuck on something application-specific. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJS7+X0AAoJEBzwKT+lPKRYRcEP/iy6lAankZZn3YqI6TKlTNbj 23XvQPdDalmxxcmLXMVh+2uCG5Pnsu8S4MtaJ8nSc5pKQctr7HmF7rdm2k0nCRLj 9Dvs2XtJWYDs+pCluxBBGC5egMafJjHo2Ivl2NHiVLVFNhEoOUVCD0mbth9nIoQv Uz1lZt/DENqZnGEUmu90w9vL7alJCceDmDEiet/Fb9r0yQXcXDmBktwOJdQjvNOB pOSuVlK+zO6axusNxOcIW47GdzFKl8USb5AJG9B6zw+KPaC6k5zam1k1NEegt9Mn d3O2zeUHEIaeO4IRemxRmPMW6haYDawW7sKUEWABVVBo1tgn0o0k1VvBNDLsO4xl nwQ617+D9xFgLWtJKyFWaLI9aT12QxA0VmBwouwyjnsN7XeU2VkFJPfQL+CUVfF/ X8MO06QQtGnfjFcvzM2QKUw24iF0Tc+vra1B8SEKVuRmX0QNvyze03lHhIsSsOql OYIW79uMMm6y8JUI3oidCdNHivHp264Ay49WDuHPZkgelIe3z1CKhj51LwfuccXc TgaDu6uCDoe2SLepRh1dZowF2v7OicLhUCxmn9loMA+f8ZeIV1edo+e/sXo/sCN6 1y9QYTKotE+zY91zM83zlstBw5E6FLEIURugOtimB9LlykSm+GafKMtr5Z91P2aU FFIcO6tw270I1jO+FQ9Y =M2Na -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Fwd: Tomcat 6.0.26 on Windows server - sessions are not expired
Hi, We are facing issues where the sessions are not being expired and eventually causing a stack overflow. We see that at some point the sessions get pilled up and we need to manually expire those sessions through the manager application, or until we will restart tomcat but after a few hours / days, sessions will start to get stuck again We see it in all the applications that are deployed on tomcat, including the manager application - this rules out (in my opinion) the possibility that its an issue with our application. Please advise. Thanks, Maor.
Re: Tomcat 6.0.26 on Windows server - sessions are not expired
2014-02-02 Maor Yosef maoryo...@gmail.com: Hi, 1. 6.0.26 is old. We are facing issues where the sessions are not being expired and eventually causing a stack overflow. 2. Non-expiring sessions may cause OOM, but they cannot cause a stack overflow. What are your evidences? We see that at some point the sessions get pilled up and we need to manually expire those sessions through the manager application, or until we will restart tomcat but after a few hours / days, sessions will start to get stuck again 3. Increasing count of session does not mean that sessions do not expire. Your evidence = ? We see it in all the applications that are deployed on tomcat, including the manager application - this rules out (in my opinion) the possibility that its an issue with our application. 4. Sessions are expired by a background thread. If the thread is stuck somewhere (e.g. doing auto-deployment work) it will affect expiration of sessions. Your thread dump = ? By default there is one background thread that is shared by all container levels in Tomcat, but you can configure a separate one in each container (container = Context, Host or Engine). 5. Web bots that do not support cookies may generate multiple sessions. See CrawlerSessionManagerValve Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
