Hi Chris,
Thanks for the help. I don't see any OOME around this time.
Before restarting I collected the heap and thread dumps. Looking at one of
the thread dumps, I see the ContainerBackgroundProcessor (below). So looks
like that thread didn't die entirely, may be it just stopped working ?
*"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10
tid=0x7fd000a5d800 nid=0x9be5 waiting on condition [0x7fcfc4955000]*
* java.lang.Thread.State: TIMED_WAITING (sleeping)*
* at java.lang.Thread.sleep(Native Method)*
* at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1508)*
* at java.lang.Thread.run(Thread.java:745)*
* Locked ownable synchronizers:*
* - None*
- I have attached a screenshot of the StandardManager runtime attributes
from the heapdump. Is there anything anyone can see wrong here?
- Is there anything else I can look for in the heapdump and threaddump?
[image: Inline image 1]
thanks,
Stephen
On Fri, Jun 3, 2016 at 5:57 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Stephen,
>
> On 6/3/16 8:47 PM, Stephen Bhadran wrote:
> > *Linux version:* 2.6.32-279.5.2.el6.x86_64 (
> > mockbu...@c6b10.bsys.dev.centos.org) (gcc version 4.4.6 20120305
> > (Red Hat 4.4.6-4) (GCC) ) *Java Version: *1.7.0_55 *Tomcat
> > Version: *7.0.40 *Session Timeout: *30 minutes
> >
> > Hi, Two of our servers reached max heap and we had to restart to
> > resolve the issue.
> >
> > In troubleshooting, we noticed Http Sessions had stopped expiring
> > 05/12th and reached 25K over a period of 11 days. I noticed the
> > same behavior in both servers, the session expiration seem to have
> > stopped on both of them around the same time. I'm saying this
> > because from looking at the monitoring graphs (NewRelic) I notice
> > the http sessions starting to climb on both servers around the same
> > time frame.
> >
> > I am thinking, the issue could be two folds: #1 - The session
> > expiration stopped working -OR #2 - The sessions themselves were
> > created with TTLs way into the future
> >
> > I suspected #1. I looked that tomcat 7.0.40 source code, extracted
> > all session/manager error messages and searched for them in the
> > logs and didn't find any hits. I was hoping to find something about
> > "Session expiration stopped working", but didn't find any.
> >
> > Can you please advise how to troubleshoot this issue? I'd very
> > much appreciate the help.
>
> Were there any errors around the 12th? I've found that in rare cases,
> the server hits an OOME while the BackgroundProcessor thread is
> running, which kills the BackgroundProcessor.
>
> Guess which thread is responsible for triggering the session-cleaner?
>
> If the BackgroundProcessor died, it should have dumped an error to a
> log file. As I said, it's pretty rare, but it does happen and then
> it's only a matter of time before your sessions bust your heap.
>
> You can manually-trigger the session-cleaner via JMX of you absolutely
> have to, but I'd personally prefer to bounce a Tomcat instance in that
> case.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJXUid5AAoJEBzwKT+lPKRYJfYQAKA2ditjB7B2vLbm0po83Tr0
> YD2EjqLBrF4O/3Wu6enIDRAQEIGgXmTr2bgPlMvu+6aoDopiCSKltRxWQFlTMyIP
> 8ura/0YlS5sX3yGPPKN5nrZnOz2dvrgPncHLNVfuM9BEm+8QZPeKNZVDhMH53m1v
> 4YcLCM5IjswAS4FrVeG9A0gcZplBd53WUuypnRYQMUcS1sACdItoZ3TLAwb/WBg/
> T9yoyftEXJVtP6lovkaWka3yw3v382P6HmSV6/TtCWjogB/tBdCsEPmGEQ0j7k8y
> 9dLmmTBawDdckaxiGf1M1M/5R+f7XqU4e02Dnhk9v4umysmu5EH+VQFO4kfi5qX+
> DvaoS13lluyN1LGMyuwHkYFHXxlgZhR/XAW9Cn/0D/NDCQks6lHG9Sby+T/kvLSO
> YNSBV78SX9wsJ/fH9MxrcTarxyUYsxnHdNkcarkmcrQIkhda/vBXQobXpYVL9poR
> HmxNGNOujCTojqPJzlpgAaUVUq6pEUcN3u1vkCqXNQGVo983PHz2JhfEBGYihKBk
> icbuFxfIy2Dv6KwuPPMV9Y7nVRNVBrVEpieZuAc7Sj9/U3kUgPUznUSbRFUCnGd1
> YI/kq9rn/qj/aqhaJdMF0Ci4+ieJMorKNUE1PWOjbnya8XoyBxXfPd5iV6n7elEw
> V6qNyO6n46isrAiVEu5c
> =MyIX
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
