Re: Tomcat hangs when reloading context if user is logged in

[James Murphy]

Chris,

Thanks for all you help :) . We tried commenting out that method (just 
to see if it was the root of the problem) and Tomcat behaved like it is 
supposed to! Now we can work on implementing a work-around for the 
functionality.


Thanks!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat hangs when reloading context if user is logged in

[James Murphy]

Chris,

Thanks for the response. I went ahead and got the info you requested.

Christopher Schultz wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

James,

On 10/30/2009 9:05 AM, James Murphy wrote:
  

Some background info:
OS: Windows Server 2003
Tomcat Version: 5.5
JDK Version: 1.6.0

I've run into a very annoying problem while trying to enable
automatic context reload on our development server. It works well
when there is not a user logged on to the system (I guess that means
while there is session info on the server), but if someone is logged
in, then Tomcat fails to reload the context.



I can confirm that Tomcat 5.5.26 both reloads and stops properly when
multiple sessions are active. Persist-to-disk works as expected, and
sessions are retained across the restart/reload.

I suspect there is a problem with your specific webapp and/or setup.
  

My thoughts too, I'm just not sure where to look.
  

Another issue that may be related to this is that Tomcat will not
shutdown cleanly when someone is logged in (Tomcat hangs at INFO:
Stopping service Catalina). Rather, we must close the console window
forcefully, then start the server back up to an error (IOException
while loading persisted sessions: java.io.EOFException)



That's interesting: your session (de)serialization is failing because
the file is incomplete. I suspect this is because the sessions are not
correctly written to the file before application stop. Perhaps session
serialization is the actual problem.

Can you attempt to shut down Tomcat and then check:
1. A few thread dumps
(http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F)
  

I put them at the bottom of the message.

2. The CPU usage of your webapp and/or threads
  
The CPU usage jumped to about 05% right before it hung, and then fell to 
00% and stayed there. Memory usage held steady at 37K.

I'm wondering if serialization is either running-away (infinite loop,
etc.) or stalling (waiting for access to a locked object) and it's never
completing.

  
Looks to me like it's waiting for a locked object, like you said, rather 
than hitting an infinite loop.

I used jstack to get a stack trace on the Tomcat process when it hung
on context reload. I won't post the details unless someone thinks
they might be helpful (Don't want to fill the email with too much
stuff).



I think they might be. Go ahead and post the thread dump.
  

See the bottom of the message.
  

However, I know I read somewhere that Tomcat will hang on shutdown
if there is a non-daemon thread still running.



While this is true, it shouldn't block a webapp from reloading (it will
leave the old webapp in memory, so the old webapp won't really go away
completely, but the new webapp should load in its place and handle
requests).

  

jstack reports three non-daemon threads: main, VM Thread, and VM
Periodic Task Thread. I don't think that's causing the problem, but
figured I'd better let you know anyway.



Nope, those are normal.
  


Here are a couple of thread dumps of tomcat, the first taken about 5 
seconds after the hang, and the second a couple of minutes later, while 
still hung.
Note that in the main methods, there is one stack frame with some XX's 
in it. I put those there to cover the name of a method which could 
identify our company. Nothing personal, I'm just not allowed to let that 
information leak out.
After seeing that stack frame, however, I looked into the method which 
it referenced. It appears to be sending a HTTP POST message to a servlet 
which we have running, which is supposed to log the user out when it 
recieves said message. Could the problem be that the servlet is shut 
down before the message reaches it?


--
--
--

2009-11-02 13:10:11
Full thread dump Java HotSpot(TM) Client VM (1.6.0-b105 mixed mode, 
sharing):


http-80-6 daemon prio=6 tid=0x0322a400 nid=0xb10 in Object.wait() 
[0x03d5f000..0x03d5fb18]

  java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x181530c0 (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)

   at java.lang.Object.wait(Object.java:485)
   at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)

   - locked 0x181530c0 (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
   at 
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)

   at java.lang.Thread.run(Thread.java:619)

http-80-5 daemon prio=6 tid=0x02f89400 nid=0xe2c in Object.wait() 
[0x03cbf000..0x03cbfc18]

  java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0x18153148 (a 
org.apache.tomcat.util.net.JIoEndpoint$Worker)

   at 

Re: Tomcat hangs when reloading context if user is logged in

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

James,

On 11/2/2009 1:32 PM, James Murphy wrote:
 Here are a couple of thread dumps of tomcat, the first taken about 5
 seconds after the hang, and the second a couple of minutes later, while
 still hung.

Okay.

 Note that in the main methods, there is one stack frame with some XX's
 in it. I put those there to cover the name of a method which could
 identify our company. Nothing personal, I'm just not allowed to let that
 information leak out.

No problem: this is always good practice. Thanks for mentioning it,
though :)

 After seeing that stack frame, however, I looked into the method which
 it referenced. It appears to be sending a HTTP POST message to a servlet
 which we have running, which is supposed to log the user out when it
 recieves said message. Could the problem be that the servlet is shut
 down before the message reaches it?


Most of these all look the same. That is, they are just waiting for
requests to come in:

 http-80-6 daemon prio=6 tid=0x0322a400 nid=0xb10 in Object.wait()
 [0x03d5f000..0x03d5fb18]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0x181530c0 (a

[snip]

 main prio=6 tid=0x00297000 nid=0xb18 runnable [0x0088f000..0x0088fe58]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked 0x1698bc98 (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:632)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
at
 sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
 
- locked 0x16981bd0 (a sun.net.www.protocol.http.HttpURLConnection)
at com.oreilly.servlet.HttpMessage.sendPostMessage(HttpMessage.java:141)
at
 com.XXX.bean.SessionListener.valueUnbound(SessionListener.java:44)
 
at
 org.apache.catalina.session.StandardSession.removeAttributeInternal(StandardSession.java:1654)
 
at
 org.apache.catalina.session.StandardSession.writeObject(StandardSession.java:1507)
 
at
 org.apache.catalina.session.StandardSession.writeObjectData(StandardSession.java:959)
 
at
 org.apache.catalina.session.StandardManager.doUnload(StandardManager.java:517)

It looks like the StandardManager is trying to persist your sessions,
which is proper behavior (unless you have disabled it). You have a
listener that, as you mentioned, is making an outgoing HTTP connection.

Is it connecting to another webapp, or to the same webapp? If you're
trying to connect to the same webapp, then it's possible that Tomcat has
paused all your Connectors, and they aren't accepting any connections.
In that case, you'll wait for a long time (and possibly repeatedly
attempt connections) before your POST fails.

If the above describes your situation, may I recommend that, instead of
making an HTTP POST back to your own webapp, you instead make a regular
Java method call to process this logout stuff? It shouldn't be that
tough to refactor your code to allow your logout logic to be called
either from an HTTP request or from an explicit method call.

Your second thread dump shows the main thread stuck in the same place:

 main prio=6 tid=0x00297000 nid=0xb18 runnable [0x0088f000..0x0088fe58]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked 0x1698bc98 (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:632)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:577)
at
 sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1004)
 
- locked 0x16981bd0 (a sun.net.www.protocol.http.HttpURLConnection)
at com.oreilly.servlet.HttpMessage.sendPostMessage(HttpMessage.java:141)
at
 com.XXX.bean.SessionListener.valueUnbound(SessionListener.java:44)

Yup: still waiting.

Try calling your logout logic without using an HTTP POST and I think
you'll find that things start working more reasonably for you.

Good luck,
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrvLKAACgkQ9CaO5/Lv0PCuZwCgrpXZFpDQKvWcHXqy3eqw/Ra2
sAIAnjnhp5a8eeK/vf3sBsTtCeJg3hCZ
=dILD
-END PGP SIGNATURE-

Tomcat hangs when reloading context if user is logged in

[James Murphy]

Hi all,

Some background info:
OS: Windows Server 2003
Tomcat Version: 5.5
JDK Version: 1.6.0

I've run into a very annoying problem while trying to enable automatic 
context reload on our development server. It works well when there is 
not a user logged on to the system (I guess that means while there is 
session info on the server), but if someone is logged in, then Tomcat 
fails to reload the context. Another issue that may be related to this 
is that Tomcat will not shutdown cleanly when someone is logged in 
(Tomcat hangs at INFO: Stopping service Catalina). Rather, we must 
close the console window forcefully, then start the server back up to an 
error (IOException while loading persisted sessions: 
java.io.EOFException), shut it back down, and restart it. This has 
become very tedious to do whenever we have to cycle the server, and has 
made context reloading impossible. After all the information I tried 
piecing together on the I guess it has something to with not being able 
to reload/shutdown if there is an open session, but I'm not sure


I tried upgrading Tomcat to 6.0.20, but the same issue occurs. If 
possible, I'd like to try fixing this on Tomcat 5.5, because I'm not 
100% sure that the migration went cleanly.


I used jstack to get a stack trace on the Tomcat process when it hung on 
context reload. I won't post the details unless someone thinks they 
might be helpful (Don't want to fill the email with too much stuff). 
However, I know I read somewhere that Tomcat will hang on shutdown if 
there is a non-daemon thread still running. jstack reports three 
non-daemon threads: main, VM Thread, and VM Periodic Task Thread. I 
don't think that's causing the problem, but figured I'd better let you 
know anyway.


I appreciate your help, as it has been very frustrating trying to 
resolve this issue.

-James Murphy


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat hangs when reloading context if user is logged in

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

James,

On 10/30/2009 9:05 AM, James Murphy wrote:
 Some background info:
 OS: Windows Server 2003
 Tomcat Version: 5.5
 JDK Version: 1.6.0
 
 I've run into a very annoying problem while trying to enable
 automatic context reload on our development server. It works well
 when there is not a user logged on to the system (I guess that means
 while there is session info on the server), but if someone is logged
 in, then Tomcat fails to reload the context.

I can confirm that Tomcat 5.5.26 both reloads and stops properly when
multiple sessions are active. Persist-to-disk works as expected, and
sessions are retained across the restart/reload.

I suspect there is a problem with your specific webapp and/or setup.

 Another issue that may be related to this is that Tomcat will not
 shutdown cleanly when someone is logged in (Tomcat hangs at INFO:
 Stopping service Catalina). Rather, we must close the console window
 forcefully, then start the server back up to an error (IOException
 while loading persisted sessions: java.io.EOFException)

That's interesting: your session (de)serialization is failing because
the file is incomplete. I suspect this is because the sessions are not
correctly written to the file before application stop. Perhaps session
serialization is the actual problem.

Can you attempt to shut down Tomcat and then check:
1. A few thread dumps
(http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F)

2. The CPU usage of your webapp and/or threads

I'm wondering if serialization is either running-away (infinite loop,
etc.) or stalling (waiting for access to a locked object) and it's never
completing.

 I used jstack to get a stack trace on the Tomcat process when it hung
 on context reload. I won't post the details unless someone thinks
 they might be helpful (Don't want to fill the email with too much
 stuff).

I think they might be. Go ahead and post the thread dump.

 However, I know I read somewhere that Tomcat will hang on shutdown
 if there is a non-daemon thread still running.

While this is true, it shouldn't block a webapp from reloading (it will
leave the old webapp in memory, so the old webapp won't really go away
completely, but the new webapp should load in its place and handle
requests).

 jstack reports three non-daemon threads: main, VM Thread, and VM
 Periodic Task Thread. I don't think that's causing the problem, but
 figured I'd better let you know anyway.

Nope, those are normal.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkrrE6IACgkQ9CaO5/Lv0PCseACgvojWegC03QeaM0Twjd4Ai/Wo
dxQAn0rqHwMGpu7rYj0Ihqsgndr854D4
=aj9o
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org