Re: Tomcat 6 does not start in a cluster. It simply hangs... Any Idea..
HI Filip:
I did three dumps. It is hanging on the same class. Please let me know any
ideas...
We are in production. I have same setup on development and it works.
However, cluster setup on development is on the same box with apache web
server in front.
In production there are three servers with hardware load balancer in the the
front.
Thanks
Jaspal
jsandhu2204 wrote:
Please see the three stack dumps I did.
http://www.nabble.com/file/p21367357/Thread-dump-three-times.txt
Thread-dump-three-times.txt
Filip Hanik - Dev Lists wrote:
Could you take more than one thread dump, take 3, wait 5 seconds between
each.
As you can see, the startup thread is still executing your application
code com.valleyoak.util.IVosTableInfo, this is the thread that starts up
tomcat
Filip
main thread - the one starting tomcat
main prio=6 tid=0x01dc1c00 nid=0x15cc in Object.wait()
[0x01f4d000..0x01f4fc70]
java.lang.Thread.State: RUNNABLE
at
com.valleyoak.util.IVosTableInfo.findColumnTables(IVosTableInfo.java:161)
at com.valleyoak.db.IVosStorable.findColumnTables(IVosStorable.java:1245)
at com.valleyoak.model.Payment.clinit(Payment.java:136)
at
com.valleyoak.model.ContextListener.contextInitialized(ContextListener.java:107)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4342)
- locked 0x41f4f390 (a
org.apache.catalina.core.StandardContext)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
- locked 0x41eef8b8 (a java.util.HashMap)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:926)
at
org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:889)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
One of the cluster threads have received the session state data from the
other node and is deserializing it
pool-2-thread-1 prio=6 tid=0x01dc5c00 nid=0xf00 in
Object.wait() [0x777c000..0x0777fae0]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at com.valleyoak.util.IVosTableInfo.class$(IVosTableInfo.java:504)
at com.valleyoak.util.IVosTableInfo.loadTables(IVosTableInfo.java:522)
at com.valleyoak.util.IVosTableInfo.clinit(IVosTableInfo.java:53)
at com.valleyoak.db.IVosStorable.findColumnTables(IVosStorable.java:1245)
at com.valleyoak.model.User.clinit(User.java:71)
at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
at
java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1696)
at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:205)
at java.security.AccessController.doPrivileged(Native Method)
at
java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:202)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:558)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
jsandhu2204 wrote:
I have created stack trace when servers hangs. Please see attached. I
would
appreciate if you can help me with this.
If I put distributable/ flag in the web.xml of the application.
Tomcat6
does not start sometimes. It simply hangs.
http://www.nabble.com/file/p21347289/CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf
CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf
http://www.nabble.com/file/p21347289/CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf
CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf
Filip Hanik - Dev Lists wrote:
When this happens, simply take a thread dump, and we can see why it is
hanging
Unix:
kill -3 pid
jstack -l pid
Windows
Ctrl+Break in the console
jstack -l pid
Filip
jsandhu2204 wrote:
Problem1 : While starting tomcat6, sometimes it starts other times it
simply
hangs on the following message:
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34
AM
received in 110 ms.
Tomcat Log:
INFO: Register manager /ivos to cluster element Engine with name
Catalina
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /ivos Dec 12, 2008 12:34:53 AM
org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [localhost#/ivos], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 88, 8,
28}:4002,{10, 88, 8, 28},4002, alive=123501
Re: Tomcat 6 does not start in a cluster. It simply hangs... Any Idea..
Thanks Filip. I am going to ask the vendor. Filip Hanik - Dev Lists wrote: hi Jaspal, the problem seems to be in the com.valleyoak package. The main thread is stuck at at com.valleyoak.model.Payment.clinit(Payment.java:136) (at com.valleyoak.util.IVosTableInfo.findColumnTables(IVosTableInfo.java:161)) - last line the cluster thread is stuck at at java.lang.Class.forName0(Native Method) because the main thread is stuck during a clinit (this is class init, happens directly after class loading, to do static initializers), the cluster thread can't continue. what you need to investigate is why at com.valleyoak.util.IVosTableInfo.findColumnTables(IVosTableInfo.java:161)) - last line is stuck, but it does look like its a problem with your code. Filip jsandhu2204 wrote: HI Filip: I did three dumps. It is hanging on the same class. Please let me know any ideas... We are in production. I have same setup on development and it works. However, cluster setup on development is on the same box with apache web server in front. In production there are three servers with hardware load balancer in the the front. Thanks Jaspal jsandhu2204 wrote: Please see the three stack dumps I did. http://www.nabble.com/file/p21367357/Thread-dump-three-times.txt Thread-dump-three-times.txt Filip Hanik - Dev Lists wrote: Could you take more than one thread dump, take 3, wait 5 seconds between each. As you can see, the startup thread is still executing your application code com.valleyoak.util.IVosTableInfo, this is the thread that starts up tomcat Filip main thread - the one starting tomcat main prio=6 tid=0x01dc1c00 nid=0x15cc in Object.wait() [0x01f4d000..0x01f4fc70] java.lang.Thread.State: RUNNABLE at com.valleyoak.util.IVosTableInfo.findColumnTables(IVosTableInfo.java:161) at com.valleyoak.db.IVosStorable.findColumnTables(IVosStorable.java:1245) at com.valleyoak.model.Payment.clinit(Payment.java:136) at com.valleyoak.model.ContextListener.contextInitialized(ContextListener.java:107) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4342) - locked 0x41f4f390 (a org.apache.catalina.core.StandardContext) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) - locked 0x41eef8b8 (a java.util.HashMap) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:926) at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:889) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492) One of the cluster threads have received the session state data from the other node and is deserializing it pool-2-thread-1 prio=6 tid=0x01dc5c00 nid=0xf00 in Object.wait() [0x777c000..0x0777fae0] java.lang.Thread.State: RUNNABLE at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) at com.valleyoak.util.IVosTableInfo.class$(IVosTableInfo.java:504) at com.valleyoak.util.IVosTableInfo.loadTables(IVosTableInfo.java:522) at com.valleyoak.util.IVosTableInfo.clinit(IVosTableInfo.java:53) at com.valleyoak.db.IVosStorable.findColumnTables(IVosStorable.java:1245) at com.valleyoak.model.User.clinit(User.java:71) at java.io.ObjectStreamClass.hasStaticInitializer(Native Method) at java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1696) at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52) at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:205) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:202) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:558) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732) jsandhu2204 wrote: I have created stack trace when servers hangs. Please see attached. I would appreciate if you can help me with this. If I put distributable/ flag in the web.xml of the application. Tomcat6 does not start sometimes. It simply hangs. http://www.nabble.com/file/p21347289/CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf http://www.nabble.com/file/p21347289/CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf Filip Hanik - Dev Lists wrote: When this happens, simply take a thread
Re: Tomcat 6 does not start in a cluster. It simply hangs... Any Idea..
Please see the three stack dumps I did.
http://www.nabble.com/file/p21367357/Thread-dump-three-times.txt
Thread-dump-three-times.txt
Filip Hanik - Dev Lists wrote:
Could you take more than one thread dump, take 3, wait 5 seconds between
each.
As you can see, the startup thread is still executing your application
code com.valleyoak.util.IVosTableInfo, this is the thread that starts up
tomcat
Filip
main thread - the one starting tomcat
main prio=6 tid=0x01dc1c00 nid=0x15cc in Object.wait()
[0x01f4d000..0x01f4fc70]
java.lang.Thread.State: RUNNABLE
at
com.valleyoak.util.IVosTableInfo.findColumnTables(IVosTableInfo.java:161)
at com.valleyoak.db.IVosStorable.findColumnTables(IVosStorable.java:1245)
at com.valleyoak.model.Payment.clinit(Payment.java:136)
at
com.valleyoak.model.ContextListener.contextInitialized(ContextListener.java:107)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3843)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4342)
- locked 0x41f4f390 (a org.apache.catalina.core.StandardContext)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
- locked 0x41eef8b8 (a java.util.HashMap)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525)
at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:926)
at
org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:889)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
One of the cluster threads have received the session state data from the
other node and is deserializing it
pool-2-thread-1 prio=6 tid=0x01dc5c00 nid=0xf00 in
Object.wait() [0x777c000..0x0777fae0]
java.lang.Thread.State: RUNNABLE
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at com.valleyoak.util.IVosTableInfo.class$(IVosTableInfo.java:504)
at com.valleyoak.util.IVosTableInfo.loadTables(IVosTableInfo.java:522)
at com.valleyoak.util.IVosTableInfo.clinit(IVosTableInfo.java:53)
at com.valleyoak.db.IVosStorable.findColumnTables(IVosStorable.java:1245)
at com.valleyoak.model.User.clinit(User.java:71)
at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
at
java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1696)
at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:205)
at java.security.AccessController.doPrivileged(Native Method)
at
java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:202)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:558)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
jsandhu2204 wrote:
I have created stack trace when servers hangs. Please see attached. I
would
appreciate if you can help me with this.
If I put distributable/ flag in the web.xml of the application. Tomcat6
does not start sometimes. It simply hangs.
http://www.nabble.com/file/p21347289/CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf
CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf
http://www.nabble.com/file/p21347289/CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf
CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf
Filip Hanik - Dev Lists wrote:
When this happens, simply take a thread dump, and we can see why it is
hanging
Unix:
kill -3 pid
jstack -l pid
Windows
Ctrl+Break in the console
jstack -l pid
Filip
jsandhu2204 wrote:
Problem1 : While starting tomcat6, sometimes it starts other times it
simply
hangs on the following message:
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34
AM
received in 110 ms.
Tomcat Log:
INFO: Register manager /ivos to cluster element Engine with name
Catalina
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /ivos Dec 12, 2008 12:34:53 AM
org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [localhost#/ivos], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 88, 8,
28}:4002,{10, 88, 8, 28},4002, alive=123501,id={27 -68 111 -64 -49 -117
68
38 -98 -27 -49 39 -22 6 -88 -38 }, payload={}, command={}, domain={},
].
This operation will timeout if no session state has been received
within
60
seconds.
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34
AM
received in 110 ms
Re: Tomcat 6 does not start in a cluster. It simply hangs... Any Idea..
I have created stack trace when servers hangs. Please see attached. I would
appreciate if you can help me with this.
If I put distributable/ flag in the web.xml of the application. Tomcat6
does not start sometimes. It simply hangs.
http://www.nabble.com/file/p21347289/CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf
CheatSheet-ltr1vs02-Tomcat6-hangs-if-put-distributeable-tag.rtf
http://www.nabble.com/file/p21347289/CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf
CheatSheet-Tomcat6-hangs-if-put-distributeable-tag.rtf
Filip Hanik - Dev Lists wrote:
When this happens, simply take a thread dump, and we can see why it is
hanging
Unix:
kill -3 pid
jstack -l pid
Windows
Ctrl+Break in the console
jstack -l pid
Filip
jsandhu2204 wrote:
Problem1 : While starting tomcat6, sometimes it starts other times it
simply
hangs on the following message:
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34 AM
received in 110 ms.
Tomcat Log:
INFO: Register manager /ivos to cluster element Engine with name Catalina
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager
start
INFO: Starting clustering manager at /ivos Dec 12, 2008 12:34:53 AM
org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [localhost#/ivos], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 88, 8,
28}:4002,{10, 88, 8, 28},4002, alive=123501,id={27 -68 111 -64 -49 -117
68
38 -98 -27 -49 39 -22 6 -88 -38 }, payload={}, command={}, domain={}, ].
This operation will timeout if no session state has been received within
60
seconds.
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34 AM
received in 110 ms.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
--
View this message in context:
http://www.nabble.com/Tomcat-6-does-not-start-in-a-cluster.--It-simply-hangs...-Any-Idea..-tp21130862p21347289.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 6 does not start in a cluster. It simply hangs... Any Idea..
Problem1 : While starting tomcat6, sometimes it starts other times it simply
hangs on the following message:
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34 AM
received in 110 ms.
Tomcat Log:
INFO: Register manager /ivos to cluster element Engine with name Catalina
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /ivos Dec 12, 2008 12:34:53 AM
org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [localhost#/ivos], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 88, 8,
28}:4002,{10, 88, 8, 28},4002, alive=123501,id={27 -68 111 -64 -49 -117 68
38 -98 -27 -49 39 -22 6 -88 -38 }, payload={}, command={}, domain={}, ].
This operation will timeout if no session state has been received within 60
seconds.
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34 AM
received in 110 ms.
--
View this message in context:
http://www.nabble.com/Tomcat-6-does-not-start-in-a-cluster.--It-simply-hangs...-Any-Idea..-tp21130862p21130862.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 6 Cluster Setup- Server1 simply hangs sometimes - won't start...
We installed.tomcat6 in production on three high ended machines
Tomcat Version JVM Version JVM Vendor OS Name OS Version OS Architecture
Apache Tomcat/6.0.18 1.6.0_07-b06 Sun Microsystems Inc. Windows 2003 5.2
amd64
1) Cluster is setup correctly in the server.xml as follows
2) web.xml has distributable/ flag
Problem1 : While starting tomcat6, sometimes it starts other times it
simply hangs on the following message:
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34 AM
received in 110 ms.
Tomcat Log:
INFO: Register manager /ivos to cluster element Engine with name Catalina
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /ivos Dec 12, 2008 12:34:53 AM
org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [localhost#/ivos], requesting session state from
org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 88, 8,
28}:4002,{10, 88, 8, 28},4002, alive=123501,id={27 -68 111 -64 -49 -117 68
38 -98 -27 -49 39 -22 6 -88 -38 }, payload={}, command={}, domain={}, ].
This operation will timeout if no session state has been received within 60
seconds.
Dec 12, 2008 12:34:53 AM org.apache.catalina.ha.session.DeltaManager
waitForSendAllSessions
INFO: Manager [localhost#/ivos]; session state send at 12/12/08 12:34 AM
received in 110 ms.
After this nothing happens simply hangs. Should I use
tcpHostListener=MYIPAddress or auto May be auto is slowing it down?
Problem2: User does not transparently fail-over to node2 if the users fails
on node1.
However, the session count on all the nodes in cluster has correct number.
Means replication does work.
I have attached server.xml
Here is multicast information
Local Address Foreign Address
TCP10.88.8.27:40010.0.0.0:0 LISTENING
TCP10.88.8.27:400110.88.8.28:1993ESTABLISHED
TCP10.88.8.27:400110.88.8.28:2682ESTABLISHED
TCP10.88.8.27:400110.88.8.30:3532ESTABLISHED
TCP10.88.8.27:400110.88.8.30:4292ESTABLISHED
TCP10.88.8.27:424510.88.2.204:1433 ESTABLISHED
TCP10.88.8.27:425010.88.8.28:4002ESTABLISHED
TCP10.88.8.27:425210.88.8.28:4002ESTABLISHED
TCP10.88.8.27:425410.88.8.30:4003ESTABLISHED
TCP10.88.8.27:425510.88.8.30:4003ESTABLISHED
http://www.nabble.com/file/p20985367/server.xml server.xml
--
View this message in context:
http://www.nabble.com/Tomcat-6-Cluster-Setup--Server1-simply-hangs-sometimes---won%27t-start...-tp20985367p20985367.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
