Hi all!
I'm running a webapp and it's successfully clustered in production using JBoss.
First of all, that's awesome and thanks to all who contributed to building
this tool.
We just have some issues with Tomcat and/or JBoss restarts. If we hot deploy a
WAR file into JBoss or if we throw some new code into Tomcat's classes
directory (on a dev server) that triggers an application restart we end up some
issues.
First, we get the INITIAL ERROR included below. I'm not a master of my MBEAN
universe yet. In fact, I've on Tomcat for years and having moved to JBoss am
very new to the concept. Any interpretation of what's happening would be
appreciated.
Next, we get the REPEATING ERROR which looks like a JGroups process gets stuck
on while the rest of the cluster is off. This makes some sense in light of the
initial error but I have no idea how to remedy the problem.
In addition to JBoss's built-in caches we're running a couple TreeCache
instances in the app. One for Hibernate which we set up and configure... we
don't use JBoss's built-in. Both configs included below. One for other stuff
we want to cache. mcast_addr is the same for both but mcast_port is different.
By simply restarting Tomcat (the full service, not just the application) or
JBoss the problem goes away. Seems like a stuck thread or something that
doesn't get properly shut down when the app restarts.
Question: Is there anything special we should be doing when the application
restarts to prevent this sort of thing? We have a ServletContextListener that
implements contextDestroyed(ServletContextEvent cse) and could easily do some
better/more graceful JGroups or MBEAN shutdown.
Apologies if this is a Hibernate-specific issue. The error appears to be from
org.jgroups.protocols.UDP so I thought I'd ping you guys and girls first.
Thanks,
Joe
INITIAL ERROR=================
124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN
org.jboss.cache.TreeCache - Error in stop
jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException:
jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown
Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown
Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke (Unknown Source)
at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:189)
at org.hibernate.cache.TreeCacheProvider.stop(TreeCacheProvider.java:84)
at org.hibernate.impl.SessionFactoryImpl.close (SessionFactoryImpl.java:800)
at
com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
at
com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:95)
at
org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload
(StandardContext.java:2947)
at
org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java
:1557)
at java.lang.Thread.run(Unknown Source)
124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN
org.jboss.cache.TreeCache - Error in destroy
jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException:
jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown
Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke (Unknown
Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at
org.jboss.system.ServiceMBeanSupport.destroy(ServiceMBeanSupport.java:204)
at org.hibernate.cache.TreeCacheProvider.stop (TreeCacheProvider.java:85)
at org.hibernate.impl.SessionFactoryImpl.close(SessionFactoryImpl.java:800)
at
com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158)
at com.fbdblog.startup.ApplicationStartup.contextDestroyed
(ApplicationStartup.java:95)
at
org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at org.apache.catalina.core.StandardContext.reload
(StandardContext.java:2947)
at
org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java
:1557)
at java.lang.Thread.run(Unknown Source)
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster
,treecache-interceptor=UnlockInterceptor
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster
,treecache-interceptor=TxInterceptor
124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CacheMgmtInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=ReplicationInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=PessimisticLockInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CallInterceptor
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO
com.fbdblog.startup.ApplicationStartup - Unregistering MBean:
jboss.cache:service=Fbdblog-TreeCache-Cluster
124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN
org.jboss.cache.TreeCache - Unexpected error during removal.
jboss.cache:service=Fbdblog-TreeCache-Cluster
javax.management.InstanceNotFoundException:
jboss.system:service=ServiceController
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean (Unknown
Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown
Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at org.jboss.system.ServiceMBeanSupport.postDeregister
(ServiceMBeanSupport.java:409)
at com.sun.jmx.mbeanserver.MBeanSupport.postDeregister(Unknown Source)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postDeregisterInvoke(Unknown
Source)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean
(Unknown Source)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(Unknown
Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(Unknown Source)
at com.fbdblog.startup.ApplicationStartup.shutdownCacheMBean
(ApplicationStartup.java:156)
at
com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:98)
at
org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306)
at
org.apache.catalina.core.StandardContext.reload(StandardContext.java:2947)
at org.apache.catalina.loader.WebappLoader.backgroundProcess
(WebappLoader.java:403)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1568)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java
:1577)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1557)
at java.lang.Thread.run(Unknown Source)
Fbdblog: Application shut down! (XPS)
REPEATING ERROR===============
49174016 [Incoming Thread,Fbdblog-TreeCache-Cluster, 192.168.1.103:1852] ERROR
org.jgroups.protocols.UDP - failed handling incoming message
java.lang.NoSuchMethodError: java.lang.ThreadDeath: method
(Ljava/lang/String;)V not found
at org.jgroups.protocols.UNICAST.handleDataReceived (UNICAST.java:539)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:246)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:596)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:154)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:298)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:145)
at org.jgroups.protocols.Discovery.up(Discovery.java:220)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage (TP.java:1542)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1496)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source)
at java.lang.Thread.run(Unknown Source)
CONFIG================
jboss:service=Naming
jboss:service=TransactionManager
org.jboss.cache.JBossTransactionManagerLookup
NONE
REPL_SYNC
true
1000
0
Fbdblog-TreeCache-Cluster
<UDP mcast_addr="228.1.2.3" mcast_port="41332"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
loopback="true"/>
<PING timeout="2000" num_initial_members="3" up_thread="false"
down_thread="false"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<!-- <FD shun="true" up_thread="true" down_thread="true" />-->
<FD_SOCK/>
<VERIFY_SUSPECT timeout="1500" up_thread="false"
down_thread="false"/>
<pbcast.NAKACK gc_lag="50"
retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false"
down_thread="false"/>
<UNICAST timeout="600,1200,2400" window_size="100"
min_threshold="10" down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="20000" up_thread="false"
down_thread="false"/>
<FRAG frag_size="8192" down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="60000" join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
true
60000
15000
10000
org.jboss.cache.eviction.LRUPolicy
5
50000
0
false
CONFIG2====================
jboss:service=Naming
jboss:service=TransactionManager
org.jboss.cache.JBossTransactionManagerLookup
NONE
REPL_SYNC
true
1000
0
Fbdblog-TreeCache-Cluster-Nondao
<UDP mcast_addr=" 228.1.2.3" mcast_port="41333"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
loopback="true"/>
<PING timeout="2000" num_initial_members="3" up_thread="false"
down_thread="false"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<!-- <FD shun="true" up_thread="true" down_thread="true" />-->
<FD_SOCK/>
<VERIFY_SUSPECT timeout="1500" up_thread="false"
down_thread="false"/>
<pbcast.NAKACK gc_lag="50"
retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false"
down_thread="false"/>
<UNICAST timeout="600,1200,2400" window_size="100"
min_threshold="10" down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="20000" up_thread="false"
down_thread="false"/>
<FRAG frag_size="8192" down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="60000" join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
true
60000
15000
10000
org.jboss.cache.eviction.LRUPolicy
5
50000
0
false
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4089136#4089136
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4089136
_______________________________________________
jboss-user mailing list
[email protected]
https://lists.jboss.org/mailman/listinfo/jboss-user
