Hi all! I'm running a webapp and it's successfully clustered in production using JBoss. First of all, that's awesome and thanks to all who contributed to building this tool.
We just have some issues with Tomcat and/or JBoss restarts. If we hot deploy a WAR file into JBoss or if we throw some new code into Tomcat's classes directory (on a dev server) that triggers an application restart we end up some issues. First, we get the INITIAL ERROR included below. I'm not a master of my MBEAN universe yet. In fact, I've on Tomcat for years and having moved to JBoss am very new to the concept. Any interpretation of what's happening would be appreciated. Next, we get the REPEATING ERROR which looks like a JGroups process gets stuck on while the rest of the cluster is off. This makes some sense in light of the initial error but I have no idea how to remedy the problem. In addition to JBoss's built-in caches we're running a couple TreeCache instances in the app. One for Hibernate which we set up and configure... we don't use JBoss's built-in. Both configs included below. One for other stuff we want to cache. mcast_addr is the same for both but mcast_port is different. Bela Ban recommended changing the mbean name for each config. We played with that and the cluster worked perfectly until we deployed something and the app got restarted. By simply restarting Tomcat (the full service, not just the application) or JBoss the problem goes away. Seems like a stuck thread or something that doesn't get properly shut down when the app restarts. Question: Is there anything special we should be doing when the application restarts to prevent this sort of thing? We have a ServletContextListener that implements contextDestroyed(ServletContextEvent cse) and could easily do some better/more graceful JGroups or MBEAN shutdown. Apologies if this is a Hibernate-specific issue. The error appears to be from org.jgroups.protocols.UDP so I thought I'd ping you guys and girls first. Thanks, Joe INITIAL ERROR================= 124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Error in stop jboss.cache:service=Fbdblog-TreeCache-Cluster javax.management.InstanceNotFoundException: jboss.system:service=ServiceController at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke (Unknown Source) at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:189) at org.hibernate.cache.TreeCacheProvider.stop(TreeCacheProvider.java:84) at org.hibernate.impl.SessionFactoryImpl.close (SessionFactoryImpl.java:800) at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158) at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:95) at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733) at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306) at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947) at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java :1557) at java.lang.Thread.run(Unknown Source) 124125 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Error in destroy jboss.cache:service=Fbdblog-TreeCache-Cluster javax.management.InstanceNotFoundException: jboss.system:service=ServiceController at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke (Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) at org.jboss.system.ServiceMBeanSupport.destroy(ServiceMBeanSupport.java:204) at org.hibernate.cache.TreeCacheProvider.stop (TreeCacheProvider.java:85) at org.hibernate.impl.SessionFactoryImpl.close(SessionFactoryImpl.java:800) at com.fbdblog.dao.hibernate.HibernateUtil.killSessionFactory(HibernateUtil.java:158) at com.fbdblog.startup.ApplicationStartup.contextDestroyed (ApplicationStartup.java:95) at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733) at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306) at org.apache.catalina.core.StandardContext.reload (StandardContext.java:2947) at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java :1557) at java.lang.Thread.run(Unknown Source) 124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=UnlockInterceptor 124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster ,treecache-interceptor=TxInterceptor 124140 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CacheMgmtInterceptor 124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=ReplicationInterceptor 124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=PessimisticLockInterceptor 124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster,treecache-interceptor=CallInterceptor 124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] INFO com.fbdblog.startup.ApplicationStartup - Unregistering MBean: jboss.cache:service=Fbdblog-TreeCache-Cluster 124156 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN org.jboss.cache.TreeCache - Unexpected error during removal. jboss.cache:service=Fbdblog-TreeCache-Cluster javax.management.InstanceNotFoundException: jboss.system:service=ServiceController at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean (Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) at org.jboss.system.ServiceMBeanSupport.postDeregister (ServiceMBeanSupport.java:409) at com.sun.jmx.mbeanserver.MBeanSupport.postDeregister(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.postDeregisterInvoke(Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean (Unknown Source) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(Unknown Source) at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(Unknown Source) at com.fbdblog.startup.ApplicationStartup.shutdownCacheMBean (ApplicationStartup.java:156) at com.fbdblog.startup.ApplicationStartup.contextDestroyed(ApplicationStartup.java:98) at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:3733) at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4306) at org.apache.catalina.core.StandardContext.reload(StandardContext.java:2947) at org.apache.catalina.loader.WebappLoader.backgroundProcess (WebappLoader.java:403) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1568) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java :1577) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1557) at java.lang.Thread.run(Unknown Source) Fbdblog: Application shut down! (XPS) REPEATING ERROR=============== 49174016 [Incoming Thread,Fbdblog-TreeCache-Cluster, 192.168.1.103:1852] ERROR org.jgroups.protocols.UDP - failed handling incoming message java.lang.NoSuchMethodError: java.lang.ThreadDeath: method (Ljava/lang/String;)V not found at org.jgroups.protocols.UNICAST.handleDataReceived (UNICAST.java:539) at org.jgroups.protocols.UNICAST.up(UNICAST.java:246) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:596) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:154) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:298) at org.jgroups.protocols.MERGE2.up(MERGE2.java:145) at org.jgroups.protocols.Discovery.up(Discovery.java:220) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage (TP.java:1542) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1496) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run (Unknown Source) at java.lang.Thread.run(Unknown Source) CONFIG================ <server> | <classpath codebase="./lib" archives="jboss-cache.jar, jgroups-all.jar"/> | <mbean code="org.jboss.cache.TreeCache" | name="jboss.cache:service=TreeCache"> | <depends>jboss:service=Naming</depends> | <depends>jboss:service=TransactionManager</depends> | <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute> | <attribute name="IsolationLevel">NONE</attribute> | <attribute name="CacheMode">REPL_SYNC</attribute> | <attribute name="UseReplQueue">true</attribute> | <attribute name="ReplQueueInterval">1000</attribute> | <attribute name="ReplQueueMaxElements">0</attribute> | <attribute name="ClusterName">Fbdblog-TreeCache-Cluster</attribute> | <attribute name="ClusterConfig"> | <config> | <UDP mcast_addr="228.1.2.3" mcast_port="41332" | ip_ttl="64" ip_mcast="true" | mcast_send_buf_size="150000" mcast_recv_buf_size="80000" | ucast_send_buf_size="150000" ucast_recv_buf_size="80000" | loopback="true"/> | <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/> | <MERGE2 min_interval="10000" max_interval="20000"/> | <!-- <FD shun="true" up_thread="true" down_thread="true" />--> | <FD_SOCK/> | <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/> | <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false" down_thread="false"/> | <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/> | <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/> | <FRAG frag_size="8192" down_thread="false" up_thread="false"/> | <pbcast.GMS join_timeout="60000" join_retry_timeout="2000" shun="true" print_local_addr="true"/> | <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/> | </config> | </attribute> | <attribute name="FetchStateOnStartup">true</attribute> | <attribute name="InitialStateRetrievalTimeout">60000</attribute> | <attribute name="SyncReplTimeout">15000</attribute> | <attribute name="LockAcquisitionTimeout">10000</attribute> | <attribute name="EvictionPolicyClass"> org.jboss.cache.eviction.LRUPolicy</attribute> | <attribute name="EvictionPolicyConfig"> | <config> | <attribute name="wakeUpIntervalSeconds">5</attribute> | <region name="/_default_"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds">0</attribute> | </region> | </config> | </attribute> | <attribute name="UseMarshalling">false</attribute> | </mbean> | </server> CONFIG2==================== <server> | <classpath codebase="./lib" archives="jboss-cache.jar, jgroups-all.jar"/> | <mbean code="org.jboss.cache.TreeCache" | name="jboss.cache:service=TreeCache"> | <depends>jboss:service=Naming</depends> | <depends>jboss:service=TransactionManager</depends> | <attribute name="TransactionManagerLookupClass"> org.jboss.cache.JBossTransactionManagerLookup</attribute> | <attribute name="IsolationLevel">NONE</attribute> | <attribute name="CacheMode">REPL_SYNC</attribute> | <attribute name="UseReplQueue">true</attribute> | <attribute name="ReplQueueInterval">1000</attribute> | <attribute name="ReplQueueMaxElements">0</attribute> | <attribute name="ClusterName">Fbdblog-TreeCache-Cluster-Nondao</attribute> | <attribute name="ClusterConfig"> | <config> | <UDP mcast_addr=" 228.1.2.3" mcast_port="41333" | ip_ttl="64" ip_mcast="true" | mcast_send_buf_size="150000" mcast_recv_buf_size="80000" | ucast_send_buf_size="150000" ucast_recv_buf_size="80000" | loopback="true"/> | <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/> | <MERGE2 min_interval="10000" max_interval="20000"/> | <!-- <FD shun="true" up_thread="true" down_thread="true" />--> | <FD_SOCK/> | <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/> | <pbcast.NAKACK gc_lag="50" retransmit_timeout="600,1200,2400,4800" max_xmit_size="8192" up_thread="false" down_thread="false"/> | <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false"/> | <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false"/> | <FRAG frag_size="8192" down_thread="false" up_thread="false"/> | <pbcast.GMS join_timeout="60000" join_retry_timeout="2000" shun="true" print_local_addr="true"/> | <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/> | </config> | </attribute> | <attribute name="FetchStateOnStartup">true</attribute> | <attribute name="InitialStateRetrievalTimeout">60000</attribute> | <attribute name="SyncReplTimeout">15000</attribute> | <attribute name="LockAcquisitionTimeout">10000</attribute> | <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute> | <attribute name="EvictionPolicyConfig"> | <config> | <attribute name="wakeUpIntervalSeconds">5</attribute> | <region name="/_default_"> | <attribute name="maxNodes">50000</attribute> | <attribute name="timeToLiveSeconds">0</attribute> | </region> | </config> | </attribute> | <attribute name="UseMarshalling">false</attribute> | </mbean> | </server> View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4089137#4089137 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4089137 _______________________________________________ jboss-user mailing list [email protected] https://lists.jboss.org/mailman/listinfo/jboss-user
