Let me know if this should be posted somewhere else.
JBoss 5.1.0
JBoss Cache 3.2.1
Three servers in cluster, HTTP Session replication, UDP, total replication,
passivation enabled
We recently rolled out two changes to our system, HTTP Session passivation and
an upgrade to JBoss Cache 3.2.1. The changes were tested for several days
under load on two other systems, but now that we've deployed to our primary
data center we've run into an issue.
Things ran fine for about 22 hours, then server A and server B went into a loop
where each server getting the following two stack traces repeatedly. This
happened twice in 24 hours. We've not disabled session passivation and have
not seen the issue.
In order to get the servers out of the loop we had to restart jboss on both
servers (one of them twice). JBoss was still responding but the cluster was in
an obviously degraded state.
2009-10-20 15:40:29,709 ERROR
[org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AttributeBasedJBossCacheService]
(ContainerBackgroundProcessor[StandardEngine[jboss.web]]) IOException occurred
unmarshalling value
| java.io.OptionalDataException
| at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1349)
| at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
| at javolution.util.FastMap.readObject(Unknown Source)
| at sun.reflect.GeneratedMethodAccessor362.invoke(Unknown Source)
| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:597)
| at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
| at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
| at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
| at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
| at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
| at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
| at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
| at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
| at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
| at
org.jboss.ha.framework.server.SimpleCachableMarshalledValue.get(SimpleCachableMarshalledValue.java:94)
| at
org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AbstractJBossCacheService.getUnMarshalledValue(AbstractJBossCacheService.java:660)
| at
org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AttributeBasedJBossCacheService.getSessionAttributes(AttributeBasedJBossCacheService.java:166)
| at
org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AbstractJBossCacheService.getDistributableSessionData(AbstractJBossCacheService.java:581)
| at
org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AbstractJBossCacheService.getSessionData(AbstractJBossCacheService.java:364)
| at
org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1832)
| at
org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:489)
| at
org.jboss.web.tomcat.service.session.JBossCacheManager.processExpirationPassivation(JBossCacheManager.java:1603)
| at
org.jboss.web.tomcat.service.session.JBossManager.backgroundProcess(JBossManager.java:385)
| at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1327)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1612)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1601)
| at java.lang.Thread.run(Thread.java:619)
|
| 2009-10-20 15:40:29,713 ERROR
[org.jboss.web.tomcat.service.session.JBossCacheManager.]
(ContainerBackgroundProcessor[StandardEngine[jboss.web]])
processExpirationPassivation(): failed handling unloaded session
O-lgMwkUSLW1vVpTRQzExg__
| java.lang.NullPointerException
| at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
| at
java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:909)
| at
org.jboss.web.tomcat.service.session.ClusteredSession.populateAttributes(ClusteredSession.java:1661)
| at
org.jboss.web.tomcat.service.session.ClusteredSession.update(ClusteredSession.java:1120)
| at
org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1835)
| at
org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:489)
| at
org.jboss.web.tomcat.service.session.JBossCacheManager.processExpirationPassivation(JBossCacheManager.java:1603)
| at
org.jboss.web.tomcat.service.session.JBossManager.backgroundProcess(JBossManager.java:385)
| at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1327)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1612)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1621)
| at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1601)
| at java.lang.Thread.run(Thread.java:619)
|
It appears that both servers were trying to clean up the exact same "unloaded"
session at the same time. That might be normal, but something in the session
data seemed to have gotten corrupted? It's not clear yet, I'm currently
looking at the JBossCacheManager code.
Relevant configs:
>From jboss-web.xml
| <passivation-config>
| <use-session-passivation>true</use-session-passivation>
| <passivation-min-idle-time>-1</passivation-min-idle-time>
| <passivation-max-idle-time>300</passivation-max-idle-time>
| </passivation-config>
|
>From jboss-cache-manager-jboss-beans.xml
<property name="cacheLoaderConfig">
| <bean class="org.jboss.cache.config.CacheLoaderConfig">
| <!-- Do not change these -->
| <property name="passivation">true</property>
| <property name="shared">false</property>
|
| <property name="individualCacheLoaderConfigs">
| <list>
| <bean
class="org.jboss.cache.loader.FileCacheLoaderConfig">
| <!-- Where passivated sessions are stored -->
| <property
name="location">${jboss.server.data.dir}${/}session</property>
| <!-- Do not change these -->
| <property name="async">false</property>
| <property
name="fetchPersistentState">true</property>
| <property name="purgeOnStartup">true</property>
| <property
name="ignoreModifications">false</property>
| <property
name="checkCharacterPortability">false</property>
| </bean>
| </list>
| </property>
| </bean>
| </property>
|
Has anyone seen this issue? I see other people referencing
java.io.OptinalDataException in the context of JBoss, but nothing that sounds
like this exact same issue. Any input would be greatly appreciated, we'd
really like to get HTTP session passivation enabled again.
View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4261558#4261558
Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4261558
_______________________________________________
jboss-user mailing list
[email protected]
https://lists.jboss.org/mailman/listinfo/jboss-user
