Hi,
I'm currently trying to establish a cluster of tomcat 5.5.27
servers to deploy sakai in, so I'm stuck on version 5.5. It is not an option to
upgrade to tomcat 6 which from all accounts has much better session replication.
I have cluster membership up and working via multicast and the two nodes add
each other as replication targets. When I try to establish a connection to the
server from my web browser to the default port of 8080 I get this neat error
in my catalina.out:
ERROR: An exception or error occurred in the container during the request
processing (2009-06-17 16:43:49,545
http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at
org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
I'll put full errors and configs at the bottom of the email. The errors occurs
regardless of whether I access an empty/nonexistent context or the
servlet-examples which are currently installed.
The servers are multi homed with 3 interfaces. Eth0 is the
primary public interface, eth1 is my iSCSI network and eth2 is the backend
cluster network. All neatly VLANd from each other in seperate subnets.
Multicast membership works, I can see the packets arriving via tcpdump and the
membership propagation is occurring.
I have tried binding the receivers to both public and private interfaces but
have had no luck. The receivers are up and appear to be functional. I can
telnet to them and when I enter gibberish into the console it tell me I am
sending malformed headers.
By using tcpdump and netstat I can see that neither tomcat instance is
originating a sender session to connect to the other servers receiver.
I have tried fastsyncqueue and pooled style senders from the documentation.
Currently configured for pooled as it looked the simpler config to diagnose.
jvmRoute is configured via -D options passed to java as a lot of our configs
are done dynamically so as to have a single server.xml across the cluster
environments. We have a shared disk backend system to enable this.
Any suggestions for diagnosis, help, fixes or otherwise would be very much
appreciated.
Regards,
Dave
Log file and startup:
INFO: The Apache Tomcat Native library which allows optimal performance in
production environments was not found on the java.library.path:
/apps/java/jdk1.5.0_19/jre/lib/amd64/server:/apps/java/jdk1.5.0_19/jre/lib/amd64:/apps/java/jdk1.5
.0_19/jre/../lib/amd64 (2009-06-17 16:40:36,874
main_org.apache.catalina.core.AprLifecycleListener)
INFO: Initializing Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17
16:40:37,019 main_org.apache.coyote.http11.Http11BaseProtocol)
INFO: Initialization processed in 1400 ms (2009-06-17 16:40:37,021
main_org.apache.catalina.startup.Catalina)
INFO: Set JAAS app name ENGINE (2009-06-17 16:40:37,114
main_org.apache.catalina.realm.JAASRealm)
INFO: Starting service SERVICE (2009-06-17 16:40:37,126
main_org.apache.catalina.core.StandardService)
INFO: Starting Servlet Engine: Apache Tomcat/5.5.27 (2009-06-17 16:40:37,133
main_org.apache.catalina.core.StandardEngine)
INFO: Cluster is about to start (2009-06-17 16:40:37,133
main_org.apache.catalina.cluster.tcp.SimpleTcpCluster)
INFO: Start ClusterSender at cluster ENGINE:type=Cluster with name
ENGINE:type=ClusterSender (2009-06-17 16:40:37,430
main_org.apache.catalina.cluster.tcp.ReplicationTransmitter)
INFO: Setting cluster mcast soTimeout to 1000 (2009-06-17 16:40:37,440
main_org.apache.catalina.cluster.mcast.McastService)
INFO: Sleeping for 4000 milliseconds to establish cluster membership
(2009-06-17 16:40:37,460 main_org.apache.catalina.cluster.mcast.McastService)
INFO: membership mbean registered (ENGINE:type=ClusterMembership) (2009-06-17
16:40:41,478 main_org.apache.catalina.cluster.mcast.McastService)
INFO: XML validation disabled (2009-06-17 16:40:41,504
main_org.apache.catalina.core.StandardHost)
INFO: ContextListener: contextInitialized() (2009-06-17 16:40:43,318
main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
INFO: SessionListener: contextInitialized() (2009-06-17 16:40:43,318
main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
INFO: JvmRouteBinderValve started (2009-06-17 16:40:43,391
main_org.apache.catalina.cluster.session.JvmRouteBinderValve)
INFO: Starting Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17
16:40:43,467 main_org.apache.coyote.http11.Http11BaseProtocol)
INFO: JK: ajp13 listening on /172.31.3.37:8009 (2009-06-17 16:40:43,640
main_org.apache.jk.common.ChannelSocket)
INFO: Jk running ID=0 time=0/99 config=null (2009-06-17 16:40:43,648
main_org.apache.jk.server.JkMain)
INFO: Find registry server-registry.xml at classpath resource (2009-06-17
16:40:44,411 main_org.apache.catalina.storeconfig.StoreLoader)
INFO: Server startup in 7485 ms (2009-06-17 16:40:44,506
main_org.apache.catalina.startup.Catalina)
INFO: Replication member
added:org.apache.catalina.cluster.mcast.McastMember[tcp://172.31.3.34:9015,SAKAINEW,172.31.3.34,9015,
alive=11] (2009-06-17 16:40:50,160
Cluster-MembershipReceiver_org.apache.catalina.cluster.tcp.SimpleTcpCluster
)
ERROR: An exception or error occurred in the container during the request
processing (2009-06-17 16:43:44,786
http-172.31.3.37-8080-Processor4_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at
org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
ERROR: An exception or error occurred in the container during the request
processing (2009-06-17 16:43:49,545
http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at
org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
server.xml
<Server port="7992" shutdown="SHUTDOWN">
<!-- TOMCAT 5 and 5.5-->
<Listener
className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
<Listener
className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.storeconfig.StoreConfigLifecycleListener"/>
<!-- JNDI -->
<GlobalNamingResources>
</GlobalNamingResources>
<Service name="SERVICE">
<!-- HTTP connector -->
<Connector address="${ClusterIP}" port="8080"
maxHttpHeaderSize="8192" protocol="HTTP/1.1"
maxThreads="50"
minSpareThreads="5" maxSpareThreads="10"
enableLookups="false"
redirectPort="8443" acceptCount="100"
connectionTimeout="0"
disableUploadTimeout="true" />
<!-- AJP connector -->
<Connector address="${ClusterIP}" port="8009"
protocol="AJP/1.3"
connectionTimeout="600000"
enableLookups="false"
redirectPort="8443" />
<!-The Engine -->
<Engine name="ENGINE"
defaultHost="localhost">
<Host name="localhost"
appBase="webapps"
unpackWARs="true"
autoDeploy="false"
xmlValidation="false"
xmlNamespaceAware="false">
<Alias>sakai-new.une.edu.au</Alias>
<Alias>espresso.une.edu.au</Alias>
<Alias>latte.une.edu.au</Alias>
</Host>
<Cluster
className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
manager.expireSessionsOnShutdown="false"
doClusterLog="true"
clusterLogName="clusterlog">
<Membership
className="org.apache.catalina.cluster.mcast.McastService"
mcastAddr="228.0.0.4"
mcastClusterDomain="SAKAINEW"
mcastPort="45564"
mcastFrequency="1000"
mcastDropTime="30000"
recoveryCounter="10"
recoveryEnabled="true"
recoverySleepTime="5000" />
<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"
tcpListenAddress="${ClusterIP}"
tcpListenPort="9015"
tcpSelectorTimeout="100"
tcpThreadCount="5"/>
<Sender
className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
replicationMode="pooled"
autoConnect="true"
maxPoolSocketLimit="10"
keepAliveTimeout="60000"
keepAliveMaxRequestCount="-1"
waitForAck="true" />
<ClusterListener
className="org.apache.catalina.cluster.session.ClusterSessionListener" />
<ClusterListener
className="org.apache.catalina.cluster.session.JvmRouteSessionIDBinderListener"
/>
<Valve
className="org.apache.catalina.cluster.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.css;.*\.png;.*\.jpeg;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"
primaryIndicator="true" />
<Valve
className="org.apache.catalina.cluster.session.JvmRouteBinderValve"
enabled="true"
/>
</Cluster>
</Engine>
</Service>
</Server>
David Nillesen
UNIX Systems Administrator
University of New England
+61 2 6773 2112
