My replies are buried in the middle. I'll surround them with ===== Question
=======
/mde/
----- Original Message -----
> From: Dhaval Jaiswal <dhaval.jais...@via.com>
> To: Tomcat Users List <users@tomcat.apache.org>
> Cc:
> Sent: Monday, March 26, 2012 3:33 AM
> Subject: org.apache.catalina.tribes.ChannelException: Operation has timed
> out(3000 ms.).; Faulty members:tcp://{-64, -88, 3, 16}:4000;
>
> Hi List,
>
> Below is my server.xml configuration for session replication.
>
>
> <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
> channelSendOptions="6"/>
> <Manager
> className="org.apache.catalina.ha.session.BackupManager"
> expireSessionsOnShutdown="false"
> notifyListenersOnReplication="true"
> mapSendOptions="6"/>
> <Channel
> className="org.apache.catalina.tribes.group.GroupChannel">
> <Membership
> className="org.apache.catalina.tribes.membership.McastService"
> address="192.168.3.14"
> port="45564"
> frequency="500"
> dropTime="30000"/>
> <Receiver
> className="org.apache.catalina.tribes.transport.nio.NioReceiver"
> address="auto"
> port="5000"
> selectorTimeout="5000"
> timeout="60000"
> maxThreads="6"/>
>
> <Sender
> className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
> <Transport
> className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"
> timeout="60000" keepAliveCount="0"/>
> </Sender>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
> <Interceptor
> className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
> </Channel>
> <Valve
> className="org.apache.catalina.ha.tcp.ReplicationValve"
>
> filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt"/>
===== Question =======
Don't you need another Valve at this point? All of my (admittedly small,
default) configurations have the following Valve in the Cluster configuration:
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
Each engine then gets a unique jvmRoute="someName" which matches the AJP worker
name that the particular Tomcat talks to. The worker name is in
workers.properties (/etc/httpd/conf.d/workers.properties on my Fedora 15 and
CentOS 5.8 systems).
I don't know for certain since I've not worked with the backup manager.
===== Question =======
> <ClusterListener
> className="org.apache.catalina.ha.session.ClusterSessionListener"/>
===== Question =======
Don't you need another ClusterListener at this point? All of my (admittedly
small, default) configurations have the following ClusterListener at the end:
<ClusterListener
className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
I will also assume that you missed copying the closing </Cluster> tag from your
server.xml.
===== Question =======
>
>
> We are continuously getting following logging in our catalina.out. Can any
> one explain me why it is saying Operation has timed out(3000 ms.)
> How do i resolve it. I have searched the archive mail & change the setting
> according to it. However, no luck. Server is getting high load after adding
> session replication for backup manager. Your valuable response will be
> highly appreciated.
>
>
>
> SEVERE: Unable to send message through cluster sender.
> org.apache.catalina.tribes.ChannelException: Operation has timed out(3000
> ms.).; Faulty members:tcp://{-64, -88, 3, 16}:4000;
> at
> org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:97)
> at
> org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:53)
> at
> org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:80)
> at
> org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:78)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:87)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:73)
> at
> org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:75)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:216)
> at
> org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:175)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:816)
> at
> org.apache.catalina.ha.tcp.SimpleTcpCluster.sendClusterDomain(SimpleTcpCluster.java:794)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:551)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:535)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:517)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:428)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:362)
> at
> org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:209)
> at
> org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:347)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
> at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:419)
> at
> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:378)
> at
> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1509)
> at java.lang.Thread.run(Thread.java:619)
> Mar 26, 2012 3:30:56 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 3,
> 10}:4000,{-64, -88, 3, 10},4000, alive=11339404,id={45 15 -8 5 54 -73 68
> 116 -124 21 124 -128 -34 -127 73 21 }, payload={}, command={}, domain={}, ]
> Mar 26, 2012 3:30:58 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> performBasicCheck
> WARNING: Member added, even though we werent
> notified:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88,
> 3, 16}:4000,{-64, -88, 3, 16},4000, alive=415653,id={-74 -55 5 -64 -44 -111
> 72 -63 -115 -105 -102 3 34 45 -109 -7 }, payload={}, command={}, domain={},
> ]
> Mar 26, 2012 3:30:58 PM org.apache.catalina.ha.tcp.SimpleTcpCluster
> memberAdded
> INFO: Replication member
> added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 3,
> 16}:4000,{-64, -88, 3, 16},4000, alive=415653,id={-74 -55 5 -64 -44 -111 72
> -63 -115 -105 -102 3 34 45 -109 -7 }, payload={}, command={}, domain={}, ]
> Mar 26, 2012 3:30:58 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Received
> memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64,
> -88, 3, 10}:4000,{-64, -88, 3, 10},4000, alive=11335305,id={45 15 -8 5 54
> -73 68 116 -124 21 124 -128 -34 -127 73 21 }, payload={}, command={},
> domain={}, ]] message. Will verify.
> Mar 26, 2012 3:30:58 PM
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector
> memberDisappeared
> INFO: Verification complete. Member still
> alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 3,
> 10}:4000,{-64, -88, 3, 10},4000, alive=11335305,id={45 15 -8 5 54 -73 68
> 116 -124 21 124 -128 -34 -127 73 21 }, payload={}, command={}, domain={}, ]]
> Mar 26, 2012 3:30:58 PM org.apache.catalina.ha.tcp.SimpleTcpCluster send
>
===== Question =======
Some networking questions now . . .
I note that in your configuration you've set up the TCP communications on port
5000. This is fine, but why do your logs show port 4000?
I just started up a 3 node cluster with fixed ports (4000, 4100, 4200), and
those appear in my cluster logs (see setup below). I don't see port 5000 in any
of your logs.
Are your Tomcats reading the server.xml file you think they're reading? Do the
following (on Linux - your command may vary):
netstat -an | grep 5000
You should see a process listening on port 5000.
If that's not the case, then you're not using the server.xml file you think you
are.
Finally, you'll need to open those ports up on your firewall. I'm guessing
you'll need something like the following (change ports and addresses to match
your environment:
-A INPUT -m state --state NEW -m udp -p udp --dport 45564 -s 192.168.3.0/24 -j
ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5000 -s 192.168.3.0/24 -j
ACCEPT
Again, this is from Linux (iptables), so you'll have to adjust accordingly.
===== Question =======
I run three and four node test clusters with my configuration with no problems.
I hope to get a more cookbook approach to clustering written sometime. I'm also
looking at how clustering interacts with WAR file versions in Tomcat 7.
I've found that managing clustering logs outside of the normal logs works
better for me as well. I modified the default logging.properties file to put
cluster logging in a separate location.
I'm using the farm deployer, so you'll see logging for that in the listing
below as well.
#
# the new handlers - sorry for the word wrap
#
handlers = 1catalina.org.apache.juli.FileHandler,
2localhost.org.apache.juli.FileHandler, 3manager.org.apache.juli.FileHandler,
4host-manager.org.apache.juli.FileHandler,
java.util.logging.ConsoleHandler,5cluster.org.apache.juli.FileHandler
#
# the handler - note this generates LOTS of logging
#
5cluster.org.apache.juli.FileHandler.level = FINER
5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
5cluster.org.apache.juli.FileHandler.prefix = cluster.
#
# the actual definitions - turned down just a little
#
org.apache.catalina.tribes.MESSAGES.level = FINE
org.apache.catalina.tribes.MESSAGES.handlers =
5cluster.org.apache.juli.FileHandler
org.apache.catalina.tribes.level = FINE
org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler
org.apache.catalina.ha.level = FINE
org.apache.catalina.ha.handlers = 5cluster.org.apache.juli.FileHander
org.apache.catalina.ha.deploy.level = INFO
org.apache.catalina.ha.deploy.handlers = 5cluster.org.apache.juli.FileHandler
===== Question =======
. . . . just my two cents.
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
