Production cluster crashed with SEVERE: TCP Worker thread... error
Hi In my 4-node TC5.5.9 cluster, the catalina.out suddenly got flooded by the error below and the whole cluster simply stopped responding to web clients. I have to restart all nodes in order to get it back to normal. It seemed to be triggerred by a surge in loading but not able to resume even after the load was reduced later on. The cluster has been working fine previously. catalina.out: SEVERE: TCP Worker thread in cluster caught 'java.io.IOException: Connection reset by peer' closing channel java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:206) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:207) at org.apache.catalina.cluster.tcp.TcpReplicationThread.drainChannel(TcpReplicationThread.java:123) at org.apache.catalina.cluster.tcp.TcpReplicationThread.run(TcpReplicationThread.java:67) ... Here is my server.xml Cluster config: Cluster className=org.apache.catalina.cluster.tcp.SimpleTcpCluster managerClassName=qpi.bochk.web.ClusterSessionManager expireSessionsOnShutdown=false useDirtyFlag=true notifyListenersOnReplication=true Membership className=org.apache.catalina.cluster.mcast.McastService mcastAddr=228.0.0.6 mcastPort=45564 mcastFrequency=500 mcastDropTime=3000/ Receiver className=org.apache.catalina.cluster.tcp.ReplicationListener tcpListenAddress=auto tcpListenPort=4001 tcpSelectorTimeout=100 tcpThreadCount=10/ Sender className=org.apache.catalina.cluster.tcp.ReplicationTransmitter replicationMode=pooled ackTimeout=15000/ Valve className=org.apache.catalina.cluster.tcp.ReplicationValve filter=.*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.css;.*\.CSS;.*\.GIF;.*\.JPG;.*\.txt;/ Deployer className=org.apache.catalina.cluster.deploy.FarmWarDeployer tempDir=/tmp/war-temp/ deployDir=/tmp/war-deploy/ watchDir=/tmp/war-listen/ watchEnabled=false/ /Cluster Regards, Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Production cluster crashed with 'SEVERE: TCP Worker thread...' error
) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:364) at java.net.Socket.connect(Socket.java:507) at java.net.Socket.connect(Socket.java:457) at java.net.Socket.init(Socket.java:365) at java.net.Socket.init(Socket.java:207) at org.apache.catalina.cluster.tcp.DataSender.openSocket(DataSender.java:418) at org.apache.catalina.cluster.tcp.DataSender.pushMessage(DataSender.java:496) at org.apache.catalina.cluster.tcp.DataSender.sendMessage(DataSender.java:378) at org.apache.catalina.cluster.tcp.PooledSocketSender.sendMessage(PooledSocketSender.java:124) at org.apache.catalina.cluster.tcp.ReplicationTransmitter.sendMessageData(ReplicationTransmitter.java:651) at org.apache.catalina.cluster.tcp.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:348) at org.apache.catalina.cluster.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:466) at org.apache.catalina.cluster.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:495) at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:210) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:595) Hi In my 4-node TC5.5.9 cluster, the catalina.out suddenly got flooded by the error below and the whole cluster simply stopped responding to web clients. I have to restart all nodes in order to get it back to normal. It seemed to be triggerred by a surge in loading but not able to resume even after the load was reduced later on. The cluster has been working fine previously. catalina.out: SEVERE: TCP Worker thread in cluster caught 'java.io.IOException: Connection reset by peer' closing channel java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:206) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:207) at org.apache.catalina.cluster.tcp.TcpReplicationThread.drainChannel(TcpReplicationThread.java:123) at org.apache.catalina.cluster.tcp.TcpReplicationThread.run(TcpReplicationThread.java:67) ... Here is my server.xml Cluster config: Cluster className=org.apache.catalina.cluster.tcp.SimpleTcpCluster managerClassName=qpi.bochk.web.ClusterSessionManager expireSessionsOnShutdown=false useDirtyFlag=true notifyListenersOnReplication=true Membership className=org.apache.catalina.cluster.mcast.McastService mcastAddr=228.0.0.6 mcastPort=45564 mcastFrequency=500 mcastDropTime=3000/ Receiver className=org.apache.catalina.cluster.tcp.ReplicationListener tcpListenAddress=auto tcpListenPort=4001 tcpSelectorTimeout=100 tcpThreadCount=10/ Sender className=org.apache.catalina.cluster.tcp.ReplicationTransmitter replicationMode=pooled ackTimeout=15000/ Valve className=org.apache.catalina.cluster.tcp.ReplicationValve filter=.*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.css;.*\.CSS;.*\.GIF;.*\.JPG;.*\.txt;/ Deployer className=org.apache.catalina.cluster.deploy.FarmWarDeployer tempDir=/tmp/war-temp/ deployDir=/tmp/war-deploy/ watchDir=/tmp/war-listen/ watchEnabled=false/ /Cluster Regards, Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: SV: Cluster: will session listeners got called again after replication?
I have it turned on already. But seems that after a node is killed and then started up again, even though it can pick up the sessions from the other nodes, my HttpSessionBindingListener and HttpSessionListener were not called at all during the replication. Joseph On Fri, 25 Feb 2005, Filip Hanik - Dev Lists wrote: there is a flag you can set so that listeners don't get called, its optional its called notifyListenersOnReplication, see server.xml for example, default is true Filip Jesper Ekberg wrote: Hello! My first mail to this list. :) I have read it for a long time tho. We have a tried to cluster 3 Tomcat 5.5.7 machines and I found that HttpSessionBindingListener will be notified when the session is replicated and the machine crashes. I think this must be a bug?? The scenario: 3 Tomcat 5.5.7 machines on Windows 2003 Server (I know, not my fault ;)). JK2 Connector for load balancing. I log on and session is created and is replicated correctly to all machines. I shut down the server that I'm working on. The session is destroyed and method valueUnbound is called on the crashed machine. It seems odd to me that the method valueUnbound is called when the session is replicated, the session still lives on the other Tomcat machines. Sorry for my sometimes bad English ;) //Jesper -Ursprungligt meddelande- Från: Joseph Lam [mailto:[EMAIL PROTECTED] Skickat: den 24 februari 2005 08:44 Till: Tomcat Users List Ämne: Cluster: will session listeners got called again after replication? Anyone knows when a session is replicated to other nodes, will the HttpSessionBindingListener and HttpSessionAttributeListener objects be notified again? On the receiver nodes, how can I detect when a session from the sender node comes in so that I can do something with it? Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
workers.properties: cache_timeout/socket_timeout/...
Anyone familiar with the tuning of the timeout settings? cache_timeout socket_timeout recycle_timeout prepost_timeout connect_timeout reply_timeout There are so many timeout settings but the doc online is a bit confusing to me. I don't know which value is used under different situations... e.g.: When Apache create a new ajp connection to TC and forward a request, will the socket_timeout or connect_timeout or prepost_timeout be used?(or a mix of them?) When an ajp connection is idled, will cache_timeout or recycle_timeout be used? Is socket_timeout always in effect together with each of the other timeouts? What if the page content from TC is lengthy and took longer than cache/recycle_timeout? Regards, Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Cluster: Is session's lastAccessedTime got replicated?
Hi, It seems that in my 4-node cluster (TC 5.5.8), the session.getLastAccessedTime() only returns the last accessed time in each particular node only. Is this normal? How can I get the last accessed time of a session across the whole cluster? Regards, Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Cluster: Is session's lastAccessedTime got replicated?
Found that only when a replication is explicitly triggered by set/removeAttribute(), the other nodes' session.getLastAccessedTime() will be synchronized. Joseph On Thu, 24 Feb 2005, Joseph Lam wrote: Hi, It seems that in my 4-node cluster (TC 5.5.8), the session.getLastAccessedTime() only returns the last accessed time in each particular node only. Is this normal? How can I get the last accessed time of a session across the whole cluster? Regards, Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Cluster: how to set mcast interface for dual LAN card? (SOLVED)
I have tried that but still got some weird behavior (seems that it was able to send out broadcast with the proper interface but unable to listen broadcast...). So I gave up that an simply set a static route in the OS for mcast. But thanks anyway. Joseph On Tue, 22 Feb 2005, Filip Hanik - Dev Lists wrote: there is an attribute mcastBindAddr that allows you to bind to the interface. Joseph Lam wrote: Hi, If I have two LAN cards and I want my Tomcat to mcast through one of them, what parameter should I set? Regards, Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Cluster: will session listeners got called again after replication?
Anyone knows when a session is replicated to other nodes, will the HttpSessionBindingListener and HttpSessionAttributeListener objects be notified again? On the receiver nodes, how can I detect when a session from the sender node comes in so that I can do something with it? Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
JSP page buffering problem
In my JSP pages, I have already set a page buffer size sufficiently large so that it can hold the whole page before having to flush. However, I still occasionally got exceptions when I try to forward the request in the middle, complaining that it can't forward the request as some content is already flushed. Are there any other factors (e.g. timing, browser, %@ include %, etc...) which might have forced the flush before the buffer is full? Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Cluster: how to set mcast interface for dual LAN card?
Hi, If I have two LAN cards and I want my Tomcat to mcast through one of them, what parameter should I set? Regards, Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: How to detect expired session vs. no session? (Solved)
Steve Kirk wrote: By default: 1. getSession(true)!=null 2. getSession(false)!=null But if a JSP page contains the tag %@ page session=false %, then: 1. getSession(true)!=null 2. getSession(false)==null In the last of these 4 cases, do you mean that the implicit JSP session object returns null, or that request.getSession(false) returns null? I could understand the first behaviour but would be surprised by the second. The problem is that it implies that JSPs execute the code in a way that is different than if it were included in a servlet, and given that JSPs are servlets, this seems puzzling. Again, perhaps I'm not fully understanding either your case, or the details of how sessions work. I'll do some testing to confirm whether the implicit session object is null (which I guess so). I'm sure request.getSession(false) can really be null coz I'm relying on that in my pages, which all are forced not to create session by default until I received the login params and authenticate them, and then manually call HttpSession mySession = request.getSession(true) to create one. For my case, sessions will only be created for logged-in users what is it about your case that makes this happen? I would have thought that session creation is independent of whether you are authenticating or not. Or is there a way to config TC to not create sessions by default, and only create them when the user successfully authenticates? Add this and TC won't create session by default: %@ page session=false % - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
how to get tomcat's tmp dir path?
any way to get the absolute path to my $CATALINA_BASE or $CATALINA_BASE/temp directory? Thx. Joseph - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: heap size
Is there a rule-of-thumb for setting the heap size based on how many concurrent Tomcat processors/threads? Mine are mostly basic jsps and servlets generating HTML. As I'll be running several Tomcat instances for different apps, I need to allocate my 512M RAM to each Tomcat. Joseph Shapira, Yoav wrote: Howdy, Yoav - Where do you specify heap size? I understand it is a parameter of the java command, but is in a Tomcat startup script somewhere? Or do I just execute it at the command line? Add JAVA_OPTS='-Xmx512m' to $CATALINA_HOME/bin/catalina.sh on unix. On windows, you can do this with a JAVA_OPTS environment variable, or (if running as a service) via the registry. Search this list's archives for more information. The 512m above specifies a maximum heap of 512 MB. Run java -X to see other options. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: JK2 performance tuning
I'm new to Apache 2Can you kindly show me how to config that? Joseph Luciano Kiniti Issoe wrote: Have you already compiled Apache for MaxClients 256 ? - Original Message - From: Simon Pabst [EMAIL PROTECTED] To: Tomcat Users List [EMAIL PROTECTED] Sent: Thursday, July 17, 2003 1:22 PM Subject: Re: JK2 performance tuning You can tune CoyoteConnector acceptCount and timeout values. acceptCount=10 should be fine, though you never know (test test test :-) timeout value (in seconds) shouldn't be set to 0 (infinite) or too high, else your socket count will go up much as time goes by (which not only consumes file descriptors but valuable JVM memory AFAIK), try reasonable values like 10 mins (600) (2 is suggested in jk2 doc). If timeout is too low you could get errors too (server not available), so test test test again. If you use mod_jk2 load balancing use multithreading Apache (worker mpm) At 20:25 17.07.2003 +0800, you wrote: I just got JK2 2.0.2 installed and working with Apache 2.0.47. Since my site will have high traffic and lots of concurrent connections, what parameters should I fine tune (other than the min/maxProcessor in server.xml) ? How does Apache manage its jk2 connections to Tomcat? Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
JK2 performance tuning
I just got JK2 2.0.2 installed and working with Apache 2.0.47. Since my site will have high traffic and lots of concurrent connections, what parameters should I fine tune (other than the min/maxProcessor in server.xml) ? How does Apache manage its jk2 connections to Tomcat? Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: JK2 performance tuning
So nothing related to the jk2 connection between Apache and TC can be tuned? How many connections will Apache make? Auto grow? Joseph Lam Simon Pabst wrote: You can tune CoyoteConnector acceptCount and timeout values. acceptCount=10 should be fine, though you never know (test test test :-) timeout value (in seconds) shouldn't be set to 0 (infinite) or too high, else your socket count will go up much as time goes by (which not only consumes file descriptors but valuable JVM memory AFAIK), try reasonable values like 10 mins (600) (2 is suggested in jk2 doc). If timeout is too low you could get errors too (server not available), so test test test again. If you use mod_jk2 load balancing use multithreading Apache (worker mpm) At 20:25 17.07.2003 +0800, you wrote: I just got JK2 2.0.2 installed and working with Apache 2.0.47. Since my site will have high traffic and lots of concurrent connections, what parameters should I fine tune (other than the min/maxProcessor in server.xml) ? How does Apache manage its jk2 connections to Tomcat? Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Q on Tomcat + Apache sessions
Would anyone pls tell me how sessions are kept and passed between Apache and Tomcat via jk2, e.g. whether it is completely transparent so that when I write session related codes in Tomcat I can ignore Apache? What about in a load balancing config (e.g. 1 Tomcat to 2 Apache, 1 Apache to two Tomcat)? Joseph Lam - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat 4.1.18/24 unstable (UPDATE)
I just found out what caused the 'ESTABLISHED' connection piling up problem. It was due to a javascript form.submit() triggered by pressing 'Enter' key which causes IE to double submit the form. What it does so it will first submit the form, then quickly send a reset and submit it again. That seems to confuse Tomcat. After fixing the Javascript the problem is largely gone. Joseph joseph lam wrote: I also had the same problem with both 4.1.18 and 24. Now I resort to setting a high maxProcessor (e.g. 800) and restart the tomcat once the connections are reaching that. I also found that those 'ESTABLISHED' connections would stay forever eating up sockets and tomcat connectors. I've also tried to tune the ipv4 kernel settings for example to shorten the keep alive timeout but no luck. I'm using Redhat 7.3 and Sun JDK 1.4.1_02. Joseph Eric Roberts wrote: We currently have secure web services deployed within tomcat 4.1.18 using axis. We have 4 or 5 backend information providers submitting information to the web service every 5 minutes. Every so often I can run a 'netstat -an | grep 8443' and see roughly 100+ connections with the ESTABLISHED status. It seems as though the connections are hanging from the tomcat end. Has anyone else experienced this? ( the limit of connections in tomcat is set to 100, time out is 2 in server.xml ) Thanks in advance - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
