Re: Frequent SEVERE: Unable to receive message through TCP channel messages
we are getting the GC printed on to the same catalina.out and we see that the memberAdded messages appear almost at the time of GB getting printed, does it prove that longer GC pauses are causing this? is there any other data points/proof can be get? rearding network problems, we are requesting the network to capture the multicast traffic between these nodes, is there anything you suggest us to do? regaring increase the membership timeout we plan to increase this to 5 minutes, do you have any other suggestions. tomcat startup takes almost 70 seconds ( it hosts almost 32 apps) and all of them are clustered. regards, Rainer Jung-3 wrote: You configured a 3 seconds timeout for your heartbeat. If a node doesn't receive a heartbeat packet for 3 seconds, it assumes the other node is dead and closes the incoming replication connection. If the other node is not really dead, it will try to use this replication connection which will not work any more. Why could this happen: one possible reason are GC pauses. If you've got longer GC pauses, than your membership heartbeat timeout, then you run into such problems. During normal operations you should not observe any memberDisappeared messages. They should only show up, ehen you stop a node or it crashes, or you've got serious network problems with impact on the multicast heartbeat packets. If you decide to increase the membership timeout (which sounds like a good idea), keep in mind, that you need to wait the given time between stopping and restarting a node. Regards, Rainer nageshsrao wrote: Hi, In our prod environment we have two tomcat's [ 5.0.27] running on two linux boxes [ RHAS 3.0 update8 ] and using mod_jk2.0 thru apache for accessing the information. very frequently we see the following messages in the catalina.out and there are about 2 instances where tomcat stopped responding and we had to restart. the only errors that we see are the following.. There are INFO which keeps telling us member is disappeared and added and once in a while we have SEVERE messages. Could you let us know, what could be causing this problem? is there any additional configuration that are needed?, This environment is running for almost 18 months in production and off-late [ in the last 6 months] we have seen this happenned twice. I have attached both the error log found in the catalina.out and also the server.xml from both the tomcat. http://www.nabble.com/file/p12142134/catalina-error.out catalina-error.out http://www.nabble.com/file/p12142134/server-app1.xml server-app1.xml http://www.nabble.com/file/p12142134/server-app2.xml server-app2.xml - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Frequent-%22SEVERE%3A-Unable-to-receive-message-through-TCP-channel%22-messages-tf4266454.html#a12176135 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Frequent SEVERE: Unable to receive message through TCP channel messages
Hi, In our prod environment we have two tomcat's [ 5.0.27] running on two linux boxes [ RHAS 3.0 update8 ] and using mod_jk2.0 thru apache for accessing the information. very frequently we see the following messages in the catalina.out and there are about 2 instances where tomcat stopped responding and we had to restart. the only errors that we see are the following.. There are INFO which keeps telling us member is disappeared and added and once in a while we have SEVERE messages. Could you let us know, what could be causing this problem? is there any additional configuration that are needed?, This environment is running for almost 18 months in production and off-late [ in the last 6 months] we have seen this happenned twice. I have attached both the error log found in the catalina.out and also the server.xml from both the tomcat. http://www.nabble.com/file/p12142134/catalina-error.out catalina-error.out http://www.nabble.com/file/p12142134/server-app1.xml server-app1.xml http://www.nabble.com/file/p12142134/server-app2.xml server-app2.xml -- View this message in context: http://www.nabble.com/Frequent-%22SEVERE%3A-Unable-to-receive-message-through-TCP-channel%22-messages-tf4266454.html#a12142134 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: core_output_filter error for apache/tomcat JK connector
i think i goofed up while uploading the workers.properties. here is the one i just picked from the server. site is hung up means even the first page ( the login page) doesn't get displayed on the browser. it just goes on and on.. - we can not connect to apache during this behaviour - access_log has the log of our request. but no response - i didn't try for non-tomcat resources ( static files ). i will try to do that today we plan to take downtime during the night hours today and i will turn the debugging mode on and try setting up the JK timeout values and also will try to access static files which are not on tomcat. please take a look at the workers.properties and let me know if you find anything unusual. let us know if there any other debug parameters which we can set to more information. also as a information, this apache also has a weblogic plugin and even for those pages we do not get any response during this state of apache non-response. I Thank you very much for closely following and helping up on this regards, nagesh http://www.nabble.com/file/p11518117/workers.properties workers.properties Rainer Jung-3 wrote: Your workers.properties don't look complete, for instance there is no worker.list in there, so all your worker definitions will not be functional. Have a look at the various timeouts mod_jk provides, especially connect_timeout and reply_timeout. You could also use apaches builtin server-status and mod_jk status worker to get an idea, what the web server is actually working on. Finally with an increased jk log level of info, you might get additional information. The Apache error simply looks like the client/browser didn't wait until it fully received the response. That's normal, especially if responses take too long. This should result in a hanging site. Maybe you can describe more pricisely, what you mean by site is hung up. - can you connect to apache - dies it log your requests into the access log - if you connect with a technical browser like curl or telnet, do you get any answers for your requests - will requests for static ressources (I mean requests that do not match your forwarding rules) still work? Although I don't think this is the source of your problems, httpd 2.0.46 is pretty outdated ... Regards, Rainer nageshsrao schrieb: Hi, here are further information about the environment and answers to the below question -Tomcat and apache are on two different servers ( separated by a firewall) -Tomcat 5.5.23 is running on Sun JDK 1.5.0_12 -Direct connection to tomcat http connector ( 8080 ) works fine. -Yes, we are able to connect from Apache server to Tomcat server over JK connector port 8009 -i haven't done any configuration on the tomcat side. and as for the webserver, i have attached the workers.properties file - Thread dump of the tomcat ( during the hung condition ) does not show anything unusual. there are few normal requests. -There is no error on mod_jk.log file, ( log level is set to error mode), on the apache error log we see the following error. [Mon Jul 09 11:27:19 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:19 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network - YES, restarting the apache server is the only option fix this behaviour. This setup is suppose to go to Staging, so for now, we are using JK2 connector and it is running fine, but we want to back to JK 1.2 as JK2 is not supported any more. with the jk2 connector the system is fine for almost 18 hours now. http://www.nabble.com/file/p11515063/workers.properties workers.properties Rainer Jung-3 wrote: - Which version of Java? - What does a thread dump of Tomcat give (kill -QUIT, result goes to catalina.out) - Does direct connect to Tomcat http connector still work? - Can you still connect from the Apache server to the Tomcat jk connector via telnet? - What's the config of your jk conector (tomcat side) and of mod_jk? - What are the contents of the mod_jk log file and the Apache error log? - If it happens, will restarting only Apache fix the hang? nageshsrao wrote: we have upgraded our environment to following software specs. on Redhat AS 3.0 (update 8), tomcat 5.5.23, apache 2.0.46 and JK 1.2.23 plugin. intermittently we see the following error in the error log of tomcat and then the whole site just gets hung. and no users will be to procede with any operation. this is suppose to go to Staging, could help us getting this resolved. [Mon Jul 09 11:27:19 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:19 2007
core_output_filter error for apache/tomcat JK connector
we have upgraded our environment to following software specs. on Redhat AS 3.0 (update 8), tomcat 5.5.23, apache 2.0.46 and JK 1.2.23 plugin. intermittently we see the following error in the error log of tomcat and then the whole site just gets hung. and no users will be to procede with any operation. this is suppose to go to Staging, could help us getting this resolved. [Mon Jul 09 11:27:19 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:19 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network let us know if any further information required, i will upload it immediately. -- View this message in context: http://www.nabble.com/core_output_filter-error-for-apache-tomcat-JK-connector-tf4048993.html#a11500807 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: core_output_filter error for apache/tomcat JK connector
Hi, here are further information about the environment and answers to the below question -Tomcat and apache are on two different servers ( separated by a firewall) -Tomcat 5.5.23 is running on Sun JDK 1.5.0_12 -Direct connection to tomcat http connector ( 8080 ) works fine. -Yes, we are able to connect from Apache server to Tomcat server over JK connector port 8009 -i haven't done any configuration on the tomcat side. and as for the webserver, i have attached the workers.properties file - Thread dump of the tomcat ( during the hung condition ) does not show anything unusual. there are few normal requests. -There is no error on mod_jk.log file, ( log level is set to error mode), on the apache error log we see the following error. [Mon Jul 09 11:27:19 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:19 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network - YES, restarting the apache server is the only option fix this behaviour. This setup is suppose to go to Staging, so for now, we are using JK2 connector and it is running fine, but we want to back to JK 1.2 as JK2 is not supported any more. with the jk2 connector the system is fine for almost 18 hours now. http://www.nabble.com/file/p11515063/workers.properties workers.properties Rainer Jung-3 wrote: - Which version of Java? - What does a thread dump of Tomcat give (kill -QUIT, result goes to catalina.out) - Does direct connect to Tomcat http connector still work? - Can you still connect from the Apache server to the Tomcat jk connector via telnet? - What's the config of your jk conector (tomcat side) and of mod_jk? - What are the contents of the mod_jk log file and the Apache error log? - If it happens, will restarting only Apache fix the hang? nageshsrao wrote: we have upgraded our environment to following software specs. on Redhat AS 3.0 (update 8), tomcat 5.5.23, apache 2.0.46 and JK 1.2.23 plugin. intermittently we see the following error in the error log of tomcat and then the whole site just gets hung. and no users will be to procede with any operation. this is suppose to go to Staging, could help us getting this resolved. [Mon Jul 09 11:27:19 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:19 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Mon Jul 09 11:27:23 2007] [info] (32)Broken pipe: core_output_filter: writing data to the network let us know if any further information required, i will upload it immediately. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/core_output_filter-error-for-apache-tomcat-JK-connector-tf4048993.html#a11515063 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
