Re: Exception in Tomcat7 when closing stream, server crashes
On 4/30/15, 10:17 AM, "Christopher Schultz" wrote: >-BEGIN PGP SIGNED MESSAGE- >Hash: SHA256 > >Osman, > >On 4/30/15 10:07 AM, Osman Ullah | Ntrepid Corp wrote: >> Just an update. We have been doing some testing and we might not be >> seeing Tomcat actually go down with 7.0.61. The errors are still >> occurring but it is possible it may not be crashing. We were seeing >> the crash with 7.0.56. I¹ll post an update once have done some more >> thorough testing. > >Lots of little things are being cleaned-up in and around the >connectors as well as tcnative lately. Using the latest Tomcat will >improve the stability of tcnative, because if the Java code has been >mis-tracking the state of the connection, the native code can >sometimes fall-over. > >I've tried to prevent actual crashes as much as possible, but the >result in those cases is that the connection is totally trashed. I'm >not sure if Tomcat is recycling those trashed connections, but every >release improves the situation. > >It would be even better if you could move up to Tomcat 8. Lots more >refactoring of the connectors has taken place there and so all >connectors are more stable. > >I'm excited about the upcoming Tomcat 9 because the connector code has >been nearly unified. This ought to make all of the connectors rock-solid >. > >- -chris Chris, We were able to replicate the crash with the versions I mentioned earlier. Do you have any suggestions as to what else we can do? One thing I was going to try was that if I get a ClientAbortException when trying to flush the output stream, I will not close the stream. I just don¹t know if this is safe or it will cause some kind of long term issues. Thanks, Osman smime.p7s Description: S/MIME cryptographic signature
Re: Exception in Tomcat7 when closing stream, server crashes
On 4/30/15, 10:17 AM, "Christopher Schultz" wrote: >Osman, > >Lots of little things are being cleaned-up in and around the >connectors as well as tcnative lately. Using the latest Tomcat will >improve the stability of tcnative, because if the Java code has been >mis-tracking the state of the connection, the native code can >sometimes fall-over. > >I've tried to prevent actual crashes as much as possible, but the >result in those cases is that the connection is totally trashed. I'm >not sure if Tomcat is recycling those trashed connections, but every >release improves the situation. > >It would be even better if you could move up to Tomcat 8. Lots more >refactoring of the connectors has taken place there and so all >connectors are more stable. > >I'm excited about the upcoming Tomcat 9 because the connector code has >been nearly unified. This ought to make all of the connectors rock-solid >. > >- -chris Thanks Chris, we will look into moving to Tomcat8 for our next release. smime.p7s Description: S/MIME cryptographic signature
Re: Exception in Tomcat7 when closing stream, server crashes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Osman, On 4/30/15 10:07 AM, Osman Ullah | Ntrepid Corp wrote: > Just an update. We have been doing some testing and we might not be > seeing Tomcat actually go down with 7.0.61. The errors are still > occurring but it is possible it may not be crashing. We were seeing > the crash with 7.0.56. I’ll post an update once have done some more > thorough testing. Lots of little things are being cleaned-up in and around the connectors as well as tcnative lately. Using the latest Tomcat will improve the stability of tcnative, because if the Java code has been mis-tracking the state of the connection, the native code can sometimes fall-over. I've tried to prevent actual crashes as much as possible, but the result in those cases is that the connection is totally trashed. I'm not sure if Tomcat is recycling those trashed connections, but every release improves the situation. It would be even better if you could move up to Tomcat 8. Lots more refactoring of the connectors has taken place there and so all connectors are more stable. I'm excited about the upcoming Tomcat 9 because the connector code has been nearly unified. This ought to make all of the connectors rock-solid . - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVQjlgAAoJEBzwKT+lPKRY7QQP/3icbNyKKRTpFDiIUT4KEqOe tuetGpc27If1yHuoXAWcmeLmVpXcuM2i4zk0rMZwIhUJT0c5YaNn7tEBft4atCUv 7+cPBAyXb55rTtLra9DUoEl8PU3JXPWTVzWFHLw94MxubA3mHdV2cVal36USzZXS QAbeBADt8szgZ5u2VBMQSqCknVYjR+hUAzqPgrirPx0LlEYR+yFkgtjOfJSTTuL9 XwodnGvzfmzHAqZXW7htkwV7koXZF8018dQ41+sW/vdYq1la05hPG5idSvcYINwY AySJ/ydrv91EdVOOhUR2m3SZMczGulGb18gsjtrgJzx2xVW/seKoCaRIUYpVIhuE 4KM53tM9qkR4QG6Un+NRrKyU+AkEpxKy2cvLatTln27KPel1+VM+CWKeLJ/U9tBo 6yJEygOoy/pWlwiBHbEy9kmF4v9FOY6SUo4u4T91KFDL1pXnAI5W0gAVKG7V4nVC 2FYbhNAhHw+68KH6dn2IU0Co24O+5OTt6Vyu9tzMqdSzEFchfQuLTrNBk3DnueWw BsjrOJ3bq3ZB1AgaKC1WVqEKY7dGyV1zGUjRucTr4dqZ9rzyBmqudFPXP/hvRNYp 4yuxtBxt8jd2ol/0ukzOfb7c5E0ABoGnykQxbUXm0q9ujdEU3a6GSSR03iLd6qrU pq6Mk5YvMIuI3fJufZMt =vDEq -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Exception in Tomcat7 when closing stream, server crashes
Just an update. We have been doing some testing and we might not be seeing Tomcat actually go down with 7.0.61. The errors are still occurring but it is possible it may not be crashing. We were seeing the crash with 7.0.56. I’ll post an update once have done some more thorough testing. Thanks -- Osman Ullah Tech Lead, Software Engineering Ntrepid Corporation, Herndon, VA (t) 571.612.8386 osman.ul...@ntrepidcorp.com On 4/29/15, 5:33 PM, "Osman Ullah | Ntrepid Corp" wrote: > >On 4/29/15, 4:39 PM, "Christopher Schultz" >wrote: >> >>When you say that Tomcat crashes, are you saying that the JVM halts >>with an hs_err_[pid] file, or do you mean you get the above stack >>traces (which are errors, I would not call them crashes). >> >>What exact version of tcnative are you using? APR? What OS are you using >>? >> >>Can you switch to the NIO connector temporarily to see if those >>problems go away? It's possible that this is only a problem with the >>APR connector. >> >>- -chris > >Thanks for the response Chris, > >Yes, eventually it crashes with a hs_err file and has to be restarted. The >errors do not show up when we use NIO so I am pretty sure it¹s a problem >in the APR connector somewhere. APR based Apache Tomcat Native library >1.1.33 using APR version 1.5.1. OS is Centos 6.4. > >Thanks smime.p7s Description: S/MIME cryptographic signature
Re: Exception in Tomcat7 when closing stream, server crashes
On 4/29/15, 4:39 PM, "Christopher Schultz" wrote: > >When you say that Tomcat crashes, are you saying that the JVM halts >with an hs_err_[pid] file, or do you mean you get the above stack >traces (which are errors, I would not call them crashes). > >What exact version of tcnative are you using? APR? What OS are you using >? > >Can you switch to the NIO connector temporarily to see if those >problems go away? It's possible that this is only a problem with the >APR connector. > >- -chris Thanks for the response Chris, Yes, eventually it crashes with a hs_err file and has to be restarted. The errors do not show up when we use NIO so I am pretty sure it¹s a problem in the APR connector somewhere. APR based Apache Tomcat Native library 1.1.33 using APR version 1.5.1. OS is Centos 6.4. Thanks smime.p7s Description: S/MIME cryptographic signature
Re: Exception in Tomcat7 when closing stream, server crashes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Osman, On 4/29/15 10:52 AM, Osman Ullah | Ntrepid Corp wrote: > Hello, > > We are using Tomcat 7.0.61 and we are seeing the following error > in catalina.out: > > Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor > setErrorState INFO: An error occurred in processing while on a > non-container thread. The connection will be closed immediately > java.io.IOException at > rg.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAp rOutp > > utBuffer.java:205) > at > org.apache.coyote.http11.InternalAprOutputBuffer.flush(InternalAprOutp utBuf > > fer.java:109) > at > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11 Proce > > ssor.java:801) > at org.apache.coyote.Response.action(Response.java:172) at > org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:3 63) > > at > org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331 ) > > at > org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStr eam.j > > ava:101) > at > org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:2 64) > > at > org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:19 9) > > at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34) > at > org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1 > > 145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java: > > 615) > at java.lang.Thread.run(Thread.java:745) > > > > Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor > setErrorState INFO: An error occurred in processing while on a > non-container thread. The connection will be closed immediately > java.io.IOException at > org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalA prOut > > putBuffer.java:205) > at > org.apache.coyote.http11.InternalAprOutputBuffer.endRequest(InternalAp rOutp > > utBuffer.java:150) > at > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11 Proce > > ssor.java:762) > at org.apache.coyote.Response.action(Response.java:174) at > org.apache.coyote.Response.finish(Response.java:274) at > org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319 ) > > at > org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStr eam.j > > ava:108) > at > org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:3 08) > > at > org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:19 9) > > at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34) > at > org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1 > > 145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java: > > 615) > at java.lang.Thread.run(Thread.java:745) > > > Apr 29, 2015 2:23:14 PM org.apache.tomcat.util.net.AprEndpoint > processSocketAsync SEVERE: Error allocating socket processor > java.lang.NullPointerException at > org.apache.tomcat.util.net.AprEndpoint.processSocketAsync(AprEndpoint. java: > > 885) > at > org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.ja va:84 > > ) > at > org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11 Proce > > ssor.java:764) > at org.apache.coyote.Response.action(Response.java:174) at > org.apache.coyote.Response.finish(Response.java:274) at > org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319 ) > > at > org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStr eam.j > > ava:108) > at > org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:3 08) > > at > org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:19 9) > > at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34) > at > org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1 > > 145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java: > > 615) > at java.lang.Thread.run(Thread.java:745) > > > > We are also seeing this, which also happens with stream.close() > around the same time: > > Exception in thread "pool-4-thread-3" java.lang.Error: > org.apache.tomcat.jni.Error: 20005: An invalid socket was returned > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1 > > 151) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java: > > 615) > at java.lang.Thread.run(Thread.java:745) Caused by: > org.apache.tomcat.jni.Error: 20005: An invalid socket was returned > at org.apache.tomcat.jni.Socket.sendbb(Native Method) at > org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(Internal
Exception in Tomcat7 when closing stream, server crashes
Hello, We are using Tomcat 7.0.61 and we are seeing the following error in catalina.out: Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor setErrorState INFO: An error occurred in processing while on a non-container thread. The connection will be closed immediately java.io.IOException at rg.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutp utBuffer.java:205) at org.apache.coyote.http11.InternalAprOutputBuffer.flush(InternalAprOutputBuf fer.java:109) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce ssor.java:801) at org.apache.coyote.Response.action(Response.java:172) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331) at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.j ava:101) at org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:264) at org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:199) at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34) at org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) at java.lang.Thread.run(Thread.java:745) Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor setErrorState INFO: An error occurred in processing while on a non-container thread. The connection will be closed immediately java.io.IOException at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOut putBuffer.java:205) at org.apache.coyote.http11.InternalAprOutputBuffer.endRequest(InternalAprOutp utBuffer.java:150) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce ssor.java:762) at org.apache.coyote.Response.action(Response.java:174) at org.apache.coyote.Response.finish(Response.java:274) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.j ava:108) at org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:308) at org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:199) at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34) at org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) at java.lang.Thread.run(Thread.java:745) Apr 29, 2015 2:23:14 PM org.apache.tomcat.util.net.AprEndpoint processSocketAsync SEVERE: Error allocating socket processor java.lang.NullPointerException at org.apache.tomcat.util.net.AprEndpoint.processSocketAsync(AprEndpoint.java: 885) at org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84 ) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce ssor.java:764) at org.apache.coyote.Response.action(Response.java:174) at org.apache.coyote.Response.finish(Response.java:274) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.j ava:108) at org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:308) at org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:199) at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34) at org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) at java.lang.Thread.run(Thread.java:745) We are also seeing this, which also happens with stream.close() around the same time: Exception in thread "pool-4-thread-3" java.lang.Error: org.apache.tomcat.jni.Error: 20005: An invalid socket was returned at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 151) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.tomcat.jni.Error: 20005: An invalid socket was returned at org.apache.tomcat.jni.Socket.sendbb(Native Method) at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOut putBuffer.java:204) at org.apache.coyote.http11.InternalAprOutputBuffer.endRequest(InternalAprOutp utBuffer.java:150) at org.apache.coyote.http11.AbstractHttp11Process
