Re: Exception in Tomcat7 when closing stream, server crashes

2015-05-04 Thread Osman Ullah | Ntrepid Corp

On 4/30/15, 10:17 AM, "Christopher Schultz" 
wrote:

>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA256
>
>Osman,
>
>On 4/30/15 10:07 AM, Osman Ullah | Ntrepid Corp wrote:
>> Just an update. We have been doing some testing and we might not be
>> seeing Tomcat actually go down with 7.0.61. The errors are still
>> occurring but it is possible it may not be crashing. We were seeing
>> the crash with 7.0.56. I¹ll post an update once have done some more
>> thorough testing.
>
>Lots of little things are being cleaned-up in and around the
>connectors as well as tcnative lately. Using the latest Tomcat will
>improve the stability of tcnative, because if the Java code has been
>mis-tracking the state of the connection, the native code can
>sometimes fall-over.
>
>I've tried to prevent actual crashes as much as possible, but the
>result in those cases is that the connection is totally trashed. I'm
>not sure if Tomcat is recycling those trashed connections, but every
>release improves the situation.
>
>It would be even better if you could move up to Tomcat 8. Lots more
>refactoring of the connectors has taken place there and so all
>connectors are more stable.
>
>I'm excited about the upcoming Tomcat 9 because the connector code has
>been nearly unified. This ought to make all of the connectors rock-solid
>.
>
>- -chris


Chris,

We were able to replicate the crash with the versions I mentioned earlier.
Do you have any suggestions as to what else we can do?

One thing I was going to try was that if I get a ClientAbortException when
trying to flush the output stream, I will not close the stream. I just
don¹t know if this is safe or it will cause some kind of long term issues.

Thanks,
Osman


smime.p7s
Description: S/MIME cryptographic signature


Re: Exception in Tomcat7 when closing stream, server crashes

2015-04-30 Thread Osman Ullah | Ntrepid Corp
On 4/30/15, 10:17 AM, "Christopher Schultz" 
wrote:


>Osman,
>
>Lots of little things are being cleaned-up in and around the
>connectors as well as tcnative lately. Using the latest Tomcat will
>improve the stability of tcnative, because if the Java code has been
>mis-tracking the state of the connection, the native code can
>sometimes fall-over.
>
>I've tried to prevent actual crashes as much as possible, but the
>result in those cases is that the connection is totally trashed. I'm
>not sure if Tomcat is recycling those trashed connections, but every
>release improves the situation.
>
>It would be even better if you could move up to Tomcat 8. Lots more
>refactoring of the connectors has taken place there and so all
>connectors are more stable.
>
>I'm excited about the upcoming Tomcat 9 because the connector code has
>been nearly unified. This ought to make all of the connectors rock-solid
>.
>
>- -chris

Thanks Chris, we will look into moving to Tomcat8 for our next release. 


smime.p7s
Description: S/MIME cryptographic signature


Re: Exception in Tomcat7 when closing stream, server crashes

2015-04-30 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Osman,

On 4/30/15 10:07 AM, Osman Ullah | Ntrepid Corp wrote:
> Just an update. We have been doing some testing and we might not be
> seeing Tomcat actually go down with 7.0.61. The errors are still
> occurring but it is possible it may not be crashing. We were seeing
> the crash with 7.0.56. I’ll post an update once have done some more
> thorough testing.

Lots of little things are being cleaned-up in and around the
connectors as well as tcnative lately. Using the latest Tomcat will
improve the stability of tcnative, because if the Java code has been
mis-tracking the state of the connection, the native code can
sometimes fall-over.

I've tried to prevent actual crashes as much as possible, but the
result in those cases is that the connection is totally trashed. I'm
not sure if Tomcat is recycling those trashed connections, but every
release improves the situation.

It would be even better if you could move up to Tomcat 8. Lots more
refactoring of the connectors has taken place there and so all
connectors are more stable.

I'm excited about the upcoming Tomcat 9 because the connector code has
been nearly unified. This ought to make all of the connectors rock-solid
.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org

iQIcBAEBCAAGBQJVQjlgAAoJEBzwKT+lPKRY7QQP/3icbNyKKRTpFDiIUT4KEqOe
tuetGpc27If1yHuoXAWcmeLmVpXcuM2i4zk0rMZwIhUJT0c5YaNn7tEBft4atCUv
7+cPBAyXb55rTtLra9DUoEl8PU3JXPWTVzWFHLw94MxubA3mHdV2cVal36USzZXS
QAbeBADt8szgZ5u2VBMQSqCknVYjR+hUAzqPgrirPx0LlEYR+yFkgtjOfJSTTuL9
XwodnGvzfmzHAqZXW7htkwV7koXZF8018dQ41+sW/vdYq1la05hPG5idSvcYINwY
AySJ/ydrv91EdVOOhUR2m3SZMczGulGb18gsjtrgJzx2xVW/seKoCaRIUYpVIhuE
4KM53tM9qkR4QG6Un+NRrKyU+AkEpxKy2cvLatTln27KPel1+VM+CWKeLJ/U9tBo
6yJEygOoy/pWlwiBHbEy9kmF4v9FOY6SUo4u4T91KFDL1pXnAI5W0gAVKG7V4nVC
2FYbhNAhHw+68KH6dn2IU0Co24O+5OTt6Vyu9tzMqdSzEFchfQuLTrNBk3DnueWw
BsjrOJ3bq3ZB1AgaKC1WVqEKY7dGyV1zGUjRucTr4dqZ9rzyBmqudFPXP/hvRNYp
4yuxtBxt8jd2ol/0ukzOfb7c5E0ABoGnykQxbUXm0q9ujdEU3a6GSSR03iLd6qrU
pq6Mk5YvMIuI3fJufZMt
=vDEq
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Exception in Tomcat7 when closing stream, server crashes

2015-04-30 Thread Osman Ullah | Ntrepid Corp
Just an update. We have been doing some testing and we might not be seeing
Tomcat actually go down with 7.0.61. The errors are still occurring but it
is possible it may not be crashing. We were seeing the crash with 7.0.56.
I’ll post an update once have done some more thorough testing.

Thanks
--
Osman Ullah
Tech Lead, Software Engineering
Ntrepid Corporation, Herndon, VA
(t) 571.612.8386 
osman.ul...@ntrepidcorp.com




On 4/29/15, 5:33 PM, "Osman Ullah | Ntrepid Corp"
 wrote:

>
>On 4/29/15, 4:39 PM, "Christopher Schultz" 
>wrote:
>>
>>When you say that Tomcat crashes, are you saying that the JVM halts
>>with an hs_err_[pid] file, or do you mean you get the above stack
>>traces (which are errors, I would not call them crashes).
>>
>>What exact version of tcnative are you using? APR? What OS are you using
>>?
>>
>>Can you switch to the NIO connector temporarily to see if those
>>problems go away? It's possible that this is only a problem with the
>>APR connector.
>>
>>- -chris
>
>Thanks for the response Chris,
>
>Yes, eventually it crashes with a hs_err file and has to be restarted. The
>errors do not show up when we use NIO so I am pretty sure it¹s a problem
>in the APR connector somewhere. APR based Apache Tomcat Native library
>1.1.33 using APR version 1.5.1. OS is Centos 6.4.
>
>Thanks


smime.p7s
Description: S/MIME cryptographic signature


Re: Exception in Tomcat7 when closing stream, server crashes

2015-04-29 Thread Osman Ullah | Ntrepid Corp

On 4/29/15, 4:39 PM, "Christopher Schultz" 
wrote:
>
>When you say that Tomcat crashes, are you saying that the JVM halts
>with an hs_err_[pid] file, or do you mean you get the above stack
>traces (which are errors, I would not call them crashes).
>
>What exact version of tcnative are you using? APR? What OS are you using
>?
>
>Can you switch to the NIO connector temporarily to see if those
>problems go away? It's possible that this is only a problem with the
>APR connector.
>
>- -chris

Thanks for the response Chris,

Yes, eventually it crashes with a hs_err file and has to be restarted. The
errors do not show up when we use NIO so I am pretty sure it¹s a problem
in the APR connector somewhere. APR based Apache Tomcat Native library
1.1.33 using APR version 1.5.1. OS is Centos 6.4.

Thanks


smime.p7s
Description: S/MIME cryptographic signature


Re: Exception in Tomcat7 when closing stream, server crashes

2015-04-29 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Osman,

On 4/29/15 10:52 AM, Osman Ullah | Ntrepid Corp wrote:
> Hello,
> 
> We are using Tomcat 7.0.61 and we are seeing the following error
> in catalina.out:
> 
> Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor 
> setErrorState INFO: An error occurred in processing while on a
> non-container thread. The connection will be closed immediately 
> java.io.IOException at 
> rg.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAp
rOutp
>
> 
utBuffer.java:205)
> at 
> org.apache.coyote.http11.InternalAprOutputBuffer.flush(InternalAprOutp
utBuf
>
> 
fer.java:109)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11
Proce
>
> 
ssor.java:801)
> at org.apache.coyote.Response.action(Response.java:172) at 
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:3
63)
>
> 
at
> org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331
)
>
> 
at
> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStr
eam.j
>
> 
ava:101)
> at 
> org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:2
64)
>
> 
at
> org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:19
9)
>
> 
at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34)
> at 
> org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52)
>
> 
at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
ava:1
>
> 
145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:
>
> 
615)
> at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor 
> setErrorState INFO: An error occurred in processing while on a
> non-container thread. The connection will be closed immediately 
> java.io.IOException at 
> org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalA
prOut
>
> 
putBuffer.java:205)
> at 
> org.apache.coyote.http11.InternalAprOutputBuffer.endRequest(InternalAp
rOutp
>
> 
utBuffer.java:150)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11
Proce
>
> 
ssor.java:762)
> at org.apache.coyote.Response.action(Response.java:174) at
> org.apache.coyote.Response.finish(Response.java:274) at 
> org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319
)
>
> 
at
> org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStr
eam.j
>
> 
ava:108)
> at 
> org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:3
08)
>
> 
at
> org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:19
9)
>
> 
at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34)
> at 
> org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52)
>
> 
at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
ava:1
>
> 
145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:
>
> 
615)
> at java.lang.Thread.run(Thread.java:745)
> 
> 
> Apr 29, 2015 2:23:14 PM org.apache.tomcat.util.net.AprEndpoint 
> processSocketAsync SEVERE: Error allocating socket processor 
> java.lang.NullPointerException at 
> org.apache.tomcat.util.net.AprEndpoint.processSocketAsync(AprEndpoint.
java:
>
> 
885)
> at 
> org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.ja
va:84
>
> 
)
> at 
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11
Proce
>
> 
ssor.java:764)
> at org.apache.coyote.Response.action(Response.java:174) at
> org.apache.coyote.Response.finish(Response.java:274) at 
> org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319
)
>
> 
at
> org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStr
eam.j
>
> 
ava:108)
> at 
> org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:3
08)
>
> 
at
> org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:19
9)
>
> 
at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34)
> at 
> org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52)
>
> 
at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
ava:1
>
> 
145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:
>
> 
615)
> at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> We are also seeing this, which also happens with stream.close()
> around the same time:
> 
> Exception in thread "pool-4-thread-3" java.lang.Error: 
> org.apache.tomcat.jni.Error: 20005: An invalid socket was returned 
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
ava:1
>
> 
151)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
java:
>
> 
615)
> at java.lang.Thread.run(Thread.java:745) Caused by:
> org.apache.tomcat.jni.Error: 20005: An invalid socket was returned 
> at org.apache.tomcat.jni.Socket.sendbb(Native Method) at 
> org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(Internal

Exception in Tomcat7 when closing stream, server crashes

2015-04-29 Thread Osman Ullah | Ntrepid Corp
Hello,

We are using Tomcat 7.0.61 and we are seeing the following error in
catalina.out:

   Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor
setErrorState
   INFO: An error occurred in processing while on a non-container thread.
The connection will be closed immediately
   java.io.IOException
at
rg.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutp
utBuffer.java:205)
at
org.apache.coyote.http11.InternalAprOutputBuffer.flush(InternalAprOutputBuf
fer.java:109)
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce
ssor.java:801)
at org.apache.coyote.Response.action(Response.java:172)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331)
at
org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.j
ava:101)
at
org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:264)
at
org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:199)
at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34)
at
org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
615)
at java.lang.Thread.run(Thread.java:745)



   Apr 29, 2015 2:23:14 PM org.apache.coyote.AbstractProcessor
setErrorState
   INFO: An error occurred in processing while on a non-container thread.
The connection will be closed immediately
   java.io.IOException
  at
org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOut
putBuffer.java:205)
  at
org.apache.coyote.http11.InternalAprOutputBuffer.endRequest(InternalAprOutp
utBuffer.java:150)
  at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce
ssor.java:762)
  at org.apache.coyote.Response.action(Response.java:174)
  at org.apache.coyote.Response.finish(Response.java:274)
  at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319)
  at
org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.j
ava:108)
  at
org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:308)
  at
org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:199)
  at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34)
  at
org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
145)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
615)
  at java.lang.Thread.run(Thread.java:745)


   Apr 29, 2015 2:23:14 PM org.apache.tomcat.util.net.AprEndpoint
processSocketAsync
   SEVERE: Error allocating socket processor
   java.lang.NullPointerException
  at
org.apache.tomcat.util.net.AprEndpoint.processSocketAsync(AprEndpoint.java:
885)
  at
org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84
)
  at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Proce
ssor.java:764)
  at org.apache.coyote.Response.action(Response.java:174)
  at org.apache.coyote.Response.finish(Response.java:274)
  at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319)
  at
org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.j
ava:108)
  at
org.granite.gravity.AbstractChannel.runReceived(AbstractChannel.java:308)
  at
org.granite.gravity.AbstractChannel.runReceive(AbstractChannel.java:199)
  at org.granite.gravity.AsyncReceiver.doRun(AsyncReceiver.java:34)
  at
org.granite.gravity.AsyncChannelRunner.run(AsyncChannelRunner.java:52)
  at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
145)
  at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
615)
  at java.lang.Thread.run(Thread.java:745)



We are also seeing this, which also happens with stream.close() around the
same time:

   Exception in thread "pool-4-thread-3" java.lang.Error:
org.apache.tomcat.jni.Error: 20005: An invalid socket was returned
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
151)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
615)
 at java.lang.Thread.run(Thread.java:745)
   Caused by: org.apache.tomcat.jni.Error: 20005: An invalid socket was
returned
 at org.apache.tomcat.jni.Socket.sendbb(Native Method)
 at
org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOut
putBuffer.java:204)
 at
org.apache.coyote.http11.InternalAprOutputBuffer.endRequest(InternalAprOutp
utBuffer.java:150)
 at
org.apache.coyote.http11.AbstractHttp11Process