The file descriptor peak show up in our monitoring application. We have
some charts showing the number of file descriptors owned by the tomcat
process (ls /proc/$(pgrep -u tomcat7)/fd/ | wc -l).
The calatalina.out log shows errors, the most frequent being a
java.io.IOException: Broken pipe.
Apr 20, 2015 12:11:02 AM org.apache.coyote.AbstractProcessor setErrorState
INFO: An error occurred in processing while on a non-container thread. The
connection will be closed immediately
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:128)
at
org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at
org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:174)
at
org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:163)
at
org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:242)
at
org.apache.coyote.http11.InternalNioOutputBuffer.flush(InternalNioOutputBuffer.java:94)
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:801)
at org.apache.coyote.Response.action(Response.java:172)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331)
at
org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:98)
at
networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:223)
at
com.stickyadstv.adex.AuctioneerResponseWriter.respondToClient(AuctioneerResponseWriter.java:322)
at
com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:24)
at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
at
com.stickyadstv.adex.bidder.openrtb.OpenRTBBid.serializeAdm(OpenRTBBid.java:158)
at
com.stickyadstv.adex.bidder.openrtb.OpenRTBBid$AdmReceived.completed(OpenRTBBid.java:310)
at
com.stickyadstv.adex.bidder.openrtb.OpenRTBBid$AdmReceived.completed(OpenRTBBid.java:254)
at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115)
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:173)
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:355)
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:242)
at
org.apache.http.impl.nio.client.LoggingAsyncRequestExecutor.inputReady(LoggingAsyncRequestExecutor.java:87)
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:264)
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:73)
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:37)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584)
at java.lang.Thread.run(Thread.java:745)
Here a count made on "Exception":
304 java.io.InvalidClassException:
com.stickyadstv.web.commands.request.RequestData; incompatible types for
field time
6477 java.io.IOException: Broken pipe
1 java.io.IOException: Connection reset by peer
3 java.lang.NullPointerException
821 java.nio.channels.AsynchronousCloseException
12 java.nio.channels.ClosedChannelException
1 - locked <0x00000000a503aa38> (a
java.lang.NumberFormatException)
21 org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException
I have uploaded a portion of the catalina log during the test I made:
http://www.filedropper.com/catalinatoday11
Thomas
2015-04-21 16:02 GMT+02:00 Christopher Schultz <ch...@christopherschultz.net
>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Thomas,
>
> On 4/20/15 8:11 AM, Thomas Boniface wrote:
> > I have tried to find help regarding an issue we experience with
> > our platform leading to random file descriptor peaks. This happens
> > more often on heavy load but can also happen on low traffic
> > periods.
> >
> > Our application is using servlet 3.0 async features and an async
> > connector. We noticed that a lot of issues regarding asynchronous
> > feature were fixed between our production version and the last
> > stable build. We decided to give it a try to see if it improves
> > things or at least give clues on what can cause the issue;
> > Unfortunately it did neither.
> >
> > The file descriptor peaks and application blocking happens
> > frequently with this version when it only happens rarely on
> > previous version (tomcat7 7.0.28-4).
> >
> > Tomcat is behind an nginx server. The tomcat connector used is
> > configured as follows:
> >
> > We use an Nio connector: <Connector port="8080"
> > protocol="org.apache.coyote. http11.Http11NioProtocol"
> > selectorTimeout="1000" maxThreads="200" maxHttpHeaderSize="16384"
> > address="127.0.0.1" redirectPort="8443"/>
>
> The default maxConnections is 10000, so nginx can open that many
> connections before Tomcat starts to refuse them.
>
> How are you observing the "file descriptor peak"? Are you getting errors
> ?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJVNlhgAAoJEBzwKT+lPKRYMWIQAKWiP7T8HkZq0L9SmqmZr3rv
> DrlWBVXBq21M3/e0P3MNEnKp73SbZtq82i6Ib58cjBi6DMQQJTGisdb1WUqNkxWL
> f0J0fizl5wwDho9FJzxHzR/uV3Nm67Bx7QzvroEEyAmE/wRXdFFOlq/rSdKWfVDC
> jlBE5Seo+AQiURCONEZ9CYHPwm50yeSy9JzGuH1VXcfUTl0NVXS63vOjLp8XeJKO
> 68jT6CuY5uzjvv6ZXeES73zvcthkCbF1/Si1KSVshQ/+aXAFDJDuXLLx0D7PWNV7
> N6jxVeHOoTdogYtfVyuOhQ4Xu6d9d9NddKC1ycMBeRfJP/5zG3YXHAbDdwWP8Sc7
> ip9Md6Y+KA089bRhQ92+6kWqWqtxx1Rg1lhRPkY9nOFc5kEFFsWT8NIEIVWtaN80
> zcanU29juMtJK/+Ov/vwyHljTxOikl2So3l19K2bBaCa1pDQ+NeKRQq4KISLlfjB
> 05w88zu7uS8gYbf+uiw/TMZte1skT8tR3AD1Ye5XRV22zz+yKy6Z7nAdGg/bvIug
> 8ngVWAQ7mxWt6QAtLRMFS4nw+xBNNNRyMYzvFEkZ6d6Wr67SnQXxRKqAIf8nhZ3h
> tqAnU0iXlhrdQCcVsMq/iv9lMgHo2x1NBNfeClkIz3XDvgVDJBbHNAr49WmlC5/H
> 3xUS2AOTCIJNuK+W1CTm
> =i7t2
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
