Hello,
Before I start with my question and stack trace, let me give you a bit
of context:
* Tomcat 7.0.61
* OpenJDK1.7.0_79
* Debian7
* Tomcat is running "behind" an nginx server (reverse proxying).
* As we have pretty stringent constraints on response time, nginx is
configured to reply to the browser after 2s if it has not heard back
from Tomcat.
* As implied in the email subject, we are using the http-nio connector.
This is because our code is calling many webservices in parallel and we
don't want to be stuck waiting for one of them (after 300ms we timeout
the request and consider they will not answer).
Now, my problem is that once in a while (with a higher probability under
load), I get this exception:
java.lang.NullPointerException
at
org.apache.tomcat.util.net.NioEndpoint.processSocketAsync(NioEndpoint.java:730)
~[tomcat-coyote.jar:7.0.61]
at
org.apache.coyote.AbstractProcessor.setErrorState(AbstractProcessor.java:84)
~[tomcat-coyote.jar:7.0.61]
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:764)
~[tomcat-coyote.jar:7.0.61]
at org.apache.coyote.Response.action(Response.java:174)
~[tomcat-coyote.jar:7.0.61]
at org.apache.coyote.Response.finish(Response.java:274)
~[tomcat-coyote.jar:7.0.61]
at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319)
~[catalina.jar:7.0.61]
at
org.apache.catalina.connector.CoyoteWriter.close(CoyoteWriter.java:112)
~[catalina.jar:7.0.61]
at
networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:239)
~[HttpCommand.class:?]
at
com.stickyadstv.adex.AuctioneerResponseWriter.respondToClient(AuctioneerResponseWriter.java:323)
~[AuctioneerResponseWriter.class:?]
at
com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.bidder.marketplace.MarketPlaceBidSerializationWriter.respondToClient(MarketPlaceBidSerializationWriter.java:92)
~[MarketPlaceBidSerializationWriter.class:?]
at
com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53)
~[BidSerializationListener.class:?]
at
com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:24)
~[BidSerializationListener.class:?]
at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
~[httpcore-4.4.1.jar:4.4.1]
at
com.stickyadstv.adex.bidder.openrtb.OpenRTBBid.serializeAdm(OpenRTBBid.java:166)
~[OpenRTBBid.class:?]
at
com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:111)
~[AdmReceived.class:?]
at
com.stickyadstv.adex.bidder.openrtb.AdmReceived.completed(AdmReceived.java:32)
~[AdmReceived.class:?]
at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
~[httpcore-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
~[httpasyncclient-4.1.jar:4.1]
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:412)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:305)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:267)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
~[httpasyncclient-4.1.jar:4.1]
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
~[httpasyncclient-4.1.jar:4.1]
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:116)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
~[httpcore-nio-4.4.1.jar:4.4.1]
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
~[httpcore-nio-4.4.1.jar:4.4.1]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
I looked a bit at the NioEndpoint and AbstractProcessor code and it
looks to me like the only possible cause is that
AbstractProcessor.socketWrapper is null at this point. Can someone
confirm my analysis ?
Now, I tried to figure out how this can happen and that's where I got
lost in the code. So, if someone could help me figure out what in our
code could lead to this, that would be very much appreciated.
Last, but not least, there seems to be a correlation between those
exceptions and spikes with the LimitLatch that never go down. My
question is : could this NPE result in the LimitLatch not going down ?
Thanks for your help and obviously, do not hesitate to ask me questions
if you need more information I did not provide.
A.
--
Antoine Bonavita (anto...@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
