Your QueuedThreadPool is way too small. our advice for a minimum number is calculated based on your environment.
minimum = (number of cpu cores * (acceptors + selectors)) + 8 With your setup now, you are starving jetty's ability to process the networking. When a task to cleanup the networking channel was offered to the threadpool it likely failed because the threadpool was exhausted. -- Joakim Erdfelt <[email protected]> webtide.com <http://www.webtide.com/> - intalio.com/jetty Expert advice, services and support from from the Jetty & CometD experts eclipse.org/jetty - cometd.org On Thu, Apr 17, 2014 at 5:27 AM, <[email protected]> wrote: > Hello! > We have a web service (Jersey based) on *Jetty 9.1.3.v2014022* which > somehow leaves sockets open. After a while we realized that problem occurs > only in our production environment where we have a monitoring daemon > calling the service with a simple ping request every 1 minute and a > balancer doing the same thing every 6 seconds. The interesting part is the > balancer ends its requests with RST (not FIN/ACK). No one else uses the > service yet. > The service is pretty simple and is going to be called once a day for > integration purposes so the configuration is fairly modest: 1 acceptor, 2 > selectors and 2 threads for request handlers. Omitting irrelevant code it's > roughly going like this: > > final QueuedThreadPool threadPool = new QueuedThreadPool(5); > final Server server = new Server(threadPool); > final Connector[] connectors = new Connector[1]; > final ServerConnector serverConnector = new ServerConnector(server, 1, 2); > serverConnector.setSoLingerTime(0); > serverConnector.setPort(35980); > connectors[0] = serverConnector; > server.setConnectors(connectors); > final ServletContextHandler context = new > ServletContextHandler(ServletContextHandler.NO_SESSIONS); > context.setContextPath("/"); > ServletHolder restApiHolder = ...; > context.addServlet(restApiHolder, "/*"); > server.setHandler(context); > server.start(); > > The problem occurs pretty rarely. Here what we found out so far: > 1) The service stops closing sockets immediately after two consequent > exceptions: > > [2014-04-14 12:29:50,552] DEBUG [qtp782189620-16 - /ping] write exception > org.eclipse.jetty.io.EofException > at > org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:189) > at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:335) > at > org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:125) > at > org.eclipse.jetty.server.HttpConnection$CommitCallback.process(HttpConnection.java:555) > at > org.eclipse.jetty.util.IteratingCallback.processIterations(IteratingCallback.java:166) > at > org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:126) > at > org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:296) > at > org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:715) > at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:751) > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:130) > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:124) > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:356) > at > java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154) > at > org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:307) > at > org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:261) > at > org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:276) > at > org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:835) > at > org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:411) > at > org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:645) > at > org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:381) > at > org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:371) > at > org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:262) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318) > at > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236) > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010) > at > org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345) > at > org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:711) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1644) > at > ru.yandex.bayan2.integration.servlet.LoggingFilter.doFilter(LoggingFilter.java:27) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1615) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) > at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1112) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:479) > at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1046) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) > at org.eclipse.jetty.server.Server.handle(Server.java:462) > at > org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:281) > at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) > at > org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) > at java.lang.Thread.run(Thread.java:744) > > Caused by: java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) > at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) > at sun.nio.ch.IOUtil.write(IOUtil.java:148) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524) > at > org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:169) > ... 50 more > > 2) When it starts doing so, we get lots of such exceptions: > > [2014-04-14 12:41:55,524] DEBUG [Scheduler-340008923] > SelectChannelEndPoint@f67ae6c{/10.0.0.1:60468<->35980,Open,in,out,-,-,30000,HttpConnection}{io=0,kio=0,kro=1} > idle timeout expired > [2014-04-14 12:41:55,524] DEBUG [Scheduler-340008923] ignored: > WriteFlusher@7fdc861b{IDLE} java.util.concurrent.TimeoutException: Idle > timeout expired: 30001/30000 ms > [2014-04-14 12:41:55,529] DEBUG [Scheduler-340008923] > SelectChannelEndPoint@69f1af68{/10.0.0.1:48730<->35980,Open,in,out,-,-,30000,HttpConnection}{io=0,kio=0,kro=1} > idle timeout check, elapsed: 30000 ms, remaining: 0 ms > > but sockets never get closed. > 3) org.eclipse.jetty.io.SelectorManager reports a growing number of > selected events: > > ... > LOG.debug("Selector loop woken up from select, {}/{} selected", selected, > _selector.keys().size()); > ... > > 4) Sockets left in CLOSE_WAIT state. > As a result we have a growing number of sockets and finally "Too many open > files" exception. > So the questions are > 1) Where can I dig further? > 2) Why > AbstractEndPoint#onIdleExpired(TimeoutException)<https://github.com/eclipse/jetty.project/blob/86d13b91a574096b442ad902b4f0b5b94db6a9d0/jetty-io/src/main/java/org/eclipse/jetty/io/AbstractEndPoint.java#L149> > doesn't > simply close itself. Instead it checks some conditions before doing that: > > @Override > protected void onIdleExpired(TimeoutException timeout) > { > boolean output_shutdown=isOutputShutdown(); > boolean input_shutdown=isInputShutdown(); > _fillInterest.onFail(timeout); > _writeFlusher.onFail(timeout); > if (isOpen() && output_shutdown || input_shutdown) > close(); > } > > 3) Can this help us? > > ServerConnector connector = new ServerConnector(...) { > @Override > protected SelectChannelEndPoint newEndPoint(SocketChannel > channel, SelectorManager.ManagedSelector selectSet, SelectionKey key) > throws IOException { > return new SelectChannelEndPoint(channel, selectSet, key, > getScheduler(), getIdleTimeout()) { > @Override > protected void onIdleExpired(TimeoutException timeout) > { > super.onIdleExpired(timeout); > close(); > } > }; > } > }; > > Thanks. > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > >
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
