Hi, On Sat, Nov 5, 2016 at 12:37 AM, John Gardiner Myers <[email protected]> wrote: > I've been having problems with some Jetty client (embedded) instances > encounter frequent ClosedByInterruptException errors attempting to query a > different service. > > I have four instances in our testing environment that have gotten into this > state. Three are Jetty 9.3.7, one is Jetty 9.3.6. All are either Java 8u72 > or 8u73. > > The stack trace is: > > java.nio.channels.ClosedByInterruptException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:753) > at > org.eclipse.jetty.io.SelectorManager.finishConnect(SelectorManager.java:337) > at > org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:341) > at > org.eclipse.jetty.io.ManagedSelector.access$900(ManagedSelector.java:56) > at > org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:278) > at > org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:170) > at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume > .java:162) > at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) > at java.lang.Thread.run(Thread.java:745) > > Looking at AbstractInterruptibleChannel.end(boolean) for 8u92, I see what > appears to be a bug in NIO. Once an AbstractInterruptibleChannel interrupts > on a given Thread, all subsequent calls to #end(boolean) from that thread > will throw ClosedByInterruptException. The line "interrupted = null;" in > that method has no effect and probably should have been "this.interrupted = > null;" > > I'm not sure what the source of interruptions are, but I do see > TimeoutException("DNS timeout " + getTimeout() + " ms") exceptions from > within SocketAddressResolver.Async.resolve() being logged.
The next line after the promise is failed in SocketAddressResolver.Async.resolve(), Jetty interrupts the thread that triggered the DNS lookup, in the attempt to return it from what it was doing back into the pool. I did not know about this JDK bug. > So my suspicion is that DNS timeouts might be causing worker threads to get > into a state where they cannot complete opening connections. > > I'd appreciate any suggestions for how to diagnose the problem further or > otherwise proceed. I think you have identified the issue. AFAIK, DNS lookups in Java don't timeout. If your DNS lookup hangs forever, that thread is forever stuck there. Jetty adds a DNS timeout, but that is not enough, since the thread doing the DNS lookup will still be stuck. So Jetty tries to interrupt it. However, this hits the JDK bug. We can make the interruption configurable (or perhaps removing it). Would you be able to modify the code (or pass your own implementation of SocketAddressResolver - basically the implementation in Async without the interruption) and verify that without interruption things work ? What would the DNS thread do after the timeout expire ? -- Simone Bordet ---- http://cometd.org http://webtide.com Developer advice, training, services and support from the Jetty & CometD experts. _______________________________________________ jetty-users mailing list [email protected] To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jetty-users
