Jetty version: 7.6.2
Java version: 1.6.0 U30
Every once in a while, Jetty all but completely stops responding to HTTP/HTTPS
connections. I got a Java thread dump, and tracing through the locks, it looks
like there's a single org.eclipse.jetty.io.nio.ChannelEndPoint.close() which,
via locks, seems to block all other close()s, bringing everything to a grinding
halt.
There are a ton of threads, all with very similar/identical stack
traces, waiting for one lock:
"qtp683347213-21440" prio=10 tid=0x00002aaab57ac000 nid=0x3172
waiting for monitor entry [0x0000000042d8f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.nio.channels.spi.AbstractSelector.cancel(Unknown
Source)
- waiting to lock <0x00000006e659db48> (a
java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(Unknown
Source)
- locked <0x00000007398808d0> (a
sun.nio.ch.SelectionKeyImpl)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:691)
at
org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
at
org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
at
org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Unknown Source)
Which is held by this thread, which is waiting for another lock:
"qtp683347213-90 Selector7" prio=10 tid=0x000000004f5c3000
nid=0x7773 waiting for monitor entry [0x0000000041bdb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
- waiting to lock <0x00000007398806a0> (a
java.lang.Object)
at sun.nio.ch.EPollSelectorImpl.implDereg(Unknown Source)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown
Source)
- locked <0x00000006e659db48> (a
java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
- locked <0x00000006e659e300> (a sun.nio.ch.Util$2)
- locked <0x00000006e659e2f0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00000006e659daf0> (a
sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(Unknown Source)
at
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:531)
at
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Unknown Source)
Which is held by this thread:
"qtp683347213-19755" prio=10 tid=0x000000005005b800 nid=0x7f5e
runnable [0x00000000440a3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcher.preClose0(Native Method)
at sun.nio.ch.SocketDispatcher.preClose(Unknown Source)
at
sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(Unknown
Source)
- locked <0x00000007398806a0> (a
java.lang.Object)
at
java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(Unknown
Source)
at
java.nio.channels.spi.AbstractInterruptibleChannel.close(Unknown
Source)
- locked <0x000000073987aea0> (a java.lang.Object)
at
org.eclipse.jetty.io.nio.ChannelEndPoint.close(ChannelEndPoint.java:205)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:700)
at
org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
at
org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
at
org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
at java.lang.Thread.run(Unknown Source)
This happens very sporadically, and so I can't reproduce it on demand. I'd
expect that there are certain environmental factors that cause this, but I
don't know what those are.
Any thoughts about this?
_______________________________________________
jetty-users mailing list
[email protected]
https://dev.eclipse.org/mailman/listinfo/jetty-users
