I know it seems like my problem may not be the same as the original poster, but in investigating this, I did find this Jetty issue that may be related: http://jira.codehaus.org/browse/JETTY-1377
Stephen Duncan Jr www.stephenduncanjr.com On Thu, Aug 4, 2011 at 1:54 PM, Stephen Duncan Jr <stephen.dun...@gmail.com> wrote: > On Thu, Aug 4, 2011 at 10:08 AM, Yonik Seeley > <yo...@lucidimagination.com> wrote: >> >> "ignores" means what? The request hangs? If so, could you get a thread >> dump? >> >> Do queries work (like /solr/select?q=*:*) ? >> >>> thous throwing no errors, no 503's.. It's like the server has a blackout and >>> stares blankly into space. >> >> Are you using a different servlet container than what is shipped with solr? >> If you did start with the solr "example" server, what jetty >> configuration changes have you made? >> >> -Yonik >> http://www.lucidimagination.com >> > > We're seeing something similar here. Not sure exactly what the > circumstances are, but occasionally our Solr 3.3 test instance is > hanging, nothing seems to be happening for several minutes. It does > seem to be happening while data is being added and continuous queries > are being sent. It also may be related to an optimize happening (we > attempt to optimize after adding all the new data from our database). > The last log message is: > > 2011-08-04 13:46:56,418 [qtp30604342-451] INFO > org.apache.solr.core.SolrCore - [report] webapp= path=/update > params={optimize=true&waitSearcher=true&maxSegments=1&waitFlush=true&wt=javabin&version=2} > status=0 QTime=109109 > > Here is our thread dump: > > > 2011-08-04 13:47:16 > Full thread dump Java HotSpot(TM) Client VM (20.1-b02 mixed mode): > > "RMI TCP Connection(13)-172.16.10.102" daemon prio=6 tid=0x47a4a400 > nid=0x1384 runnable [0x4861f000] > java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > - locked <0x183a55a0> (a java.io.BufferedInputStream) > at java.io.FilterInputStream.read(FilterInputStream.java:66) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - <0x183a7c68> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) > > "qtp30604342-451" prio=6 tid=0x475c4800 nid=0x1a58 waiting on > condition [0x4897f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x18214c08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "qtp30604342-450" prio=6 tid=0x47ad1c00 nid=0x1ca4 waiting on > condition [0x49d2f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x18214c08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "qtp30604342-449" prio=6 tid=0x47a57c00 nid=0xb2c waiting on condition > [0x49c2f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x18214c08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "JMX server connection timeout 438" daemon prio=6 tid=0x47604c00 > nid=0x1c88 in Object.wait() [0x48c7f000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at > com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) > - locked <0x18331dc8> (a [I) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "RMI Scheduler(0)" daemon prio=6 tid=0x48fd0400 nid=0x608 waiting on > condition [0x48b7f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x18331e58> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "RMI TCP Accept-0" daemon prio=6 tid=0x48ffd400 nid=0x1e4c runnable > [0x4841f000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) > - locked <0x18331fc8> (a java.net.SocksSocketImpl) > at java.net.ServerSocket.implAccept(ServerSocket.java:462) > at java.net.ServerSocket.accept(ServerSocket.java:430) > at > sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "qtp30604342-424" prio=6 tid=0x48f96400 nid=0x1214 waiting on > condition [0x49a2f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x18214c08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "qtp30604342-423" prio=6 tid=0x46d46400 nid=0x1df4 waiting on > condition [0x4992f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x18214c08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "qtp30604342-92" prio=6 tid=0x478d7400 nid=0x15e4 waiting on condition > [0x49b2f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x18214c08> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at > org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "pool-4-thread-1" prio=6 tid=0x46d00800 nid=0x1bf0 waiting on > condition [0x4887f000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x181a6628> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "qtp30604342-14 Selector0 SelectChannelConnector@127.0.0.1:8080 > STARTED" prio=6 tid=0x47a0c800 nid=0x1e60 runnable [0x4821f000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.WindowsSelectorImpl.discardUrgentData(Native Method) > at > sun.nio.ch.WindowsSelectorImpl.access$2200(WindowsSelectorImpl.java:32) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.processFDSet(WindowsSelectorImpl.java:336) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.processSelectedKeys(WindowsSelectorImpl.java:299) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$2900(WindowsSelectorImpl.java:255) > at > sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:471) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:149) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <0x18217d98> (a sun.nio.ch.Util$2) > - locked <0x18217da8> (a java.util.Collections$UnmodifiableSet) > - locked <0x18217d20> (a sun.nio.ch.WindowsSelectorImpl) > at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:88) > at > org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:475) > at > org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) > at > org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:257) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "qtp30604342-12 Acceptor0 SelectChannelConnector@127.0.0.1:8080 > STARTED" prio=6 tid=0x479f8800 nid=0x1f58 runnable [0x4801f000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) > - locked <0x18223600> (a java.lang.Object) > at > org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) > at > org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) > at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "pool-3-thread-1" prio=6 tid=0x479ec400 nid=0x18f4 waiting on > condition [0x47f1f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x182132a0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "pool-1-thread-1" prio=6 tid=0x479a4400 nid=0x970 waiting on condition > [0x47e1f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x17fdd938> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:662) > > Locked ownable synchronizers: > - None > > "HashSessionScavenger-0" daemon prio=6 tid=0x478e4400 nid=0x154c in > Object.wait() [0x47d0f000] > java.lang.Thread.State: TIMED_WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.util.TimerThread.mainLoop(Timer.java:509) > - locked <0x17e57de8> (a java.util.TaskQueue) > at java.util.TimerThread.run(Timer.java:462) > > Locked ownable synchronizers: > - None > > "Low Memory Detector" daemon prio=6 tid=0x46cf0000 nid=0xcfc runnable > [0x00000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "C1 CompilerThread0" daemon prio=10 tid=0x46cea000 nid=0x1494 waiting > on condition [0x00000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "Attach Listener" daemon prio=10 tid=0x46ce8400 nid=0x1024 waiting on > condition [0x00000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "Signal Dispatcher" daemon prio=10 tid=0x46ce7000 nid=0x13f8 runnable > [0x00000000] > java.lang.Thread.State: RUNNABLE > > Locked ownable synchronizers: > - None > > "Finalizer" daemon prio=8 tid=0x46cd6000 nid=0xb44 in Object.wait() > [0x4705f000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) > - locked <0x17df02c0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > > Locked ownable synchronizers: > - None > > "Reference Handler" daemon prio=10 tid=0x46cd4c00 nid=0xc90 in > Object.wait() [0x46f5f000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x17df0350> (a java.lang.ref.Reference$Lock) > > Locked ownable synchronizers: > - None > > "main" prio=6 tid=0x003e6400 nid=0x880 in Object.wait() [0x0061f000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x17e5ea80> (a java.lang.Object) > at java.lang.Object.wait(Object.java:485) > at > org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:382) > - locked <0x17e5ea80> (a java.lang.Object) > at org.eclipse.jetty.server.Server.join(Server.java:399) > at <ourapp>.App.main(App.java:42) > > Locked ownable synchronizers: > - None > > "VM Thread" prio=10 tid=0x46cd1000 nid=0x1a74 runnable > > "VM Periodic Task Thread" prio=10 tid=0x46cfac00 nid=0x1430 waiting on > condition > > JNI global references: 1665 > > --- > Stephen Duncan Jr > www.stephenduncanjr.com >
