There are no requests running against the shards. - Bernhard
On May 10, 2011, at 5:31 AM, Michael McCandless wrote: > Hmm, one thread is actively running a query (stuck building up field > cache), while another thread is waiting to acquire lock to commit? > Are you sure this is deadlock, vs it taks a really really long time to > init the field cache for this one query? > > Also, you might want to try the tiny patch on > https://issues.apache.org/jira/browse/SOLR-2342 -- this gives commit > "priority" over threads just doing searching, when it tries to acquire > its lock. > > Mike > > http://blog.mikemccandless.com > > On Mon, May 9, 2011 at 4:42 PM, Bernhard C Gass <bg...@apple.com> wrote: >> I found locking issues reported against 1.2 and 1.4 on the mailing lists, >> but not against 1.3. >> >> We are running five Solr-1.3 shards under OSX on Tomcat under JDK-1.6 >> without any problems for quite a while. We copied the instance over to a >> Linux environment and now observe occasional deadlocks. >> >> This shard would lock and never recover. Each time a different shard would >> would lock. >> >> Any ideas or recommendations? >> >> >> Regards, >> Bernhard >> >> >> Below the stack trace: >> >> Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.2-b04 mixed mode): >> >> "http-12003-2" daemon prio=10 tid=0x000000005c45f800 nid=0x573 in >> Object.wait() [0x0000000042391000] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> - waiting on <0x00002aaaea0bf008> (a >> org.apache.tomcat.util.net.JIoEndpoint$Worker) >> at java.lang.Object.wait(Object.java:485) >> at >> org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423) >> - locked <0x00002aaaea0bf008> (a >> org.apache.tomcat.util.net.JIoEndpoint$Worker) >> at >> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449) >> at java.lang.Thread.run(Thread.java:619) >> >> "http-12003-1" daemon prio=10 tid=0x000000005c714000 nid=0x50c9 waiting on >> condition [0x000000004228f000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00002aaae19f2238> (a >> java.util.concurrent.FutureTask$Sync) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217) >> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218) >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> at >> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:389) >> at >> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77) >> at >> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:104) >> at >> org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:113) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> at >> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) >> at >> org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) >> at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >> at >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567) >> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) >> at >> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849) >> at >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) >> at >> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454) >> at java.lang.Thread.run(Thread.java:619) >> >> "http-12003-Acceptor-0" daemon prio=10 tid=0x00002aab181de800 nid=0x5088 >> runnable [0x0000000041831000] >> java.lang.Thread.State: RUNNABLE >> at java.net.PlainSocketImpl.socketAccept(Native Method) >> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >> - locked <0x00002aaac6b0e4c0> (a java.net.SocksSocketImpl) >> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >> at java.net.ServerSocket.accept(ServerSocket.java:421) >> at >> org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) >> at >> org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:317) >> at java.lang.Thread.run(Thread.java:619) >> >> "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 >> tid=0x00002aab180d2000 nid=0x5087 runnable [0x0000000041db1000] >> java.lang.Thread.State: RUNNABLE >> at java.lang.StringCoding.set(StringCoding.java:53) >> at java.lang.StringCoding.encode(StringCoding.java:270) >> at java.lang.String.getBytes(String.java:946) >> at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) >> at >> java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228) >> at java.io.File.exists(File.java:733) >> at >> org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1055) >> - locked <0x00002aaac59cab28> (a >> org.apache.catalina.startup.HostConfig) >> at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1271) >> at >> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296) >> at >> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) >> at >> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) >> at >> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) >> at java.lang.Thread.run(Thread.java:619) >> >> "pool-3-thread-1" prio=10 tid=0x00002aab1824f000 nid=0x5080 waiting on >> condition [0x0000000040c43000] >> java.lang.Thread.State: RUNNABLE >> at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122) >> at >> org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167) >> at >> org.apache.lucene.index.TermInfosReader.ensureIndexIsRead(TermInfosReader.java:177) >> - locked <0x00002aaae19f0790> (a >> org.apache.lucene.index.TermInfosReader) >> at >> org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:224) >> at >> org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:331) >> at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:709) >> at >> org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.<init>(MultiSegmentReader.java:463) >> at >> org.apache.lucene.index.MultiSegmentReader.terms(MultiSegmentReader.java:368) >> at >> org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:363) >> at >> org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72) >> - locked <0x00002aaadf827cd8> (a >> org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder) >> at >> org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352) >> at >> org.apache.lucene.search.FieldSortedHitQueue.comparatorString(FieldSortedHitQueue.java:416) >> at >> org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSortedHitQueue.java:207) >> at >> org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72) >> - locked <0x00002aaadf827b70> (a >> org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder) >> at >> org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(FieldSortedHitQueue.java:168) >> at >> org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue.java:56) >> at >> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:907) >> at >> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838) >> at >> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269) >> at >> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160) >> at >> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1228) >> at >> org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:50) >> at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1051) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> "pool-1-thread-1" prio=10 tid=0x00002aab18222800 nid=0x507d waiting on >> condition [0x000000004154b000] >> java.lang.Thread.State: WAITING (parking) >> at sun.misc.Unsafe.park(Native Method) >> - parking to wait for <0x00002aaac5ac21e0> (a >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) >> at >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >> at java.lang.Thread.run(Thread.java:619) >> >> "Low Memory Detector" daemon prio=10 tid=0x000000005c30d000 nid=0x5074 >> runnable [0x0000000000000000] >> java.lang.Thread.State: RUNNABLE >> >> "CompilerThread1" daemon prio=10 tid=0x000000005c30b000 nid=0x5070 waiting >> on condition [0x0000000000000000] >> java.lang.Thread.State: RUNNABLE >> >> "CompilerThread0" daemon prio=10 tid=0x000000005c306000 nid=0x5069 waiting >> on condition [0x0000000000000000] >> java.lang.Thread.State: RUNNABLE >> >> "Signal Dispatcher" daemon prio=10 tid=0x000000005c304000 nid=0x5063 >> runnable [0x0000000000000000] >> java.lang.Thread.State: RUNNABLE >> >> "Finalizer" daemon prio=10 tid=0x000000005c2df800 nid=0x505d in >> Object.wait() [0x0000000041aaf000] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> - waiting on <0x00002aaac67a13a0> (a >> java.lang.ref.ReferenceQueue$Lock) >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) >> - locked <0x00002aaac67a13a0> (a java.lang.ref.ReferenceQueue$Lock) >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) >> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >> >> "Reference Handler" daemon prio=10 tid=0x000000005c2dd800 nid=0x5059 in >> Object.wait() [0x00000000419ae000] >> java.lang.Thread.State: WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> - waiting on <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock) >> at java.lang.Object.wait(Object.java:485) >> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >> - locked <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock) >> >> "main" prio=10 tid=0x000000005c27d800 nid=0x503a runnable >> [0x0000000040895000] >> java.lang.Thread.State: RUNNABLE >> at java.net.PlainSocketImpl.socketAccept(Native Method) >> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >> - locked <0x00002aaac90350a8> (a java.net.SocksSocketImpl) >> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >> at java.net.ServerSocket.accept(ServerSocket.java:421) >> at >> org.apache.catalina.core.StandardServer.await(StandardServer.java:389) >> at org.apache.catalina.startup.Catalina.await(Catalina.java:647) >> at org.apache.catalina.startup.Catalina.start(Catalina.java:607) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) >> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) >> >> "VM Thread" prio=10 tid=0x000000005c2d9000 nid=0x5056 runnable >> >> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c290800 nid=0x5047 >> runnable >> >> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c292000 nid=0x5050 >> runnable >> >> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c294000 nid=0x5051 >> runnable >> >> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c296000 nid=0x5052 >> runnable >> >> "VM Periodic Task Thread" prio=10 tid=0x000000005c310000 nid=0x5078 waiting >> on condition >> >> JNI global references: 1203 >> >> Heap >> PSYoungGen total 431360K, used 409536K [0x00002aaaf3840000, >> 0x00002aab13840000, 0x00002aab13840000) >> eden space 409536K, 100% used >> [0x00002aaaf3840000,0x00002aab0c830000,0x00002aab0c830000) >> from space 21824K, 0% used >> [0x00002aab122f0000,0x00002aab122f0000,0x00002aab13840000) >> to space 21824K, 0% used >> [0x00002aab10da0000,0x00002aab10da0000,0x00002aab122f0000) >> PSOldGen total 1048576K, used 893874K [0x00002aaab3840000, >> 0x00002aaaf3840000, 0x00002aaaf3840000) >> object space 1048576K, 85% used >> [0x00002aaab3840000,0x00002aaaea12cb28,0x00002aaaf3840000) >> PSPermGen total 21248K, used 18913K [0x00002aaaae440000, >> 0x00002aaaaf900000, 0x00002aaab3840000) >> object space 21248K, 89% used >> [0x00002aaaae440000,0x00002aaaaf6b8438,0x00002aaaaf900000) >> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > Bernhard C Gass | Apple Inc. | Enterprise Architect | Retail IS&T | O - 408.974.3224 | C - 650.861.1996
