Hmm, one thread is actively running a query (stuck building up field cache), while another thread is waiting to acquire lock to commit? Are you sure this is deadlock, vs it taks a really really long time to init the field cache for this one query?
Also, you might want to try the tiny patch on https://issues.apache.org/jira/browse/SOLR-2342 -- this gives commit "priority" over threads just doing searching, when it tries to acquire its lock. Mike http://blog.mikemccandless.com On Mon, May 9, 2011 at 4:42 PM, Bernhard C Gass <bg...@apple.com> wrote: > I found locking issues reported against 1.2 and 1.4 on the mailing lists, but > not against 1.3. > > We are running five Solr-1.3 shards under OSX on Tomcat under JDK-1.6 without > any problems for quite a while. We copied the instance over to a Linux > environment and now observe occasional deadlocks. > > This shard would lock and never recover. Each time a different shard would > would lock. > > Any ideas or recommendations? > > > Regards, > Bernhard > > > Below the stack trace: > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.2-b04 mixed mode): > > "http-12003-2" daemon prio=10 tid=0x000000005c45f800 nid=0x573 in > Object.wait() [0x0000000042391000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaaea0bf008> (a > org.apache.tomcat.util.net.JIoEndpoint$Worker) > at java.lang.Object.wait(Object.java:485) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423) > - locked <0x00002aaaea0bf008> (a > org.apache.tomcat.util.net.JIoEndpoint$Worker) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449) > at java.lang.Thread.run(Thread.java:619) > > "http-12003-1" daemon prio=10 tid=0x000000005c714000 nid=0x50c9 waiting on > condition [0x000000004228f000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaae19f2238> (a > java.util.concurrent.FutureTask$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217) > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:389) > at > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77) > at > org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:104) > at > org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:113) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269) > at > org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454) > at java.lang.Thread.run(Thread.java:619) > > "http-12003-Acceptor-0" daemon prio=10 tid=0x00002aab181de800 nid=0x5088 > runnable [0x0000000041831000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) > - locked <0x00002aaac6b0e4c0> (a java.net.SocksSocketImpl) > at java.net.ServerSocket.implAccept(ServerSocket.java:453) > at java.net.ServerSocket.accept(ServerSocket.java:421) > at > org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) > at > org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:317) > at java.lang.Thread.run(Thread.java:619) > > "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 > tid=0x00002aab180d2000 nid=0x5087 runnable [0x0000000041db1000] > java.lang.Thread.State: RUNNABLE > at java.lang.StringCoding.set(StringCoding.java:53) > at java.lang.StringCoding.encode(StringCoding.java:270) > at java.lang.String.getBytes(String.java:946) > at java.io.UnixFileSystem.getBooleanAttributes0(Native Method) > at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228) > at java.io.File.exists(File.java:733) > at > org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1055) > - locked <0x00002aaac59cab28> (a > org.apache.catalina.startup.HostConfig) > at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1271) > at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296) > at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119) > at > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) > at > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) > at java.lang.Thread.run(Thread.java:619) > > "pool-3-thread-1" prio=10 tid=0x00002aab1824f000 nid=0x5080 waiting on > condition [0x0000000040c43000] > java.lang.Thread.State: RUNNABLE > at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122) > at > org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167) > at > org.apache.lucene.index.TermInfosReader.ensureIndexIsRead(TermInfosReader.java:177) > - locked <0x00002aaae19f0790> (a > org.apache.lucene.index.TermInfosReader) > at > org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:224) > at > org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:331) > at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:709) > at > org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.<init>(MultiSegmentReader.java:463) > at > org.apache.lucene.index.MultiSegmentReader.terms(MultiSegmentReader.java:368) > at > org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:363) > at > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72) > - locked <0x00002aaadf827cd8> (a > org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder) > at > org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352) > at > org.apache.lucene.search.FieldSortedHitQueue.comparatorString(FieldSortedHitQueue.java:416) > at > org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSortedHitQueue.java:207) > at > org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72) > - locked <0x00002aaadf827b70> (a > org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder) > at > org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(FieldSortedHitQueue.java:168) > at > org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue.java:56) > at > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:907) > at > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838) > at > org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269) > at > org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1228) > at > org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:50) > at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1051) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > "pool-1-thread-1" prio=10 tid=0x00002aab18222800 nid=0x507d waiting on > condition [0x000000004154b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaac5ac21e0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > > "Low Memory Detector" daemon prio=10 tid=0x000000005c30d000 nid=0x5074 > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "CompilerThread1" daemon prio=10 tid=0x000000005c30b000 nid=0x5070 waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "CompilerThread0" daemon prio=10 tid=0x000000005c306000 nid=0x5069 waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Signal Dispatcher" daemon prio=10 tid=0x000000005c304000 nid=0x5063 runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > > "Finalizer" daemon prio=10 tid=0x000000005c2df800 nid=0x505d in Object.wait() > [0x0000000041aaf000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaac67a13a0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) > - locked <0x00002aaac67a13a0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > > "Reference Handler" daemon prio=10 tid=0x000000005c2dd800 nid=0x5059 in > Object.wait() [0x00000000419ae000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:485) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock) > > "main" prio=10 tid=0x000000005c27d800 nid=0x503a runnable [0x0000000040895000] > java.lang.Thread.State: RUNNABLE > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) > - locked <0x00002aaac90350a8> (a java.net.SocksSocketImpl) > at java.net.ServerSocket.implAccept(ServerSocket.java:453) > at java.net.ServerSocket.accept(ServerSocket.java:421) > at > org.apache.catalina.core.StandardServer.await(StandardServer.java:389) > at org.apache.catalina.startup.Catalina.await(Catalina.java:647) > at org.apache.catalina.startup.Catalina.start(Catalina.java:607) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) > > "VM Thread" prio=10 tid=0x000000005c2d9000 nid=0x5056 runnable > > "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c290800 nid=0x5047 > runnable > > "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c292000 nid=0x5050 > runnable > > "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c294000 nid=0x5051 > runnable > > "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c296000 nid=0x5052 > runnable > > "VM Periodic Task Thread" prio=10 tid=0x000000005c310000 nid=0x5078 waiting > on condition > > JNI global references: 1203 > > Heap > PSYoungGen total 431360K, used 409536K [0x00002aaaf3840000, > 0x00002aab13840000, 0x00002aab13840000) > eden space 409536K, 100% used > [0x00002aaaf3840000,0x00002aab0c830000,0x00002aab0c830000) > from space 21824K, 0% used > [0x00002aab122f0000,0x00002aab122f0000,0x00002aab13840000) > to space 21824K, 0% used > [0x00002aab10da0000,0x00002aab10da0000,0x00002aab122f0000) > PSOldGen total 1048576K, used 893874K [0x00002aaab3840000, > 0x00002aaaf3840000, 0x00002aaaf3840000) > object space 1048576K, 85% used > [0x00002aaab3840000,0x00002aaaea12cb28,0x00002aaaf3840000) > PSPermGen total 21248K, used 18913K [0x00002aaaae440000, > 0x00002aaaaf900000, 0x00002aaab3840000) > object space 21248K, 89% used > [0x00002aaaae440000,0x00002aaaaf6b8438,0x00002aaaaf900000) > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
