I found locking issues reported against 1.2 and 1.4 on the mailing lists, but
not against 1.3.
We are running five Solr-1.3 shards under OSX on Tomcat under JDK-1.6 without
any problems for quite a while. We copied the instance over to a Linux
environment and now observe occasional deadlocks.
This shard would lock and never recover. Each time a different shard would
would lock.
Any ideas or recommendations?
Regards,
Bernhard
Below the stack trace:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.2-b04 mixed mode):
"http-12003-2" daemon prio=10 tid=0x000000005c45f800 nid=0x573 in Object.wait()
[0x0000000042391000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaaea0bf008> (a
org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:423)
- locked <0x00002aaaea0bf008> (a
org.apache.tomcat.util.net.JIoEndpoint$Worker)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:449)
at java.lang.Thread.run(Thread.java:619)
"http-12003-1" daemon prio=10 tid=0x000000005c714000 nid=0x50c9 waiting on
condition [0x000000004228f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaae19f2238> (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:389)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:104)
at
org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:113)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
at
org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)
at java.lang.Thread.run(Thread.java:619)
"http-12003-Acceptor-0" daemon prio=10 tid=0x00002aab181de800 nid=0x5088
runnable [0x0000000041831000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x00002aaac6b0e4c0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
at
org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:317)
at java.lang.Thread.run(Thread.java:619)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10
tid=0x00002aab180d2000 nid=0x5087 runnable [0x0000000041db1000]
java.lang.Thread.State: RUNNABLE
at java.lang.StringCoding.set(StringCoding.java:53)
at java.lang.StringCoding.encode(StringCoding.java:270)
at java.lang.String.getBytes(String.java:946)
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
at java.io.File.exists(File.java:733)
at
org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1055)
- locked <0x00002aaac59cab28> (a org.apache.catalina.startup.HostConfig)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1271)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:296)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)
at java.lang.Thread.run(Thread.java:619)
"pool-3-thread-1" prio=10 tid=0x00002aab1824f000 nid=0x5080 waiting on
condition [0x0000000040c43000]
java.lang.Thread.State: RUNNABLE
at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
at
org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
at
org.apache.lucene.index.TermInfosReader.ensureIndexIsRead(TermInfosReader.java:177)
- locked <0x00002aaae19f0790> (a
org.apache.lucene.index.TermInfosReader)
at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:224)
at
org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:331)
at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:709)
at
org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.<init>(MultiSegmentReader.java:463)
at
org.apache.lucene.index.MultiSegmentReader.terms(MultiSegmentReader.java:368)
at
org.apache.lucene.search.FieldCacheImpl$10.createValue(FieldCacheImpl.java:363)
at
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
- locked <0x00002aaadf827cd8> (a
org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder)
at
org.apache.lucene.search.FieldCacheImpl.getStringIndex(FieldCacheImpl.java:352)
at
org.apache.lucene.search.FieldSortedHitQueue.comparatorString(FieldSortedHitQueue.java:416)
at
org.apache.lucene.search.FieldSortedHitQueue$1.createValue(FieldSortedHitQueue.java:207)
at
org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:72)
- locked <0x00002aaadf827b70> (a
org.apache.lucene.search.FieldCacheImpl$CreationPlaceholder)
at
org.apache.lucene.search.FieldSortedHitQueue.getCachedComparator(FieldSortedHitQueue.java:168)
at
org.apache.lucene.search.FieldSortedHitQueue.<init>(FieldSortedHitQueue.java:56)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:907)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:838)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:269)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:160)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1228)
at
org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:50)
at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1051)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-1" prio=10 tid=0x00002aab18222800 nid=0x507d waiting on
condition [0x000000004154b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaac5ac21e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Low Memory Detector" daemon prio=10 tid=0x000000005c30d000 nid=0x5074 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x000000005c30b000 nid=0x5070 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000005c306000 nid=0x5069 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x000000005c304000 nid=0x5063 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x000000005c2df800 nid=0x505d in Object.wait()
[0x0000000041aaf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaac67a13a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaac67a13a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x000000005c2dd800 nid=0x5059 in
Object.wait() [0x00000000419ae000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaac67a1160> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x000000005c27d800 nid=0x503a runnable [0x0000000040895000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x00002aaac90350a8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at
org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:647)
at org.apache.catalina.startup.Catalina.start(Catalina.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
"VM Thread" prio=10 tid=0x000000005c2d9000 nid=0x5056 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c290800 nid=0x5047
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c292000 nid=0x5050
runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c294000 nid=0x5051
runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c296000 nid=0x5052
runnable
"VM Periodic Task Thread" prio=10 tid=0x000000005c310000 nid=0x5078 waiting on
condition
JNI global references: 1203
Heap
PSYoungGen total 431360K, used 409536K [0x00002aaaf3840000,
0x00002aab13840000, 0x00002aab13840000)
eden space 409536K, 100% used
[0x00002aaaf3840000,0x00002aab0c830000,0x00002aab0c830000)
from space 21824K, 0% used
[0x00002aab122f0000,0x00002aab122f0000,0x00002aab13840000)
to space 21824K, 0% used
[0x00002aab10da0000,0x00002aab10da0000,0x00002aab122f0000)
PSOldGen total 1048576K, used 893874K [0x00002aaab3840000,
0x00002aaaf3840000, 0x00002aaaf3840000)
object space 1048576K, 85% used
[0x00002aaab3840000,0x00002aaaea12cb28,0x00002aaaf3840000)
PSPermGen total 21248K, used 18913K [0x00002aaaae440000,
0x00002aaaaf900000, 0x00002aaab3840000)
object space 21248K, 89% used
[0x00002aaaae440000,0x00002aaaaf6b8438,0x00002aaaaf900000)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
