Guido Dunker created OAK-7755:
---------------------------------
Summary: deadlock TarMK flush, lucene
Key: OAK-7755
URL: https://issues.apache.org/jira/browse/OAK-7755
Project: Jackrabbit Oak
Issue Type: Bug
Components: core
Affects Versions: 1.6.1
Environment: apache sling9, oak 1.6.1
Reporter: Guido Dunker
We are getting the following deadlock. Please help! (Production environment)
I have already annotated possible locks and synchronized blocks in between:
{{
"TarMK flush [/opt/condat/epet9/sling/repository/segmentstore]":
waiting to lock Monitor@0x00007fedfc00cc28 (Object@0x00000004795519a8, a
org/apache/jackrabbit/oak/segment/SegmentId),
which is held by "oak-lucene-14"
"oak-lucene-14":
waiting for ownable synchronizer 0x00000003c13818c0, (a
java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync),
which is held by "TarMK flush
[/opt/condat/epet9/sling/repository/segmentstore]"
Thread 28883: (state = BLOCKED)
- org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=12, line=121
(Compiled frame)
synchronized (this)
- org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70
(Compiled frame)
- org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int)
@bci=49, line=57 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int)
@bci=314, line=189 (Compiled frame)
- com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int)
@bci=43, line=828 (Compiled frame)
- com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int,
int) @bci=4, line=695 (Compiled frame)
- com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[])
@bci=5, line=676 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93,
line=103 (Compiled frame)
- org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524
(Compiled frame)
-
org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment,
int) @bci=15, line=212 (Compiled frame)
-
org.apache.jackrabbit.oak.segment.SegmentBlob.readBlobId(org.apache.jackrabbit.oak.segment.Segment,
int) @bci=37, line=167 (Compiled frame)
- org.apache.jackrabbit.oak.segment.file.AbstractFileStore$4.consume(int,
org.apache.jackrabbit.oak.segment.RecordType, int) @bci=24, line=354 (Compiled
frame)
-
org.apache.jackrabbit.oak.segment.Segment.forEachRecord(org.apache.jackrabbit.oak.segment.Segment$RecordConsumer)
@bci=48, line=716 (Compiled frame)
-
org.apache.jackrabbit.oak.segment.file.AbstractFileStore.populateTarBinaryReferences(org.apache.jackrabbit.oak.segment.Segment,
org.apache.jackrabbit.oak.segment.file.TarWriter) @bci=25, line=349 (Compiled
frame)
-
org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(org.apache.jackrabbit.oak.segment.SegmentId,
byte[], int, int) @bci=136, line=657 (Compiled frame)
fileStoreLock.writeLock().lock(); Zeile 639
bis: populateTarBinaryReferences
- org.apache.jackrabbit.oak.segment.SegmentBufferWriter.flush() @bci=383,
line=383 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.flush() @bci=165,
line=148 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentWriter.flush() @bci=4, line=143
(Compiled frame)
- org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=7, line=373
(Compiled frame)
- org.apache.jackrabbit.oak.segment.file.FileStore$7.call() @bci=1, line=370
(Compiled frame)
-
org.apache.jackrabbit.oak.segment.file.TarRevisions.doFlush(java.util.concurrent.Callable)
@bci=25, line=224 (Compiled frame)
-
org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(java.util.concurrent.Callable)
@bci=42, line=212 (Compiled frame)
- org.apache.jackrabbit.oak.segment.file.FileStore.flush() @bci=20, line=370
(Compiled frame)
- org.apache.jackrabbit.oak.segment.file.FileStore$2.run() @bci=15, line=233
(Compiled frame)
- org.apache.jackrabbit.oak.segment.file.SafeRunnable.run() @bci=21, line=67
(Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.runAndReset() @bci=47, line=308 (Compiled
frame)
-
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
@bci=1, line=180 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()
@bci=37, line=294 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)
Locked ownable synchronizers:
- <0x00000003c1361228>, (a
java/util/concurrent/locks/ReentrantLock$NonfairSync)
- <0x00000003c13818c0>, (a
java/util/concurrent/locks/ReentrantReadWriteLock$NonfairSync)
- <0x00000003c1c3a0d8>, (a java/util/concurrent/ThreadPoolExecutor$Worker)
Thread 31035: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int)
@bci=83, line=967 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int)
@bci=10, line=1283 (Compiled frame)
- java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5,
line=727 (Compiled frame)
- org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=158, line=567
(Compiled frame)
fileStoreLock.readLock().lock();
- org.apache.jackrabbit.oak.segment.file.FileStore$8.call() @bci=1, line=542
(Compiled frame)
-
org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(org.apache.jackrabbit.oak.segment.SegmentId,
java.util.concurrent.Callable) @bci=1, line=95 (Compiled frame)
-
org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(org.apache.jackrabbit.oak.segment.SegmentId)
@bci=14, line=542 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentId.getSegment() @bci=38, line=125
(Compiled frame)
synchronized (this)
- org.apache.jackrabbit.oak.segment.Record.getSegment() @bci=4, line=70
(Compiled frame)
- org.apache.jackrabbit.oak.segment.BlockRecord.read(int, byte[], int, int)
@bci=49, line=57 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentStream.read(byte[], int, int)
@bci=314, line=189 (Compiled frame)
- com.google.common.io.ByteStreams.read(java.io.InputStream, byte[], int, int)
@bci=64, line=833 (Compiled frame)
- com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[], int,
int) @bci=4, line=695 (Compiled frame)
- com.google.common.io.ByteStreams.readFully(java.io.InputStream, byte[])
@bci=5, line=676 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentStream.getString() @bci=93,
line=103 (Compiled frame)
- org.apache.jackrabbit.oak.segment.Segment.readString(int) @bci=189, line=524
(Compiled frame)
-
org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(org.apache.jackrabbit.oak.segment.Segment,
int) @bci=15, line=212 (Compiled frame)
- org.apache.jackrabbit.oak.segment.SegmentBlob.length() @bci=124, line=115
(Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.<init>(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String,
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory)
@bci=204, line=409 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.<init>(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeBuilder, java.lang.String,
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$BlobFactory)
@bci=25, line=589 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory.fileLength(java.lang.String)
@bci=64, line=176 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.copyFilesToLocal(org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory$CORFileReference,
boolean, boolean) @bci=195, line=214 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.prefetchIndexFiles()
@bci=96, line=170 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.directory.CopyOnReadDirectory.<init>(org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier,
org.apache.lucene.store.Directory, org.apache.lucene.store.Directory, boolean,
java.lang.String, java.util.concurrent.Executor) @bci=85, line=81 (Compiled
frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(java.lang.String,
org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition,
org.apache.lucene.store.Directory, java.lang.String) @bci=35, line=122
(Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReader(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition,
org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String,
java.lang.String, java.lang.String) @bci=61, line=102 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.reader.DefaultIndexReaderFactory.createReaders(org.apache.jackrabbit.oak.plugins.index.lucene.IndexDefinition,
org.apache.jackrabbit.oak.spi.state.NodeState, java.lang.String) @bci=20,
line=61 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.plugins.index.lucene.reader.LuceneIndexReaderFactory,
org.apache.jackrabbit.oak.plugins.index.lucene.hybrid.NRTIndexFactory) @bci=17,
line=68 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker$1.leave(org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeState) @bci=30, line=132 (Compiled
frame)
-
org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeState) @bci=66, line=153 (Compiled
frame)
-
org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=197, line=415
(Compiled frame)
-
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608
(Compiled frame)
-
org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(java.lang.String,
org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeState) @bci=43, line=148 (Compiled
frame)
-
org.apache.jackrabbit.oak.segment.MapRecord.compare(org.apache.jackrabbit.oak.segment.MapRecord,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=400, line=457
(Compiled frame)
-
org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeStateDiff) @bci=909, line=608
(Compiled frame)
-
org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(org.apache.jackrabbit.oak.spi.commit.Editor,
org.apache.jackrabbit.oak.spi.state.NodeState,
org.apache.jackrabbit.oak.spi.state.NodeState) @bci=34, line=52 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.diffAndUpdate(org.apache.jackrabbit.oak.spi.state.NodeState)
@bci=140, line=142 (Compiled frame)
-
org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.update(org.apache.jackrabbit.oak.spi.state.NodeState)
@bci=36, line=113 (Compiled frame)
- org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=79,
line=135 (Compiled frame)
- org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call() @bci=1,
line=128 (Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=748 (Compiled frame)
Locked ownable synchronizers:
- <0x0000000476194f30>, (a java/util/concurrent/ThreadPoolExecutor$Worker)
}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
