[
https://issues.apache.org/jira/browse/OAK-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006660#comment-15006660
]
Alex Parvulescu commented on OAK-3612:
--------------------------------------
one problem with the patch: it introduces a deadlock [1]. run
_SegmentCompactionIT_ with 10 writers for a bit to verify it's not just my
setup, also see the existing warning in the FileStore about possible deadlocks
when locking access to the tracker's writer [0].
I don't disagree with what you are saying, but changes like these have impacts
that we cannot predict, so I'd rather approach this with a bit more caution.
[0]
https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L752
[1]
{code}
Found one Java-level deadlock:
=============================
"RandomWriter":
waiting to lock monitor 0x00007f89ab006518 (object 0x00000007000c73c8, a
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter),
which is held by "RandomWriter"
"RandomWriter":
waiting for ownable synchronizer 0x00000007001311b8, (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "TarMK flush thread
[target/SegmentCompactionIT4821636020606907980dir], active since Mon Nov 16
14:24:48 CET 2015, previous max duration 356ms"
"TarMK flush thread [target/SegmentCompactionIT4821636020606907980dir], active
since Mon Nov 16 14:24:48 CET 2015, previous max duration 356ms":
waiting to lock monitor 0x00007f89ab006518 (object 0x00000007000c73c8, a
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter),
which is held by "RandomWriter"
Java stack information for the threads listed above:
===================================================
"RandomWriter":
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeTemplate(SegmentWriter.java:1157)
- waiting to lock <0x00000007000c73c8> (a
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1326)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeAdded(SegmentWriter.java:1344)
at
org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:413)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1352)
at
org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:417)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1352)
at
org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:417)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1352)
at
org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:417)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1341)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:85)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:214)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:81)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:346)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:337)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.addRandomNode(SegmentCompactionIT.java:550)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.modify(SegmentCompactionIT.java:501)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.access$4(SegmentCompactionIT.java:487)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:466)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:1)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.run(SegmentCompactionIT.java:446)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:457)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"RandomWriter":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007001311b8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.writeSegment(FileStore.java:1257)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.flush(SegmentWriter.java:304)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.prepare(SegmentWriter.java:487)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeTemplate(SegmentWriter.java:1228)
- locked <0x00000007000c73c8> (a
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1326)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:1)
at
org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:73)
at
org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.childNodeChanged(MergingNodeStateDiff.java:92)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531)
at
org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:70)
at
org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.childNodeChanged(MergingNodeStateDiff.java:92)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:488)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compareBranch(MapRecord.java:565)
at
org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:470)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at
org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:70)
at
org.apache.jackrabbit.oak.plugins.commit.MergingNodeStateDiff.merge(MergingNodeStateDiff.java:65)
at
org.apache.jackrabbit.oak.plugins.commit.ConflictHook.processCommit(ConflictHook.java:53)
at
org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:484)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:507)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:563)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:226)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:473)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter$1.call(SegmentCompactionIT.java:1)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.run(SegmentCompactionIT.java:446)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:457)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"TarMK flush thread [target/SegmentCompactionIT4821636020606907980dir], active
since Mon Nov 16 14:24:48 CET 2015, previous max duration 356ms":
at
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.dropCache(SegmentWriter.java:1043)
- waiting to lock <0x00000007000c73c8> (a
org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.clearCache(SegmentTracker.java:191)
- locked <0x000000070017fd08> (a
org.apache.jackrabbit.oak.plugins.segment.SegmentTracker)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:816)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:776)
- locked <0x00000007001b9c48> (a
java.util.concurrent.atomic.AtomicReference)
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore$1.run(FileStore.java:486)
at java.lang.Thread.run(Thread.java:744)
at
org.apache.jackrabbit.oak.plugins.segment.file.BackgroundThread.run(BackgroundThread.java:71)
Found 1 deadlock.
{code}
> The string/template cache of the SegmentWriter should be cleared before
> cleanup
> -------------------------------------------------------------------------------
>
> Key: OAK-3612
> URL: https://issues.apache.org/jira/browse/OAK-3612
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: segmentmk
> Reporter: Francesco Mari
> Fix For: 1.4
>
> Attachments: OAK-3612-01.patch
>
>
> The string/template cache of the SegmentWriter
> (org.apache.jackrabbit.oak.plugins.segment.SegmentWriter#records) is not
> cleared before the cleanup phase. This might maintain in-memory references to
> segments, thus preventing them to be cleaned up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
