[ https://issues.apache.org/jira/browse/OAK-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Dürig updated OAK-4675: ------------------------------- Attachment: OAK-4675-02.patch Initial patch for [^OAK-4675-02.patch] {{oak-segment}}. There is a problem however. The test occasionally fails with {code} at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT.randomAccessFileConcurrentReadAndLength(CompactionAndCleanupIT.java:711) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) Caused by: java.io.IOException: Stream Closed at java.io.RandomAccessFile.length(Native Method) at org.apache.jackrabbit.oak.plugins.segment.file.FileAccess$Random.length(FileAccess.java:101) at org.apache.jackrabbit.oak.plugins.segment.file.TarReader.loadGraph(TarReader.java:893) at org.apache.jackrabbit.oak.plugins.segment.file.TarReader.getGraph(TarReader.java:877) at org.apache.jackrabbit.oak.plugins.segment.file.TarReader.calculateForwardReferences(TarReader.java:721) at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.includeForwardReferences(FileStore.java:948) at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:878) at org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:690) at org.apache.jackrabbit.oak.plugins.segment.CompactionAndCleanupIT$13.call(CompactionAndCleanupIT.java:687) at java.util.concurrent.FutureTask.run(FutureTask.java:262) {code} This is caused by concurrent calls to {{FileStore.cleanup()}}. [~dulceanu], I think we should avoid concurrent calls to {{FileStore.cleanup()}} and also {{FileStore.compact()}} (and also make them mutually exclusive). This deserves an issue in its own. Once we have this we need to check whether {{randomAccessFileConcurrentReadAndLength()}} still reproduces this issue and otherwise try to rework it. Again I suggest we start with fixing this in {{oak-segment-tar}} followed by {{oak-segment}} followed by the branches. > SNFE thrown while testing FileStore.cleanup() running concurrently with writes > ------------------------------------------------------------------------------ > > Key: OAK-4675 > URL: https://issues.apache.org/jira/browse/OAK-4675 > Project: Jackrabbit Oak > Issue Type: Bug > Components: segment-tar, segmentmk > Affects Versions: 1.0.32, 1.4.6, 1.2.18, Segment Tar 0.0.8 > Reporter: Andrei Dulceanu > Assignee: Andrei Dulceanu > Labels: cleanup, gc > Fix For: Segment Tar 0.0.10 > > Attachments: OAK-4675-01.patch, OAK-4675-02.patch, test-case.patch > > > {{SegmentNotFoundException}} is thrown from time to time in the following > scenario: plenty of concurrent writes (each creating a {{625 bytes}} blob) > interrupted by a cleanup. > Stack trace (including some debugging statements added by me): > {code:java} > Pre cleanup readers: [] > Before cleanup readers: [/Users/dulceanu/work/test-repo/data00000a.tar] > Initial size: 357.4 kB > After cleanup readers: [/Users/dulceanu/work/test-repo/data00000a.tar] > After cleanup size: 357.4 kB > Final size: 361.0 kB > Exception in thread "pool-5-thread-74" > org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Cannot copy > record from a generation that has been gc'ed already > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1207) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1096) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1074) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:987) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.access$700(SegmentWriter.java:379) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$8.execute(SegmentWriter.java:337) > at > org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.execute(SegmentBufferWriterPool.java:105) > at > org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode(SegmentWriter.java:334) > at > org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111) > at > org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:550) > at > org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:571) > at > org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:627) > at > org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:287) > at > org.apache.jackrabbit.oak.segment.CompactionAndCleanupIT$1.run(CompactionAndCleanupIT.java:961) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.jackrabbit.oak.segment.SegmentNotFoundException: > Segment 4fb637cc-5013-4925-ab13-0629c4406481 not found > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1341) > at > org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123) > at > org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:94) > at > org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1199) > ... 18 more > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: Invalid segment format. Dumping segment > 4fb637cc-5013-4925-ab13-0629c4406481 > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000140 39 37 39 31 31 36 30 38 2D 63 31 63 65 2D 34 62 97911608-c1ce-4b > 00000150 35 63 2D 61 36 33 37 2D 39 36 61 65 39 34 38 38 5c-a637-96ae9488 > 00000160 61 37 65 38 2E 30 61 62 34 30 36 38 36 00 00 00 a7e8.0ab40686... > 00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000001A0 00 00 00 00 30 30 30 30 34 30 30 00 30 30 30 30 ....0000400.0000 > 000001B0 30 30 30 00 30 30 30 30 30 30 30 00 30 30 30 30 000.0000000.0000 > 000001C0 30 30 30 31 33 30 30 00 31 32 37 35 34 36 30 33 0001300.12754603 > 000001D0 37 32 32 00 30 31 32 33 30 37 00 20 30 00 00 00 722.012307. 0... > 000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000002A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000002B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > at > org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1015) > at > org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:972) > at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:283) > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:92) > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1275) > ... 21 more > Caused by: java.lang.IllegalStateException: Invalid segment format. Dumping > segment 4fb637cc-5013-4925-ab13-0629c4406481 > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000000F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000140 39 37 39 31 31 36 30 38 2D 63 31 63 65 2D 34 62 97911608-c1ce-4b > 00000150 35 63 2D 61 36 33 37 2D 39 36 61 65 39 34 38 38 5c-a637-96ae9488 > 00000160 61 37 65 38 2E 30 61 62 34 30 36 38 36 00 00 00 a7e8.0ab40686... > 00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000001A0 00 00 00 00 30 30 30 30 34 30 30 00 30 30 30 30 ....0000400.0000 > 000001B0 30 30 30 00 30 30 30 30 30 30 30 00 30 30 30 30 000.0000000.0000 > 000001C0 30 30 30 31 33 30 30 00 31 32 37 35 34 36 30 33 0001300.12754603 > 000001D0 37 32 32 00 30 31 32 33 30 37 00 20 30 00 00 00 722.012307. 0... > 000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 00000290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000002A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > 000002B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > at > com.google.common.base.Preconditions.checkState(Preconditions.java:150) > at org.apache.jackrabbit.oak.segment.Segment.<init>(Segment.java:185) > at > org.apache.jackrabbit.oak.segment.file.FileStore$15.call(FileStore.java:1292) > at > org.apache.jackrabbit.oak.segment.file.FileStore$15.call(FileStore.java:1) > at > org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1011) > ... 25 more > 0 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)