[jira] [Commented] (OAK-5001) ISE and SNFE when running offline compaction on sidegraded SegmentStore
[ https://issues.apache.org/jira/browse/OAK-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607883#comment-15607883 ] Andrei Dulceanu commented on OAK-5001: -- [~mduerig], with your patch everything works smoothly. > ISE and SNFE when running offline compaction on sidegraded SegmentStore > --- > > Key: OAK-5001 > URL: https://issues.apache.org/jira/browse/OAK-5001 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Affects Versions: Segment Tar 0.0.16 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Blocker > Fix For: 1.6, 1.5.13 > > Attachments: temp.patch > > > {code} > java -jar oak-run-1.6-SNAPSHOT.jar compact > some-instance/sidegraded-repository/segmentstore/ > {code} > {code} > size 4.4 GB (4435703980 bytes) > -> compacting > 11:06:28.253 [main] ERROR o.a.jackrabbit.oak.segment.SegmentId - Segment not > found: fafea2ad-73c2-457e-a6c4-5578c0ffb20c. SegmentId age=7ms > org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment > fafea2ad-73c2-457e-a6c4-5578c0ffb20c not found > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:596) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getGcGeneration(SegmentId.java:213) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.getGcGeneration(FileStore.java:269) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.access$000(FileStore.java:95) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$GarbageCollector.compact(FileStore.java:840) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.compact(FileStore.java:400) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:117) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:109) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:203) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:76) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Mode.execute(Mode.java:63) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Main.main(Main.java:49) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > Caused by: java.util.concurrent.ExecutionException: > java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:103) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:531) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 12 common frames omitted > Caused by: java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.Segment.readRecordNumberOffsets(Segment.java:242) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.segment.Segment.(Segment.java:201) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:548) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:531) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:98) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 13 common frames omitted > {code} > {code} > 11:09:47.047 [main] ERROR o.a.j.oak.segment.file.FileStore - TarMK GC #0: > compaction encountered an error > java.lang.IllegalStateException: invalid record number > at org.apache.jackrabbit.oak.segment.Segment.pos(Segment.java:338) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.Segment.readRecordId(Segment.java:462) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentNodeState.getTemplateId(SegmentNodeState.java:93) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:102) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at >
[jira] [Commented] (OAK-5001) ISE and SNFE when running offline compaction on sidegraded SegmentStore
[ https://issues.apache.org/jira/browse/OAK-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15606543#comment-15606543 ] Michael Dürig commented on OAK-5001: Reported OAK-5004 to improve this. > ISE and SNFE when running offline compaction on sidegraded SegmentStore > --- > > Key: OAK-5001 > URL: https://issues.apache.org/jira/browse/OAK-5001 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Affects Versions: Segment Tar 0.0.16 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Blocker > Fix For: 1.6, 1.5.13 > > Attachments: temp.patch > > > {code} > java -jar oak-run-1.6-SNAPSHOT.jar compact > some-instance/sidegraded-repository/segmentstore/ > {code} > {code} > size 4.4 GB (4435703980 bytes) > -> compacting > 11:06:28.253 [main] ERROR o.a.jackrabbit.oak.segment.SegmentId - Segment not > found: fafea2ad-73c2-457e-a6c4-5578c0ffb20c. SegmentId age=7ms > org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment > fafea2ad-73c2-457e-a6c4-5578c0ffb20c not found > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:596) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getGcGeneration(SegmentId.java:213) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.getGcGeneration(FileStore.java:269) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.access$000(FileStore.java:95) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$GarbageCollector.compact(FileStore.java:840) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.compact(FileStore.java:400) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:117) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:109) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:203) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:76) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Mode.execute(Mode.java:63) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Main.main(Main.java:49) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > Caused by: java.util.concurrent.ExecutionException: > java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:103) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:531) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 12 common frames omitted > Caused by: java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.Segment.readRecordNumberOffsets(Segment.java:242) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.segment.Segment.(Segment.java:201) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:548) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:531) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:98) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 13 common frames omitted > {code} > {code} > 11:09:47.047 [main] ERROR o.a.j.oak.segment.file.FileStore - TarMK GC #0: > compaction encountered an error > java.lang.IllegalStateException: invalid record number > at org.apache.jackrabbit.oak.segment.Segment.pos(Segment.java:338) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.Segment.readRecordId(Segment.java:462) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentNodeState.getTemplateId(SegmentNodeState.java:93) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:102) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at >
[jira] [Commented] (OAK-5001) ISE and SNFE when running offline compaction on sidegraded SegmentStore
[ https://issues.apache.org/jira/browse/OAK-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15606529#comment-15606529 ] Michael Dürig commented on OAK-5001: [~dulceanu], can you confirm that those exceptions go away if you run with the attached patch? If so we need to see how to get those changes released. > ISE and SNFE when running offline compaction on sidegraded SegmentStore > --- > > Key: OAK-5001 > URL: https://issues.apache.org/jira/browse/OAK-5001 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Affects Versions: Segment Tar 0.0.16 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Blocker > Fix For: 1.6, 1.5.13 > > Attachments: temp.patch > > > {code} > java -jar oak-run-1.6-SNAPSHOT.jar compact > some-instance/sidegraded-repository/segmentstore/ > {code} > {code} > size 4.4 GB (4435703980 bytes) > -> compacting > 11:06:28.253 [main] ERROR o.a.jackrabbit.oak.segment.SegmentId - Segment not > found: fafea2ad-73c2-457e-a6c4-5578c0ffb20c. SegmentId age=7ms > org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment > fafea2ad-73c2-457e-a6c4-5578c0ffb20c not found > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:596) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getGcGeneration(SegmentId.java:213) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.getGcGeneration(FileStore.java:269) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.access$000(FileStore.java:95) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$GarbageCollector.compact(FileStore.java:840) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.compact(FileStore.java:400) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:117) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:109) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:203) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:76) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Mode.execute(Mode.java:63) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Main.main(Main.java:49) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > Caused by: java.util.concurrent.ExecutionException: > java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:103) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:531) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 12 common frames omitted > Caused by: java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.Segment.readRecordNumberOffsets(Segment.java:242) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.segment.Segment.(Segment.java:201) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:548) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:531) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:98) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 13 common frames omitted > {code} > {code} > 11:09:47.047 [main] ERROR o.a.j.oak.segment.file.FileStore - TarMK GC #0: > compaction encountered an error > java.lang.IllegalStateException: invalid record number > at org.apache.jackrabbit.oak.segment.Segment.pos(Segment.java:338) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.Segment.readRecordId(Segment.java:462) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentNodeState.getTemplateId(SegmentNodeState.java:93) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at >
[jira] [Commented] (OAK-5001) ISE and SNFE when running offline compaction on sidegraded SegmentStore
[ https://issues.apache.org/jira/browse/OAK-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15606526#comment-15606526 ] Michael Dürig commented on OAK-5001: This growth is caused by checkpoints: the repository I was using has 2 checkpoints, which offline compaction apparently exploded into 3 full copies. When I first remove the checkpoints the repository size stays the same after offline compaction. > ISE and SNFE when running offline compaction on sidegraded SegmentStore > --- > > Key: OAK-5001 > URL: https://issues.apache.org/jira/browse/OAK-5001 > Project: Jackrabbit Oak > Issue Type: Bug > Components: run >Affects Versions: Segment Tar 0.0.16 >Reporter: Andrei Dulceanu >Assignee: Andrei Dulceanu >Priority: Blocker > Fix For: 1.6, 1.5.13 > > Attachments: temp.patch > > > {code} > java -jar oak-run-1.6-SNAPSHOT.jar compact > some-instance/sidegraded-repository/segmentstore/ > {code} > {code} > size 4.4 GB (4435703980 bytes) > -> compacting > 11:06:28.253 [main] ERROR o.a.jackrabbit.oak.segment.SegmentId - Segment not > found: fafea2ad-73c2-457e-a6c4-5578c0ffb20c. SegmentId age=7ms > org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment > fafea2ad-73c2-457e-a6c4-5578c0ffb20c not found > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:596) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentId.getGcGeneration(SegmentId.java:213) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.getGcGeneration(FileStore.java:269) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.access$000(FileStore.java:95) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$GarbageCollector.compact(FileStore.java:840) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.compact(FileStore.java:400) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.compact(Compact.java:117) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:109) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.SegmentTarUtils.compact(SegmentTarUtils.java:203) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.run.CompactCommand.execute(CompactCommand.java:76) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Mode.execute(Mode.java:63) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.run.Main.main(Main.java:49) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > Caused by: java.util.concurrent.ExecutionException: > java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:103) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:531) > [oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 12 common frames omitted > Caused by: java.lang.ArrayIndexOutOfBoundsException: -81 > at > org.apache.jackrabbit.oak.segment.Segment.readRecordNumberOffsets(Segment.java:242) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at org.apache.jackrabbit.oak.segment.Segment.(Segment.java:201) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:548) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.file.FileStore$8.call(FileStore.java:531) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:98) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > ... 13 common frames omitted > {code} > {code} > 11:09:47.047 [main] ERROR o.a.j.oak.segment.file.FileStore - TarMK GC #0: > compaction encountered an error > java.lang.IllegalStateException: invalid record number > at org.apache.jackrabbit.oak.segment.Segment.pos(Segment.java:338) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.Segment.readRecordId(Segment.java:462) > ~[oak-run-1.6-SNAPSHOT.jar:1.6-SNAPSHOT] > at > org.apache.jackrabbit.oak.segment.SegmentNodeState.getTemplateId(SegmentNodeState.java:93) >