Hi Tobias, I copied some documentation [1] below that was written by Michael Duerig.
If the steps below don¹t work for you then please submit an AEM support ticket so we can have Adobe engineering team assist further. [1] Reverting the Oak repository to fix ³SegmentNotFoundException" ------------------------------------ The check run-mode [2] of oak-run can be used to determine the last known good revision of a segment store, which can subequently be restored manually. This utility can be used to revert a corrupt segment store to its latest good revision change into Oak's working directory (the one containing the crx-quickstart folder) and backup all files in ./crx-quickstart/repository/segmentstore/. 1. Run the consistency check, java -jar oak-run-*.jar check -d1 --bin=-1 -p crx-quickstart/repository/segmentstore/ it will search backwards through the revisions until it finds a consistent one: 14:00:30.783 [main] INFO o.a.j.o.p.s.f.t.ConsistencyChecker - Found latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880 2. Revert the repository to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision. 3. Remove all ./crx-quickstart/repository/segmentstore/*.bak files. 4. Run checkpoint clean-up to remove orphaned checkpoints: java -jar oak-run-*.jar checkpoints ./crx-quickstart/repository/segmentstore rm-unreferenced 5. Finally compact the repository: java -jar oak-run-*.jar compact ./crx-quickstart/repository/segmentstore/ [2] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-run/README.md#check Thanks, Andrew On 11/13/15, 4:44 AM, "Tobias Frech" <[email protected]> wrote: >Hi there! > >We are using Adobe Experience Manager 6.0 which includes Apache OAK >1.0.x and are looking into ways to shrink the size of our repository. >Since offline compaction comes with additonal ops burden online >compaction seems to be a better solution for us. >I enabled online compaction on a test system and triggered a compaction >run. After a while the JVM (-Xmx6g) ran into an OutOfMemory-Error >(Heap). I tried to shutdown AEM cleanly but had to kill the process >after waiting for an hour or so. >Now after trying to start AEM again it does not fully start, some >service seems not to be started anymore. But no error message is logged >by AEM. > >Trying to check the repository I ran the following command and got the >exception below: > >[cq@xxxxxxx crx-quickstart]$ java -jar oak-run-1.0.19.jar backup >./repository/segmentstore /opt/cq/backup/ >Apache Jackrabbit Oak 1.0.19 >05:54:27.495 [main] WARN o.a.j.o.p.segment.file.TarReader - Unexpected >size 1024 of tar file data01186a.tar >05:54:27.510 [main] WARN o.a.j.o.p.segment.file.TarReader - Could not >find a valid tar index in [./repository/segmentstore/data01186a.tar], >recovering... >06:09:05.078 [main] ERROR o.a.j.o.p.segment.SegmentTracker - Segment not >found: 11f36041-cece-48d6-b95c-029be7c7a91c. Creation date delta is 23 ms. >org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: >Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found > at >org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileS >tore.java:855) >~[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(Segmen >tTracker.java:134) >~[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.j >ava:101) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82 >) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.jav >a:52) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream >.java:168) >[oak-run-1.0.19.jar:1.0.19] > at com.google.common.io.ByteStreams.read(ByteStreams.java:828) >[oak-run-1.0.19.jar:1.0.19] > at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303) >[oak-run-1.0.19.jar:1.0.19] > at com.google.common.io.ByteStreams.equal(ByteStreams.java:661) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.j >ava:67) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.j >ava:214) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java >:263) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java >:219) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.j >ava:50) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAd >ded(Compactor.java:140) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:155) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstB >aseState(SegmentNodeState.java:434) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java >:93) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java >:98) >[oak-run-1.0.19.jar:1.0.19] > at >org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreB >ackup.java:78) >[oak-run-1.0.19.jar:1.0.19] > at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193) >[oak-run-1.0.19.jar:1.0.19] > at org.apache.jackrabbit.oak.run.Main.main(Main.java:119) >[oak-run-1.0.19.jar:1.0.19] >Exception in thread "main" >org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: >Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found > at >org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileS >tore.java:855) > at >org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(Segmen >tTracker.java:134) > at >org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.j >ava:101) > at >org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82 >) > at >org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.jav >a:52) > at >org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream >.java:168) > at com.google.common.io.ByteStreams.read(ByteStreams.java:828) > at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303) > at com.google.common.io.ByteStreams.equal(ByteStreams.java:661) > at >org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.j >ava:67) > at >org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.j >ava:214) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java >:263) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java >:219) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.j >ava:50) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAd >ded(Compactor.java:140) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:155) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeA >dded(Compactor.java:167) > at >org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmpt >yState(EmptyNodeState.java:160) > at >org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstB >aseState(SegmentNodeState.java:434) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java >:93) > at >org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java >:98) > at >org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreB >ackup.java:78) > at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193) > at org.apache.jackrabbit.oak.run.Main.main(Main.java:119) > > >What can we do now to get the repository back into a clean state? > >Thanks in advance, >Tobias Frech > >-- >Frech IT GmbH / Am Brünnele 7 / 71642 Ludwigsburg >phone : +49-(0)7141-9113037 / HR B 744851 / AG Stuttgart >-------------------------------------------------------- >Geschäftsführer: Tobias Frech >mobile: +49-(0)172-7112352 / email: [email protected] >
