Hi there! We are using Adobe Experience Manager 6.0 which includes Apache OAK 1.0.x and are looking into ways to shrink the size of our repository. Since offline compaction comes with additonal ops burden online compaction seems to be a better solution for us. I enabled online compaction on a test system and triggered a compaction run. After a while the JVM (-Xmx6g) ran into an OutOfMemory-Error (Heap). I tried to shutdown AEM cleanly but had to kill the process after waiting for an hour or so. Now after trying to start AEM again it does not fully start, some service seems not to be started anymore. But no error message is logged by AEM.
Trying to check the repository I ran the following command and got the
exception below:
[cq@xxxxxxx crx-quickstart]$ java -jar oak-run-1.0.19.jar backup
./repository/segmentstore /opt/cq/backup/
Apache Jackrabbit Oak 1.0.19
05:54:27.495 [main] WARN o.a.j.o.p.segment.file.TarReader - Unexpected
size 1024 of tar file data01186a.tar
05:54:27.510 [main] WARN o.a.j.o.p.segment.file.TarReader - Could not
find a valid tar index in [./repository/segmentstore/data01186a.tar],
recovering...
06:09:05.078 [main] ERROR o.a.j.o.p.segment.SegmentTracker - Segment not
found: 11f36041-cece-48d6-b95c-029be7c7a91c. Creation date delta is 23 ms.
org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException:
Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:855)
~[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:134)
~[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:52)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:168)
[oak-run-1.0.19.jar:1.0.19]
at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
[oak-run-1.0.19.jar:1.0.19]
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
[oak-run-1.0.19.jar:1.0.19]
at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:214)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:263)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:219)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.java:50)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Compactor.java:140)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:155)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:434)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:93)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:98)
[oak-run-1.0.19.jar:1.0.19]
at
org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreBackup.java:78)
[oak-run-1.0.19.jar:1.0.19]
at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193)
[oak-run-1.0.19.jar:1.0.19]
at org.apache.jackrabbit.oak.run.Main.main(Main.java:119)
[oak-run-1.0.19.jar:1.0.19]
Exception in thread "main"
org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException:
Segment 11f36041-cece-48d6-b95c-029be7c7a91c not found
at
org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:855)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:134)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
at
org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
at
org.apache.jackrabbit.oak.plugins.segment.BlockRecord.read(BlockRecord.java:52)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentStream.read(SegmentStream.java:168)
at com.google.common.io.ByteStreams.read(ByteStreams.java:828)
at com.google.common.io.ByteSource.contentEquals(ByteSource.java:303)
at com.google.common.io.ByteStreams.equal(ByteStreams.java:661)
at
org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:214)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:263)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:219)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.access$100(Compactor.java:50)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.propertyAdded(Compactor.java:140)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:155)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor$CompactDiff.childNodeAdded(Compactor.java:167)
at
org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
at
org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:434)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.process(Compactor.java:93)
at
org.apache.jackrabbit.oak.plugins.segment.Compactor.compact(Compactor.java:98)
at
org.apache.jackrabbit.oak.plugins.backup.FileStoreBackup.backup(FileStoreBackup.java:78)
at org.apache.jackrabbit.oak.run.Main.backup(Main.java:193)
at org.apache.jackrabbit.oak.run.Main.main(Main.java:119)
What can we do now to get the repository back into a clean state?
Thanks in advance,
Tobias Frech
--
Frech IT GmbH / Am Brünnele 7 / 71642 Ludwigsburg
phone : +49-(0)7141-9113037 / HR B 744851 / AG Stuttgart
--------------------------------------------------------
Geschäftsführer: Tobias Frech
mobile: +49-(0)172-7112352 / email: [email protected]
smime.p7s
Description: S/MIME Cryptographic Signature
