Hi everyone,
We are currently performing a repository migration from Segment TAR to
MongoDB (DocumentNodeStore) using the oak-upgrade utility.
We are using this command:
java -jar oak-upgrade-1.74.0.jar \
--copy-binaries \
--src-datastore=/home/app-files/drive/segmentstore/repository/datastore
--datastore=/home/app-files/drive/segmentstore/repository/datastore_new
/home/app-files/drive
"mongodb://user:password@localhost:27017/drive?authSource=admin"
The migration process runs for approximately 15 minutes, but then it
fails with an exception (stacktrace included at the end of this
email).
Do you have any insights into the root cause of this issue? We are
wondering if there are specific configurations or flags to mitigate
this.
We would appreciate any guidance on further troubleshooting.
Best regards,
Marco Matessi
//Error Stacktrace:
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.MigrationOptions -
copyVersions parameter set to 1970-01-01
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.MigrationOptions -
copyOrphanedVersions parameter set to 1970-01-01
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.MigrationOptions - Cache
size: 256 MB
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.StoreArguments - Source:
SEGMENT_TAR[/home/app-files/drive]
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.StoreArguments -
Destination:
MONGO[mongodb://user:password@localhost:27017/drive?authSource=admin]
[INFO ] 21:30:07 org.apache.jackrabbit.oak.segment.file.FileStore -
Creating file store FileStoreBuilder{version=1.74.0,
directory=/home/app-files/drive/segmentstore, blobStore=null,
binariesInlineThreshold=16512, maxFileSize=256, segmentCacheSize=256,
stringCacheSize=256, templateCacheSize=64,
stringDeduplicationCacheSize=15000,
templateDeduplicationCacheSize=3000,
nodeDeduplicationCacheSize=1048576, memoryMapping=false,
offHeapAccess=false, gcOptions=SegmentGCOptions{paused=false,
estimationDisabled=false, gcSizeDeltaEstimation=1073741824,
retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL,
compactorType=PARALLEL_COMPACTOR}}
[INFO ] 21:30:07
org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore - TarMK
ReadOnly opened: /home/app-files/drive/segmentstore (mmap=false)
[INFO ] 21:30:07
org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore - TarMK
closed: /home/app-files/drive/segmentstore
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.DatastoreArguments -
Blobs stored in
FileDataStore[/home/app-files/drive/segmentstore/repository/datastore]
will be copied to
FileDataStore[/home/app-files/drive/segmentstore/repository/datastore_new]
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.DatastoreArguments -
Source blob store:
FileDataStore[/home/app-files/drive/segmentstore/repository/datastore]
[INFO ] 21:30:07 org.apache.jackrabbit.oak.segment.file.FileStore -
Creating file store FileStoreBuilder{version=1.74.0,
directory=/home/app-files/drive/segmentstore, blobStore=DataStore
backed BlobStore
[org.apache.jackrabbit.oak.plugins.blob.datastore.OakFileDataStore],
binariesInlineThreshold=16512, maxFileSize=256, segmentCacheSize=256,
stringCacheSize=256, templateCacheSize=64,
stringDeduplicationCacheSize=15000,
templateDeduplicationCacheSize=3000,
nodeDeduplicationCacheSize=1048576, memoryMapping=true,
offHeapAccess=false, gcOptions=SegmentGCOptions{paused=false,
estimationDisabled=false, gcSizeDeltaEstimation=1073741824,
retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL,
compactorType=PARALLEL_COMPACTOR}}
[INFO ] 21:30:07
org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore - TarMK
ReadOnly opened: /home/app-files/drive/segmentstore (mmap=true)
[INFO ] 21:30:07
org.apache.jackrabbit.oak.segment.SegmentNodeStore$SegmentNodeStoreBuilder
- Creating segment node store
SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore
[org.apache.jackrabbit.oak.plugins.blob.datastore.OakFileDataStore]}
[INFO ] 21:30:07
org.apache.jackrabbit.oak.segment.scheduler.LockBasedScheduler -
Initializing SegmentNodeStore with the commitFairLock option enabled.
[INFO ] 21:30:07
org.apache.jackrabbit.oak.upgrade.cli.parser.DatastoreArguments -
Destination blob store:
FileDataStore[/home/app-files/drive/segmentstore/repository/datastore_new]
[INFO ] 21:30:08 org.mongodb.driver.cluster - Cluster created with
settings {hosts=[app-drive-storage:27017], mode=SINGLE,
requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms',
maxWaitQueueSize=500}
[INFO ] 21:30:08 org.mongodb.driver.cluster - Cluster description not
yet available. Waiting for 30000 ms before timing out
[INFO ] 21:30:08 org.mongodb.driver.connection - Opened connection
[connectionId{localValue:1, serverValue:140}] to
app-drive-storage:27017
[INFO ] 21:30:08 org.mongodb.driver.cluster - Monitor thread
successfully connected to server with description
ServerDescription{address=app-drive-storage:27017, type=STANDALONE,
state=CONNECTED, ok=true, version=ServerVersion{versionList=[5, 0,
31]}, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216,
logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2768922}
[INFO ] 21:30:08 org.mongodb.driver.connection - Opened connection
[connectionId{localValue:2, serverValue:141}] to
app-drive-storage:27017
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore - System
property oak.disableJournalDiff found to be 'true'
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore -
Creating Collection nodes, with collection storage options
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore -
Connected to MongoDB 5.0.31 with maxReplicationLagMillis 21600000,
maxDeltaForModTimeIdxSecs 60, disableIndexHint false,
leaseSocketTimeout 0, clientSessionSupported true, clientSessionInUse
true, WriteConcern{w=null, wTimeout=null ms, fsync=null, journal=null,
serverStatus {process=mongod, mem={"bits": 64, "resident": 120,
"virtual": 1569, "supported": true},
host=app-drive-storage-6df95bb94c-xsgtz, connections={"current": 2,
"available": 52426, "totalCreated": 141, "active": 1, "threaded": 2,
"exhaustIsMaster": 0, "exhaustHello": 0, "awaitingTopologyChanges":
0}, storageEngine={"name": "wiredTiger", "supportsCommittedReads":
true, "oldestRequiredTimestampForCrashRecovery": {"$timestamp": {"t":
0, "i": 0}}, "supportsPendingDrops": true, "dropPendingIdents":
{"$numberLong": "0"}, "supportsSnapshotReadConcern": true, "readOnly":
false, "persistent": true, "backupCursorOpen": false,
"supportsResumableIndexBuilds": true}}, throttlingSupported false
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore - Server
time difference: 0ms (max allowed: 2000ms)
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore -
FormatVersion set to 1.8.0
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo - Acquired
(new) clusterId 1. MachineId mac:6a861aba1ebd, InstanceId
/home/upgrade-executor, RuntimeId 15b9210e-63fb-4a84-b333-45ba2c966526
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector.auditFGC
- <init> VersionGarbageCollector created with fullGcMode: NONE,
batchSize: 1000, progressSize: 10000, delayFactor: 2.0
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector.auditFGC
- Full GC paths set to include: [] and exclude: [] in mode NONE
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.Sweep2StatusDocument -
acquireOrUpdateSweep2Lock: sweep2 status set to checking, locked for
clusterId=1
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore -
ChangeSetBuilder enabled and size set to maxItems: 50, maxDepth: 9
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore -
Initialized DocumentNodeStore with clusterNodeId: 1, updateLimit:
100000 (id: 1, startTime: 1763501408536, machineId: mac:6a861aba1ebd,
instanceId: /home/upgrade-executor, runtimeId:
15b9210e-63fb-4a84-b333-45ba2c966526, pid: 7, uuid:
922726b1-398f-421d-82ff-e2d631474eed, readWriteMode: null,
leaseCheckMode: STRICT, state: ACTIVE, oakVersion: 1.74.0,
formatVersion: 1.8.0, invisible: false)
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore -
prevNoProp cache is disabled
[INFO ] 21:30:08
org.apache.jackrabbit.oak.plugins.document.bundlor.BundlingConfigHandler
- Bundling of nodes enabled
[INFO ] 21:30:08 org.apache.jackrabbit.oak.plugins.version.Utils -
Repository does not define nt:frozenNode. Assuming frozen nodes are
not referenceable.
[INFO ] 21:30:08 org.apache.jackrabbit.oak.upgrade.RepositorySidegrade
- Migrating first checkpoint: 17af4481-9e6c-4ce3-b0f0-22afca6efeb0
[INFO ] 21:30:08 org.apache.jackrabbit.oak.upgrade.RepositorySidegrade
- Checkpoint expiry time: 1847955188932, metadata:
{creator=AsyncIndexUpdate, created=2025-10-27T08:53:08.932Z,
name=async, thread=oak-scheduled-executor-962956}
[INFO ] 21:30:09 org.mongodb.driver.connection - Opened connection
[connectionId{localValue:3, serverValue:147}] to
app-drive-storage:27017
[INFO ] 21:30:16 org.apache.jackrabbit.oak.upgrade.RepositorySidegrade
- Copying node #10000:
/oak:index/sessionIdIndex/:index/2829/Users/user-4/My Data
..................
<other copying node logs>
..................
INFO ] 21:42:28 org.apache.jackrabbit.oak.upgrade.RepositorySidegrade
- Copying node #1110000:
/oak:index/idIndex/:index/5fe10d59-8058-470d-b0c0-0d7b245f0e2a/Users/user-2/My
Data
[INFO ] 21:42:28 org.apache.jackrabbit.oak.upgrade.RepositorySidegrade
- Copying node #1120000:
/oak:index/idIndex/:index/0ee891c6-2bb6-4f93-90dc-bc208e891c9b/Users/user-2/My
Data/2a556ba2-8c38-4417-930c-d220d911ed43/510b00a1-a8dd-4d32-a340-e506f73d8e74/893b0108-94c7-40da-9ef6-dfe818cf3e02
[INFO ] 21:45:49
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore -
invalidatePathsOnCancel : cancelInvalidationEnabled = false
[INFO ] 21:45:49
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore -
Starting disposal of DocumentNodeStore with clusterNodeId: 1 (id: 1,
startTime: 1763501408536, machineId: mac:6a861aba1ebd, instanceId:
/home/upgrade-executor, runtimeId:
15b9210e-63fb-4a84-b333-45ba2c966526, pid: 7, uuid:
922726b1-398f-421d-82ff-e2d631474eed, readWriteMode: null,
leaseCheckMode: STRICT, state: ACTIVE, oakVersion: 1.74.0,
formatVersion: 1.8.0, invisible: false)
[INFO ] 21:45:50
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore -
Disposed DocumentNodeStore with clusterNodeId: 1, (successful)
[INFO ] 21:45:50
org.apache.jackrabbit.oak.segment.file.ReadOnlyFileStore - TarMK
closed: /home/app-files/drive/segmentstore
Exception in thread "main" java.lang.RuntimeException:
javax.jcr.RepositoryException: Failed to copy content
at org.apache.jackrabbit.guava.common.io.Closer.rethrow(Closer.java:137)
at
org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:82)
at
org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:68)
at
com.company.upgrade.segmentstoremigration.OakUpgradeWrapper.migrate(OakUpgradeWrapper.java:22)
at
com.company.upgrade.segmentstoremigration.DriveMongoMigrator.migrate(DriveMongoMigrator.java:54)
at
com.company.upgrade.executor.MongoMigrationExecutor.migrateDrive(MongoMigrationExecutor.java:63)
at
com.company.upgrade.executor.MongoMigrationExecutor.run(MongoMigrationExecutor.java:49)
at
com.company.upgrade.executor.UpgradeExecutorApp.main(UpgradeExecutorApp.java:54)
Caused by: javax.jcr.RepositoryException: Failed to copy content
at
org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:293)
at
org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:93)
at
org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:79)
... 6 more
Caused by: java.lang.IllegalMonitorStateException: attempt to unlock
read lock, not locked by current thread
at
java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:448)
at
java.base/java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:432)
at
java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1146)
at
java.base/java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:897)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.canceled(DocumentNodeStore.java:1188)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.persist(DocumentNodeStoreBranch.java:324)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.persist(DocumentNodeStoreBranch.java:281)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch$Persisted.persistTransientHead(DocumentNodeStoreBranch.java:717)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch$Persisted.setRoot(DocumentNodeStoreBranch.java:663)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.setRoot(DocumentNodeStoreBranch.java:111)
at
org.apache.jackrabbit.oak.plugins.document.DocumentRootBuilder.purge(DocumentRootBuilder.java:185)
at
org.apache.jackrabbit.oak.plugins.document.DocumentRootBuilder.updated(DocumentRootBuilder.java:105)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:212)
at
org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:344)
at
org.apache.jackrabbit.oak.plugins.document.AbstractDocumentNodeBuilder.setChildNode(AbstractDocumentNodeBuilder.java:56)
at
org.apache.jackrabbit.oak.plugins.document.AbstractDocumentNodeBuilder.setChildNode(AbstractDocumentNodeBuilder.java:61)
at
org.apache.jackrabbit.oak.plugins.document.AbstractDocumentNodeBuilder.setChildNode(AbstractDocumentNodeBuilder.java:61)
at
org.apache.jackrabbit.oak.plugins.document.AbstractDocumentNodeBuilder.setChildNode(AbstractDocumentNodeBuilder.java:61)
at
org.apache.jackrabbit.oak.plugins.document.AbstractDocumentNodeBuilder.setChildNode(AbstractDocumentNodeBuilder.java:61)
at
org.apache.jackrabbit.oak.plugins.document.AbstractDocumentNodeBuilder.setChildNode(AbstractDocumentNodeBuilder.java:61)
at
org.apache.jackrabbit.oak.spi.state.ApplyDiff.childNodeAdded(ApplyDiff.java:80)
at
org.apache.jackrabbit.oak.plugins.migration.AbstractDecoratedNodeState$DecoratingDiff.childNodeAdded(AbstractDecoratedNodeState.java:254)
at
org.apache.jackrabbit.oak.plugins.migration.AbstractDecoratedNodeState.compareAgainstBaseState(AbstractDecoratedNodeState.java:232)
at
org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copyDiffToTarget(RepositorySidegrade.java:403)
at
org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.migrateWithCheckpoints(RepositorySidegrade.java:351)
at
org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copyState(RepositorySidegrade.java:316)
at
org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:286)
... 8 more
