José Andrés Cordero Benítez created OAK-10758:
-------------------------------------------------
Summary: Avoid or mitigate Cursor Timeout in Mongo when running
detailedGC
Key: OAK-10758
URL: https://issues.apache.org/jira/browse/OAK-10758
Project: Jackrabbit Oak
Issue Type: Improvement
Components: documentmk, oak-run
Reporter: José Andrés Cordero Benítez
Assignee: José Andrés Cordero Benítez
When running oak-run revisions detailedGC command, if the cleanup is too big
the processing time can exceed the Mongo cursor timeout, which defaults to 30
minutes:
{code:java}
java.io.IOException: com.mongodb.MongoCursorNotFoundException: Query failed
with error code -5 and error message 'Cursor 1007212049743349339 not found on
server mongohost' on server mongohost:27017
at
org.apache.jackrabbit.oak.run.RevisionsCommand.collectOnce(RevisionsCommand.java:427)
at
org.apache.jackrabbit.oak.run.RevisionsCommand.collect(RevisionsCommand.java:377)
at
org.apache.jackrabbit.oak.run.RevisionsCommand.execute(RevisionsCommand.java:245)
at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
Caused by: com.mongodb.MongoCursorNotFoundException: Query failed with error
code -5 and error message 'Cursor 1007212049743349339 not found on server
mongohost:27017' on server mongohost:27017
at
com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:27)
at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:268)
at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:139)
at
com.mongodb.client.internal.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:54)
at
org.apache.jackrabbit.guava.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
at
org.apache.jackrabbit.guava.common.collect.Iterators$5.computeNext(Iterators.java:670)
at
org.apache.jackrabbit.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
at
org.apache.jackrabbit.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
at
org.apache.jackrabbit.guava.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1395)
at
org.apache.jackrabbit.oak.plugins.document.SplitDocumentCleanUp.disconnect(SplitDocumentCleanUp.java:63)
at
org.apache.jackrabbit.oak.plugins.document.VersionGCSupport.deleteSplitDocuments(VersionGCSupport.java:146)
at
org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.collectSplitDocuments(VersionGarbageCollector.java:842)
at
org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.gc(VersionGarbageCollector.java:686)
at
org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.run(VersionGarbageCollector.java:653)
at
org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector.gc(VersionGarbageCollector.java:221)
at
org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:401)
at
org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:398)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
15:45:08.184 [DocumentNodeStore cluster update thread (0)] WARN
o.a.j.o.p.document.DocumentNodeStore - Background operation failed:
java.lang.reflect.UndeclaredThrowableException
java.lang.reflect.UndeclaredThrowableException: null
at com.sun.proxy.$Proxy0.query(Unknown Source)
at
org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfoDocument.all(ClusterNodeInfoDocument.java:154)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.updateClusterState(DocumentNodeStore.java:2480)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundClusterUpdate.execute(DocumentNodeStore.java:3931)
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:3734)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.reflect.InvocationTargetException: null
at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.jackrabbit.oak.plugins.document.util.ReadOnlyDocumentStoreWrapperFactory$1.invoke(ReadOnlyDocumentStoreWrapperFactory.java:41)
... 6 common frames omitted
Caused by: java.lang.IllegalStateException: The pool is closed
at
com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:137)
at
com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:123)
at
com.mongodb.internal.session.ServerSessionPool.get(ServerSessionPool.java:80)
at
com.mongodb.internal.session.BaseClientSessionImpl.<init>(BaseClientSessionImpl.java:44)
at
com.mongodb.client.internal.ClientSessionImpl.<init>(ClientSessionImpl.java:56)
at
com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:120)
at com.mongodb.Mongo.createClientSession(Mongo.java:837)
at com.mongodb.MongoClient.startSession(MongoClient.java:582)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoSessionFactory.createClientSession(MongoSessionFactory.java:55)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDBConnection.createClientSession(MongoDBConnection.java:115)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createClientSession(MongoDocumentStore.java:2405)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.execute(MongoDocumentStore.java:2389)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryInternal(MongoDocumentStore.java:846)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryWithRetry(MongoDocumentStore.java:774)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:751)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:739)
at
org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:728)
... 10 common frames omitted {code}
Check if this needs mitigation or could happen as part of the internal
repository detailedGC execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
