Hi,
Indeed the current behaviour is as you describe: a lease failure will
require a restart of oak (and thus of the VM it runs in). This is due to
a combination of facts: (a) when an instance fails to update the lease
other instances consider it crashed and will do a recovery on behalf of
it, so the instance must not come back after a lease failure and (b) the
current implementation doesn't support reconnecting to the document
backend (say with a different ID) without a restart (there have been
plans to support this, but they weren't followed up yet).
Cheers,
Stefan
On 28.01.19 21:02, Jesse . wrote:
Hello,
I currently am using JCR-OAK 1.8.11 running with Spring 4.2 and Tomcat
7. I
am using Spring based configuration where the DocumentNodeStore (Mongo)
and
Repository are both Spring Beans. This has worked well with the exception
of an how the application reacts when a the connection to the Mongo
Database is lost. When the connection to mongo is lost, eventually we
get
a Lease Expiration notice that says "This oak instance failed to update
the
lease in time and can therefore no longer access this DocumentNodeStore".
When the connection can be restored to mongo, there is no graceful
recovery
and reconnect. My application can no longer perform operations and must
be
restarted to regain functionality.
Is there a suggested way to handle this scenario? I looked at the failure
handler functionality, but I can't just recreate a new DocumentNodeStore
and Repository because they are Spring Beans.
Thanks
Example Error:
------------
[ERROR] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update
thread (1)/[]]
<org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo>
[ClusterNodeInfo:leaseExpired(1163)] - This oak instance failed to update
the lease in time and can therefore no longer access this
DocumentNodeStore.
[WARN ] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update
thread (1)/[]]
<org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore>
[DocumentNodeStore:runBackgroundUpdateOperations(2021)] - Background
update
operation failed (will be retried with next run):
org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This
oak
instance failed to update the lease in time and can therefore no longer
access this DocumentNodeStore.
org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This
oak
instance failed to update the lease in time and can therefore no longer
access this DocumentNodeStore.
at
org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.leaseExpired(ClusterNodeInfo.java:1165)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.performLeaseCheck(ClusterNodeInfo.java:665)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.performLeaseCheck(LeaseCheckDocumentStoreWrapper.java:58)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.find(LeaseCheckDocumentStoreWrapper.java:64)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.cleanCollisions(DocumentNodeStore.java:2291)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:2033)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:2016)
[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundUpdateOperation.execute(DocumentNodeStore.java:3095)
[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:3064)
[oak-store-document-1.8.11.jar:1.8.11]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Example code:
------------------------
@Configuration
public class DocumentNodeStoreConfig {
@Value("${mongodb.uri}")
private String mongodbURI;
@Value("${mongodb.port}")
private int mongodbPort;
@Value("${mongodb.database}")
private String mongodbDatabase;
@Value("${mongodb.username}")
private String mongodbUsername;
@Value("${mongodb.password}")
private String mongodbPassword;
@Bean(destroyMethod = "dispose")
public DocumentNodeStore documentNodeStore() {
DocumentNodeStore store = null;
try {
store =
MongoDocumentNodeStoreBuilder.newMongoDocumentNodeStoreBuilder()
.setMongoDB("mongodb://" + mongodbUsername + ":" +
mongodbPassword + "@" + mongodbURI + ":" + mongodbPort + "/" +
mongodbDatabase, mongodbDatabase, 0).build();
} catch (UnknownHostException e) {
e.printStackTrace();
logger.error("UnknownHostException: " + e);
throw new RuntimeException(e);
}
return store;
}
@Bean
public Repository jcrRepository() {
try {
return new Jcr(new
Oak(documentNodeStore())).createRepository();
} catch (Exception e) {
logger.error(e.getMessage());
throw new RuntimeException(e);
}
}
}
