Re: Graceful recovery from LeaseExpired when usign JCR-OAK and Spring based configuration.
Hi Jesse, Yes, there's OAK-3250. Besides that there is OAK-3373 (Observers not supporting a restart) and in the discussion therein a few more details as to what prevents a successful restart of Oak without a VM restart (there might be other scenarios I'm not aware of too). Your particular application might work fine with the approach you describe, based on how you use Oak (to be thoroughly verified). But generally speaking Oak itself doesn't support a restart 'on the fly' atm. Cheers, Stefan On 29.01.19 19:22, Jesse . wrote: Stefan, Thank your for this information. Is there an issue I can vote on or watch to see if this functionality does get implemented in the future? In the meantime, I have resorted to converting my Repository bean into a proxy object where I can then update the actual underlying Repository and DocumentNodeStore when failure occurs. I am using the LeaseFailureHandler to call DocumentNodeStore.dispose() on the old DocumentNodeStore object, and then creating new DocumentNodeStore and Repository objects without restarting the application or VM. Would you know if this approach violates any best practices? Thanks, Jesse On Tue, Jan 29, 2019 at 4:36 AM Stefan Egli wrote: Hi, Indeed the current behaviour is as you describe: a lease failure will require a restart of oak (and thus of the VM it runs in). This is due to a combination of facts: (a) when an instance fails to update the lease other instances consider it crashed and will do a recovery on behalf of it, so the instance must not come back after a lease failure and (b) the current implementation doesn't support reconnecting to the document backend (say with a different ID) without a restart (there have been plans to support this, but they weren't followed up yet). Cheers, Stefan On 28.01.19 21:02, Jesse . wrote: Hello, I currently am using JCR-OAK 1.8.11 running with Spring 4.2 and Tomcat 7. I am using Spring based configuration where the DocumentNodeStore (Mongo) and Repository are both Spring Beans. This has worked well with the exception of an how the application reacts when a the connection to the Mongo Database is lost. When the connection to mongo is lost, eventually we get a Lease Expiration notice that says "This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore". When the connection can be restored to mongo, there is no graceful recovery and reconnect. My application can no longer perform operations and must be restarted to regain functionality. Is there a suggested way to handle this scenario? I looked at the failure handler functionality, but I can't just recreate a new DocumentNodeStore and Repository because they are Spring Beans. Thanks Example Error: [ERROR] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update thread (1)/[]] [ClusterNodeInfo:leaseExpired(1163)] - This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore. [WARN ] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update thread (1)/[]] [DocumentNodeStore:runBackgroundUpdateOperations(2021)] - Background update operation failed (will be retried with next run): org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore. org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This oak instance failed to update the lease in time and can therefore no longer access this DocumentNodeStore. at org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.leaseExpired(ClusterNodeInfo.java:1165) ~[oak-store-document-1.8.11.jar:1.8.11] at org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.performLeaseCheck(ClusterNodeInfo.java:665) ~[oak-store-document-1.8.11.jar:1.8.11] at org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.performLeaseCheck(LeaseCheckDocumentStoreWrapper.java:58) ~[oak-store-document-1.8.11.jar:1.8.11] at org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.find(LeaseCheckDocumentStoreWrapper.java:64) ~[oak-store-document-1.8.11.jar:1.8.11] at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.cleanCollisions(DocumentNodeStore.java:2291) ~[oak-store-document-1.8.11.jar:1.8.11] at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:2033) ~[oak-store-document-1.8.11.jar:1.8.11] at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:2016) [oak-store-document-1.8.11.jar:1.8.11] at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundUpdateOperation.execute(DocumentNodeStore.java:3095) [oak-store-document-1.8.11.jar:1.8.11] at
Re: Graceful recovery from LeaseExpired when usign JCR-OAK and Spring based configuration.
Stefan, Thank your for this information. Is there an issue I can vote on or watch to see if this functionality does get implemented in the future? In the meantime, I have resorted to converting my Repository bean into a proxy object where I can then update the actual underlying Repository and DocumentNodeStore when failure occurs. I am using the LeaseFailureHandler to call DocumentNodeStore.dispose() on the old DocumentNodeStore object, and then creating new DocumentNodeStore and Repository objects without restarting the application or VM. Would you know if this approach violates any best practices? Thanks, Jesse On Tue, Jan 29, 2019 at 4:36 AM Stefan Egli wrote: > Hi, > > Indeed the current behaviour is as you describe: a lease failure will > require a restart of oak (and thus of the VM it runs in). This is due to > a combination of facts: (a) when an instance fails to update the lease > other instances consider it crashed and will do a recovery on behalf of > it, so the instance must not come back after a lease failure and (b) the > current implementation doesn't support reconnecting to the document > backend (say with a different ID) without a restart (there have been > plans to support this, but they weren't followed up yet). > > Cheers, > Stefan > > On 28.01.19 21:02, Jesse . wrote: > > Hello, > > > > I currently am using JCR-OAK 1.8.11 running with Spring 4.2 and Tomcat > 7. I > > am using Spring based configuration where the DocumentNodeStore (Mongo) > and > > Repository are both Spring Beans. This has worked well with the exception > > of an how the application reacts when a the connection to the Mongo > > Database is lost. When the connection to mongo is lost, eventually we > get > > a Lease Expiration notice that says "This oak instance failed to update > the > > lease in time and can therefore no longer access this DocumentNodeStore". > > > > When the connection can be restored to mongo, there is no graceful > recovery > > and reconnect. My application can no longer perform operations and must > be > > restarted to regain functionality. > > > > Is there a suggested way to handle this scenario? I looked at the failure > > handler functionality, but I can't just recreate a new DocumentNodeStore > > and Repository because they are Spring Beans. > > > > Thanks > > > > Example Error: > > > > [ERROR] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update > > thread (1)/[]] > > > [ClusterNodeInfo:leaseExpired(1163)] - This oak instance failed to update > > the lease in time and can therefore no longer access this > DocumentNodeStore. > > [WARN ] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update > > thread (1)/[]] > > > > [DocumentNodeStore:runBackgroundUpdateOperations(2021)] - Background > update > > operation failed (will be retried with next run): > > org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This > oak > > instance failed to update the lease in time and can therefore no longer > > access this DocumentNodeStore. > > org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This > oak > > instance failed to update the lease in time and can therefore no longer > > access this DocumentNodeStore. > > at > > > org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.leaseExpired(ClusterNodeInfo.java:1165) > > ~[oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.performLeaseCheck(ClusterNodeInfo.java:665) > > ~[oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.performLeaseCheck(LeaseCheckDocumentStoreWrapper.java:58) > > ~[oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.find(LeaseCheckDocumentStoreWrapper.java:64) > > ~[oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.cleanCollisions(DocumentNodeStore.java:2291) > > ~[oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:2033) > > ~[oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:2016) > > [oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundUpdateOperation.execute(DocumentNodeStore.java:3095) > > [oak-store-document-1.8.11.jar:1.8.11] > > at > > > org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:3064) > > [oak-store-document-1.8.11.jar:1.8.11] > > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] > > > > > > Example code: > > > > @Configuration > > public class
Re: Graceful recovery from LeaseExpired when usign JCR-OAK and Spring based configuration.
Hi,
Indeed the current behaviour is as you describe: a lease failure will
require a restart of oak (and thus of the VM it runs in). This is due to
a combination of facts: (a) when an instance fails to update the lease
other instances consider it crashed and will do a recovery on behalf of
it, so the instance must not come back after a lease failure and (b) the
current implementation doesn't support reconnecting to the document
backend (say with a different ID) without a restart (there have been
plans to support this, but they weren't followed up yet).
Cheers,
Stefan
On 28.01.19 21:02, Jesse . wrote:
Hello,
I currently am using JCR-OAK 1.8.11 running with Spring 4.2 and Tomcat 7. I
am using Spring based configuration where the DocumentNodeStore (Mongo) and
Repository are both Spring Beans. This has worked well with the exception
of an how the application reacts when a the connection to the Mongo
Database is lost. When the connection to mongo is lost, eventually we get
a Lease Expiration notice that says "This oak instance failed to update the
lease in time and can therefore no longer access this DocumentNodeStore".
When the connection can be restored to mongo, there is no graceful recovery
and reconnect. My application can no longer perform operations and must be
restarted to regain functionality.
Is there a suggested way to handle this scenario? I looked at the failure
handler functionality, but I can't just recreate a new DocumentNodeStore
and Repository because they are Spring Beans.
Thanks
Example Error:
[ERROR] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update
thread (1)/[]]
[ClusterNodeInfo:leaseExpired(1163)] - This oak instance failed to update
the lease in time and can therefore no longer access this DocumentNodeStore.
[WARN ] 28 Jan 2019 13:55:40,183 [DocumentNodeStore background update
thread (1)/[]]
[DocumentNodeStore:runBackgroundUpdateOperations(2021)] - Background update
operation failed (will be retried with next run):
org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This oak
instance failed to update the lease in time and can therefore no longer
access this DocumentNodeStore.
org.apache.jackrabbit.oak.plugins.document.DocumentStoreException: This oak
instance failed to update the lease in time and can therefore no longer
access this DocumentNodeStore.
at
org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.leaseExpired(ClusterNodeInfo.java:1165)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfo.performLeaseCheck(ClusterNodeInfo.java:665)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.performLeaseCheck(LeaseCheckDocumentStoreWrapper.java:58)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.find(LeaseCheckDocumentStoreWrapper.java:64)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.cleanCollisions(DocumentNodeStore.java:2291)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.internalRunBackgroundUpdateOperations(DocumentNodeStore.java:2033)
~[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.runBackgroundUpdateOperations(DocumentNodeStore.java:2016)
[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundUpdateOperation.execute(DocumentNodeStore.java:3095)
[oak-store-document-1.8.11.jar:1.8.11]
at
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:3064)
[oak-store-document-1.8.11.jar:1.8.11]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Example code:
@Configuration
public class DocumentNodeStoreConfig {
@Value("${mongodb.uri}")
private String mongodbURI;
@Value("${mongodb.port}")
private int mongodbPort;
@Value("${mongodb.database}")
private String mongodbDatabase;
@Value("${mongodb.username}")
private String mongodbUsername;
@Value("${mongodb.password}")
private String mongodbPassword;
@Bean(destroyMethod = "dispose")
public DocumentNodeStore documentNodeStore() {
DocumentNodeStore store = null;
try {
store =
MongoDocumentNodeStoreBuilder.newMongoDocumentNodeStoreBuilder()
.setMongoDB("mongodb://" + mongodbUsername + ":" +
mongodbPassword + "@" + mongodbURI + ":" + mongodbPort + "/" +
mongodbDatabase, mongodbDatabase, 0).build();
} catch (UnknownHostException e) {
e.printStackTrace();
logger.error("UnknownHostException: " + e);
throw new RuntimeException(e);
}
return store;
}
@Bean
