Hi there

Our setup:
We have a multi-node clustered Java application running in an ESXi environment. 
Each cluster node has Geode embedded via. Spring Data for Apache Geode and has 
its own locator. Multiple replicated regions are shared among the nodes where 
each node has its own disk store.
* Java runtime version: 1.8.0_151
* Geode version: 1.0.0-incubating
* Spring Data Geode version: 1.0.0.INCUBATING-RELEASE
* Spring Data version: 1.12.1.RELEASE

Our problem:
We had situation that caused our geode processes to quit abruptly e.g.
* VM being abruptly powered off (no guest shutdown) or...
* ...CPU-freezes caused by IO degradation.
After restarting the cluster nodes (one after another or all at once), geode 
logs on all nodes show the following:
        Region /XXX has potentially stale data. It is waiting for another 
member to recover the latest data.
        My persistent id:
          DiskStore ID: XXX
          Name:
          Location: /XXX
        Members with potentially new data:
        [
          DiskStore ID: XXX
          Name: XXX
          Location: /XXX
        ]
The problem however is that each node is waiting for the other nodes to join 
although they are already started. Any combination of starting/stopping the 
nodes that are shown as "missing" doesn't seem to do anything.

Our temporary solution:
We managed to "recover" from such a deadlock using gfsh:
* Revoke all missing disk stores except for one "chosen" (preferably the last 
running) node.
* Delete those disk stores.
* Restart the nodes.
As of today we're not able to add the "Spring Shell" dependency to our 
application easily which is why we have to run gfsh with its own locator. This 
requires us to define such a "gfsh locator" in all of our cluster nodes.

What we're looking for:
Our temporary solution comes with some flaws: we're dependent of the gfsh 
tooling with its own locator and manual intervention is required. Getting the 
cluster up-and-running again is complicated from an admin perspective. Is there 
any way detect/handle such a deadlock situation from within the application? 
Are there any best practices that you could recommend?

Thanks in advance for your help!

Best
Daniel
persistent security in a changing world.

Reply via email to