Hi there Our setup: We have a multi-node clustered Java application running in an ESXi environment. Each cluster node has Geode embedded via. Spring Data for Apache Geode and has its own locator. Multiple replicated regions are shared among the nodes where each node has its own disk store. * Java runtime version: 1.8.0_151 * Geode version: 1.0.0-incubating * Spring Data Geode version: 1.0.0.INCUBATING-RELEASE * Spring Data version: 1.12.1.RELEASE
Our problem: We had situation that caused our geode processes to quit abruptly e.g. * VM being abruptly powered off (no guest shutdown) or... * ...CPU-freezes caused by IO degradation. After restarting the cluster nodes (one after another or all at once), geode logs on all nodes show the following: Region /XXX has potentially stale data. It is waiting for another member to recover the latest data. My persistent id: DiskStore ID: XXX Name: Location: /XXX Members with potentially new data: [ DiskStore ID: XXX Name: XXX Location: /XXX ] The problem however is that each node is waiting for the other nodes to join although they are already started. Any combination of starting/stopping the nodes that are shown as "missing" doesn't seem to do anything. Our temporary solution: We managed to "recover" from such a deadlock using gfsh: * Revoke all missing disk stores except for one "chosen" (preferably the last running) node. * Delete those disk stores. * Restart the nodes. As of today we're not able to add the "Spring Shell" dependency to our application easily which is why we have to run gfsh with its own locator. This requires us to define such a "gfsh locator" in all of our cluster nodes. What we're looking for: Our temporary solution comes with some flaws: we're dependent of the gfsh tooling with its own locator and manual intervention is required. Getting the cluster up-and-running again is complicated from an admin perspective. Is there any way detect/handle such a deadlock situation from within the application? Are there any best practices that you could recommend? Thanks in advance for your help! Best Daniel persistent security in a changing world.