He-Pin opened a new pull request, #3709: URL: https://github.com/apache/pekko/pull/3709
### Motivation When the shard **coordinator singleton's** lease is lost (`pekko.cluster.sharding.coordinator-singleton.use-lease`), `ClusterSingletonManager` sends the configured `terminationMessage` (`ShardCoordinator.Internal.Terminate`) and returns to `AcquiringLease`, intending to re-acquire and carry on. But `ShardCoordinator.terminate()` unconditionally treated the message as a node shutdown: it gracefully shut down the **local** `ShardRegion` and deferred its own stop until the region terminated. The node was still `Up`, yet its local region was gone and never recreated — `ClusterSharding` caches the region ref and `start` short-circuits on the cache. A transient lease-backend blip therefore permanently disabled that shard type on that node for the lifetime of the `ActorSystem`. ### Modification - `ShardCoordinator.terminate()`: only take the drain-local-region path when this node is actually leaving the cluster or shutting down — `CoordinatedShutdown.shutdownReason().isDefined`, `cluster.isTerminated`, or self member status `Leaving`/`Exiting`/`Down`. Otherwise (e.g. lease loss) stop only the coordinator; the local region keeps running and re-registers with the restarted coordinator via the normal coordinator-terminated flow. - Docs: note the coordinator-singleton lease behavior in the Lease section of `typed/cluster-sharding.md`. - New `ClusterShardingCoordinatorLeaseLossSpec` following the reproduction recipe from the issue: start sharding with a coordinator singleton `TestLease`, confirm an entity responds, fire `LeaseFailed` via the lease callback, wait for the re-acquire, then assert the *same cached region ref* still serves entities. The complementary hardening suggested in the issue (evicting terminated regions from the `ClusterSharding.regions` cache, same missing-watch pattern as #3691) is deliberately left out of this PR to keep it scoped; it can be a follow-up. ### Result A coordinator singleton lease loss restarts the coordinator without killing the local `ShardRegion`; entities on the node keep being served. Leaving/down/shutdown hand-off behavior is unchanged. ### Tests - `sbt "cluster-sharding / Test / testOnly org.apache.pekko.cluster.sharding.ClusterShardingCoordinatorLeaseLossSpec"`: passed (848 ms) - Directional check: reverted the fix, re-ran — failed as expected (region gracefully shut down; entity messages never answered, 20 s timeout) - Regression: `ClusterShardingInternalsSpec`, `HandOffStopperSpec`, `DDataClusterShardingLeaseSpec`: 8/8 passed - Full validation on CI (multi-jvm hand-off suites included) ### References Fixes #3697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
