He-Pin opened a new issue, #3697:
URL: https://github.com/apache/pekko/issues/3697
## Summary
When the shard **coordinator singleton's** lease is lost,
`ClusterSingletonManager` sends the configured `terminationMessage` to the
singleton and returns to `AcquiringLease`, intending to re-acquire and carry
on. For cluster sharding that message is `ShardCoordinator.Internal.Terminate`,
which makes the coordinator gracefully shut down the **local** `ShardRegion`
and then stop itself. The node is still `Up` and still supposed to host shards,
but its local region is gone and is never recreated, because `ClusterSharding`
caches the region ref and `start` short-circuits on the cache.
A transient lease-backend blip therefore permanently disables that shard
type on that node for the lifetime of the `ActorSystem`.
### Scope
This requires a lease to be configured **for the coordinator singleton**,
i.e. `pekko.cluster.sharding.coordinator-singleton.use-lease`. That is a
separate setting from `pekko.cluster.sharding.use-lease`, which per
`cluster-sharding/src/main/resources/reference.conf:423-426` applies only to
per-shard leases:
```
# Config path of the lease that each shard must acquire before starting
entity actors
# default is no lease
# A lease can also be used for the singleton coordinator by settings it in
the coordinator-singleton properties
use-lease = ""
```
and `reference.conf:369-370`:
```
# A lease can be configured in these settings for the coordinator singleton
coordinator-singleton = ${pekko.cluster.singleton}
```
## The chain
**1. The termination message is `ShardCoordinator.Internal.Terminate`** —
`cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/ClusterSharding.scala:833-837`:
```scala
context.actorOf(
ClusterSingletonManager
.props(singletonProps, terminationMessage =
ShardCoordinator.Internal.Terminate, singletonSettings)
```
Note also `:826`, `.withFinalStopMessage(_ ==
ShardCoordinator.Internal.Terminate)` — the backoff supervisor treats this
message as final, so the coordinator child is not restarted by backoff either.
**2. Lease loss fires it while the node is still `Up`** —
`cluster-tools/src/main/scala/org/apache/pekko/cluster/singleton/ClusterSingletonManager.scala:990-997`:
```scala
case Event(LeaseLost(reason), OldestData(singleton)) =>
log.warning("Lease has been lost. Reason: {}. Terminating singleton
and trying to re-acquire lease", reason)
singleton match {
case Some(s) =>
s ! terminationMessage
goto(AcquiringLease).using(AcquiringLeaseData(leaseRequestInProgress = false,
singleton))
case None =>
tryAcquireLease()
}
```
Nothing here consults the node's own member status. The intent, per the log
message, is explicitly to re-acquire afterwards.
**3. The coordinator shuts down the local region** —
`cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/ShardCoordinator.scala:964-978`:
```scala
private def terminate(): Unit = {
if (aliveRegions.exists(_.path.address.hasLocalScope) ||
gracefulShutdownInProgress.exists(
_.path.address.hasLocalScope)) {
aliveRegions
.find(_.path.address.hasLocalScope)
.foreach(region =>
// region will get this from taking part in coordinated shutdown,
but for good measure
region ! ShardRegion.GracefulShutdown)
log.debug("{}: Deferring coordinator termination until local region
has terminated", typeName)
waitingForLocalRegionToTerminate = true
```
`terminate()` has no notion of *why* it was called. The comment shows the
assumption is a coordinated shutdown, which is not the case here.
**4. The region drains and stops** — `ShardRegion.scala:935-957` sets
`gracefulShutdownInProgress = true` (the
`CoordinatedShutdown.getShutdownReason()` check at `:945-952` only gates the
*timeout timer*, not the shutdown itself), then `:1109-1113`:
```scala
private def tryCompleteGracefulShutdownIfInProgress(): Unit =
if (gracefulShutdownInProgress && shards.isEmpty &&
shardBuffers.isEmpty) {
log.debug("{}: Completed graceful shutdown of region.", typeName)
context.stop(self) // all shards have been rebalanced, complete
graceful shutdown
}
```
**5. The coordinator then stops itself** —
`ShardCoordinator.scala:1236-1243`:
```scala
aliveRegions -= ref
allocateShardHomesForRememberEntities()
if (ref.path.address.hasLocalScope &&
waitingForLocalRegionToTerminate) {
// handoff optimization: singleton told coordinator to stop but we
deferred stop until the local region
// had completed the handoff
log.debug("{}: Local region stopped, terminating coordinator",
typeName)
terminate()
}
```
**6. The dead region ref stays cached forever** —
`ClusterSharding.scala:189` `private val regions: ConcurrentHashMap[String,
ActorRef]`; `start` at `:301-315`:
```scala
regions.get(typeName) match {
case null => ... regions.put(typeName, shardRegion); shardRegion
case ref => ref // already started, use cached ActorRef
}
```
and `shardRegion(typeName)` at `:644-654` returns the same cached ref.
Nothing watches the region or removes the entry when it terminates.
**Net effect:** the coordinator singleton does come back —
`ClusterSingletonManager` re-acquires the lease and starts a fresh one — but
the local `ShardRegion` does not.
`ClusterSharding(system).shardRegion(typeName)` keeps returning a terminated
ref, so entity messages go to dead letters, with no log line connecting it to
the earlier lease warning.
## Existing coverage gap
`cluster-sharding/src/test/scala/org/apache/pekko/cluster/sharding/ClusterShardingLeaseSpec.scala`
has a "recover if lease lost" case, and it does pass — but it exercises a
**shard** lease, not the coordinator's:
```scala
"recover if lease lost" in {
region ! 4
...
val testLease = leaseForShard(4)
testLease.initialPromise.complete(Success(true))
expectMsg(4)
testLease.getCurrentCallback()(Option(LeaseFailed("oh dear")))
awaitAssert({ region ! 4; expectMsg(4) }, max = 10.seconds)
}
```
with `leaseForShard` / `leaseNameFor` at `:109-114` resolving to
`s"${system.name}-shard-$typeName-$shardId"`. The spec's config sets
`pekko.cluster.sharding.use-lease`, which does not give the coordinator
singleton a lease. So the coordinator path is untested.
## Suggested direction
`ShardCoordinator.terminate()` needs to distinguish "this node is leaving
the cluster, so drain and stop the local region" from "the singleton is being
restarted for a reason unrelated to membership". A guard along the lines of *is
this node actually leaving* — `cluster.isTerminated`, or a
`CoordinatedShutdown` reason is present, or the self member's status is
`Leaving`/`Exiting`/`Down` — would let the lease-loss path stop only the
coordinator and leave the local region running.
An independent, complementary hardening is to stop serving a terminated ref
from the `regions` cache (watch the region and evict the entry, so a later
`start` recreates it). That is tracked separately and would limit the blast
radius of any future path that stops a region unexpectedly.
## Reproduction recipe
Configure `pekko.cluster.sharding.coordinator-singleton.use-lease =
"test-lease"` on top of the existing `ClusterShardingLeaseSpec` setup. The
coordinator singleton's lease name is derived at
`ClusterSingletonManager.scala:521-525`:
```scala
val lease: Option[Lease] = settings.leaseSettings.map { settings =>
val leaseName =
if (settings.leaseName.isEmpty)
s"${context.system.name}-singleton-${self.path}"
else settings.leaseName
```
so it can be resolved from `TestLeaseExt` once the singleton has started.
Then:
1. join a single-node cluster, start sharding, let the coordinator acquire
its lease;
2. send a message and confirm an entity responds;
3. fire `coordinatorLease.getCurrentCallback()(Option(LeaseFailed("blip")))`;
4. assert that `ClusterSharding(system).shardRegion(typeName)` still
delivers — today it should instead be a terminated ref, with the message
landing in dead letters.
Step 4 is the assertion that should fail before the fix.
## Verification status
The chain above is traced statically; every link is cited from current
`main`. I have **not** yet run the reproduction — the recipe is included so the
fix can be developed against a failing test, per the contribution guideline
that a bug-fix test must expose the issue first. If the reproduction does not
fail as described, the analysis above is wrong somewhere and this should be
closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
