He-Pin opened a new issue, #3691:
URL: https://github.com/apache/pekko/issues/3691
## Summary
`ClusterShardingGuardian` creates one `Replicator` per role as its own child
and caches the `ActorRef`, but never watches it. If the replicator stops, the
cached ref stays in the map and is handed out to every later caller, so the
remember-entities store and the shard coordinator end up sending to dead
letters with no path to recovery.
## Evidence
`cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/ClusterSharding.scala:723`
declares the guardian; `:731` the cache; `:751-764` the accessor:
```scala
private def replicator(role: Option[String], settings:
ClusterShardingSettings): ActorRef = {
// one Replicator per role
replicatorByRole.get(role) match {
case Some(ref) => ref
case None =>
val name = role match {
case Some(r) => URLEncoder.encode(r, StandardCharsets.UTF_8) +
"Replicator"
case None => "replicator"
}
val ref = context.actorOf(Replicator.props(replicatorSettings(role,
settings)), name)
replicatorByRole = replicatorByRole.updated(role, ref)
ref
}
}
```
The file contains no `context.watch` and no `case Terminated` at all.
`Replicator.props`
(`distributed-data/src/main/scala/org/apache/pekko/cluster/ddata/Replicator.scala:365-370`)
is a plain `Props` with no backoff supervision:
```scala
def props(settings: ReplicatorSettings): Props = {
require(
settings.durableKeys.isEmpty || (settings.durableStoreProps !=
Right(Props.empty)),
"durableStoreProps must be defined when durableKeys are defined")
Props(new
Replicator(settings)).withDeploy(Deploy.local).withDispatcher(settings.dispatcher)
}
```
Callers of `replicator(...)` are `:783` (the
`DdataRememberEntitiesShardStore` / coordinator store) and `:809` (the
coordinator itself).
## When the replicator actually stops
The default supervision strategy restarts, and a restart preserves the
`ActorRef`, so an ordinary failure does not trigger this. The reachable path is
the replicator stopping *itself* — `Replicator.scala:2468-2470`:
```scala
def receiveMemberRemoved(m: Member): Unit = {
if (m.address == selfAddress)
context.stop(self)
```
reached from the dispatch at `:1620-1621`:
```scala
case MemberRemoved(m, _) => receiveMemberRemoved(m)
case ClusterShuttingDown => receiveMemberRemoved(cluster.selfMember)
```
So a node that is removed from the cluster (or whose cluster shuts down)
stops its replicator while the guardian keeps the stale ref. Any later
`replicator(role, settings)` call — for example starting a shard region again
on the same `ActorSystem` after rejoining — returns the dead ref.
## Impact
Remember-entities reads and writes, and coordinator state propagation,
silently go to dead letters. There is no log line pointing at the cause and no
automatic re-creation, so the shard type is unusable for the lifetime of that
`ActorSystem`.
## Suggested direction
`context.watch` the replicator when caching it and drop the
`replicatorByRole` entry on `Terminated`, so the next `replicator(...)` call
creates a fresh one. Whether a stale-ref caller should instead fail fast is a
separate question, but at minimum the cache should not keep serving a ref known
to be dead.
## Test
`cluster-sharding/src/test/scala/org/apache/pekko/cluster/sharding/ClusterShardingLeaseSpec.scala`
and `ShardWithLeaseSpec.scala` already build sharding setups over `TestLease`;
a ddata-mode case can watch the replicator, stop it, then start another shard
region of the same type and assert the new replicator is alive rather than a
terminated ref.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
