He-Pin opened a new issue, #3686:
URL: https://github.com/apache/pekko/issues/3686
## Summary
While a shard is waiting to acquire its lease, its receive function stashes
every message that is not lease- or membership-related. `HandOff` is one of
those, so the coordinator never receives `ShardStopped` and the rebalance
cannot complete. Because a failed acquisition is retried indefinitely, this
persists for as long as the lease backend is unavailable.
## Evidence
`cluster-sharding/src/main/scala/org/apache/pekko/cluster/sharding/Shard.scala:535-559`:
```scala
// Don't send back ShardInitialized so that messages are buffered in the
ShardRegion
// while awaiting the lease
private def awaitingLease(): Receive = {
case LeaseAcquireResult(true, _) => ...
tryLoadRememberedEntities()
case LeaseAcquireResult(false, None) => ...
timers.startSingleTimer(LeaseRetryTimer, LeaseRetry, leaseRetryInterval)
case LeaseAcquireResult(false, Some(t)) => ...
timers.startSingleTimer(LeaseRetryTimer, LeaseRetry, leaseRetryInterval)
case LeaseRetry => tryGetLease(lease.get)
case ll: LeaseLost => receiveLeaseLost(ll)
case me: MemberEvent => receiveMemberEvent(me)
case msg =>
...
stash() // :551-558 — HandOff lands here
}
```
There is no `HandOff` case. The stash is only replayed once the lease is
acquired (`LeaseAcquireResult(true, _)` → `tryLoadRememberedEntities()`), and a
failed acquisition reschedules `LeaseRetry` forever — so an unobtainable lease
means the stashed `HandOff` is never processed.
## Impact
The coordinator's `RebalanceWorker` waits for `ShardStopped` from the shard
being handed off. It never arrives, so the rebalance times out and the shard
cannot be moved away. This is precisely the scenario where rebalancing matters:
a degraded lease backend combined with a topology change.
## Suggested direction
A shard in `awaitingLease` has not sent `ShardInitialized` and therefore
hosts no entities (see the comment at `:533-534`). It can answer `HandOff`
immediately with `ShardStopped(shardId)` and stop itself, rather than stashing
it.
## Test
`cluster-sharding/src/test/scala/org/apache/pekko/cluster/sharding/ShardWithLeaseSpec.scala`
already drives a shard through lease acquisition using `TestLease`. A case
that leaves acquisition pending and then sends `HandOff` should assert that
`ShardStopped` is received by the coordinator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
