venkata91 commented on PR #3842:
URL: https://github.com/apache/celeborn/pull/3842#issuecomment-5755178493
**Production evidence for this, from a ~500-worker 0.6.2 fleet.**
We see sustained `PUSH_DATA_TIMEOUT_REPLICA` bursts that match this failure
mode exactly. Over 7 days in one cluster:
```
ReplicateDataTimeoutCount 8,448,683 95.8%
ReplicateDataConnectionExceptionCount 310,022
ReplicateDataCreateConnectionFailCount 57,049
ReplicateDataFailCount 0
ReplicateDataWriteFailCount 0
```
`ReplicateDataFailCount` and `ReplicateDataWriteFailCount` are exactly zero,
so no replica ever rejected a write. It is all requests expiring in
`failExpiredPushRequest`.
A representative burst:
```
02:12 worker-447 restarts on node A, re-registers
02:33 worker-447 restarts again, on node B (new pod IP), re-registers
02:33+ replication timeout burst begins
02:49 PrimaryPushDataTime p99 pins at 305,285 ms (client push timeout is
300s),
sustained ~15 min; baseline is 0.02-10 ms
03:46 same primary/peer pairs timing out repeatedly, same second, many
partitions
```
Two things corroborate the stale-pooled-client diagnosis:
- **Zero** `"Still have N requests outstanding when connection ... is
closed"` log lines during the burst. `channelInactive` never fired, so the
channel did report active, exactly as described here.
- `PbWorkerLost` was 0 throughout. These are graceful restarts with stable
worker identity, so the master re-registers rather than declaring the worker
lost. Nothing upstream of the transport layer notices.
**On scale:** without invalidation the poisoned client stays eligible for
reuse, so every subsequent push to that peer expires at 300s as well. One
20-minute window recorded 1,974,971 increments, 34% of a three-day total. The
repeat rate looks unbounded. Adding `unavailablePeers` on the timeout branch
bounds it to roughly one generation per peer per `replicate.fastFail.duration`.
Client-visible cost is real: `PrimaryPushDataTime` is the time a Spark
executor's push is held, so a pinned p99 is `shuffleWriteTime` directly.
Would be glad to see this merged, and we would need a `branch-0.6` backport
to pick it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
