wang-haihua opened a new pull request, #3805:
URL: https://github.com/apache/celeborn/pull/3805
### What changes were proposed in this pull request?
Add an INFO-level log in `ShuffleClientImpl.readPartition()`, right before
the
`CelebornInputStream` is created, recording the worker hosts a reduce
partition
reads its data from:
```
Reduce task for shuffle {} partition {} reads data from worker hosts: {}
```
Hosts are de-duplicated (order preserved) since a partition split across
epochs
may reuse the same worker. `readPartition()` is the single, engine-agnostic
read
entry point, so the log applies to the Spark, MR and Tez clients.
### Why are the changes needed?
When a Spark job is slow on shuffle read, the Stages UI shows almost the
entire
task duration is spent in "Shuffle Read Blocked Time" (e.g. 14-19 min out of
a
16-20 min task). But the only host shown there is the executor running the
reduce task, not the Celeborn worker actually serving the data — so operators
cannot tell which worker is the slow one. This log provides that missing
partition-to-worker(s) mapping on the normal read path.
JIRA: https://issues.apache.org/jira/browse/CELEBORN-2424
### Does this PR introduce any user-facing change?
No. Log output only.
### How was this patch tested?
Existing tests. The change is a single log statement on the read path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
