yew1eb commented on code in PR #3872:
URL: https://github.com/apache/celeborn/pull/3872#discussion_r4348052295
##########
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornListener.scala:
##########
@@ -62,16 +67,42 @@ private[celeborn] class CelebornListener(
return
}
Option(taskEnd.taskMetrics).foreach { metrics =>
- totalWriteBytes.addAndGet(metrics.shuffleWriteMetrics.bytesWritten)
- // writeTime is in nanoseconds; normalize to ms.
- totalWriteTimeMs.addAndGet(metrics.shuffleWriteMetrics.writeTime /
1000000L)
+ taskEnd.reason match {
+ case Success =>
+ // Write-side metrics only for successful attempts. The compression
ratio
Review Comment:
You are right, it was left over from the earlier split-source ratio —
reverted in 5643bb773. Write bytes/time now count every attempt like the read
side and task duration, matching `AppStatusListener`'s stage accounting.
`shuffleWriteMetricsCountEveryAttempt` covers a killed attempt being counted.
##########
client/src/main/java/org/apache/celeborn/client/ShuffleClient.java:
##########
@@ -372,9 +373,19 @@ public abstract int mergeData(
// Report partition locations written by the completed map task of
ReducePartition Shuffle Type.
public abstract void mapperEnd(
- int shuffleId, int mapId, int attemptId, int numMappers, int
numPartitions)
+ int shuffleId,
Review Comment:
Fixed in 45ee042a3 — the 5-arg `mapperEnd` is abstract again, and the new
6-arg overload is a default method that discards the metrics and delegates to
the 5-arg one, so external subclasses keep compiling with just the 5-arg
implementation. `ShuffleClientImpl` overrides the 6-arg one to pass the metrics
through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
