zaynt4606 commented on code in PR #3872:
URL: https://github.com/apache/celeborn/pull/3872#discussion_r4332762439
##########
client-spark/spark-3-ui/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornListener.scala:
##########
@@ -71,6 +73,17 @@ private[celeborn] class CelebornListener(
mayUpdate()
}
+ override def onOtherEvent(event: SparkListenerEvent): Unit = {
+ event match {
+ case e: CelebornMapperMetricsEvent =>
+ if (pluginEnabled) {
+ totalUncompressedWriteBytes.addAndGet(e.getUncompressedBytesWritten)
Review Comment:
Thanks for the update. There is still a gap in the assumption that failed
attempts never report MapperEnd: the writer can finish and report MapperEnd
before Spark finishes the task. A task-completion listener can then throw,
causing that attempt to fail and be retried.
I reproduced this on ea7a4fa3 with Spark 3.5.8, local[2,2], two map
partitions, and an injected task-completion-listener failure on partition 0 /
attempt 0 after the shuffle writer completes. The retry succeeds and the job
result is correct, but the metrics are:
- MapperMetrics events (uncompressed bytes): 20, 20, 20, including the
failed attempt.
- Successful map attempts (compressed bytes): 94, 94.
- Failed map attempt (compressed bytes): 94, excluded by onTaskEnd.
- UI totals: 60 / 188, rather than 40 / 188 for the successful attempts,
overstating the ratio by 50%.
The onTaskEnd and mapper-event accumulation code, both writers, and the
event payload are unchanged in 96f8cc84. The event still lacks map/attempt
identity to reconcile it with the final task outcome. Could we derive both
counters from the same set of attempts (with enough identity to reconcile them
if using task-success filtering) and add a regression where the task fails
after writer completion? This is distinct from deduplicating MapperEnd RPC
retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
