yew1eb commented on code in PR #3872:
URL: https://github.com/apache/celeborn/pull/3872#discussion_r4348048528
##########
client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java:
##########
@@ -162,6 +163,19 @@ private void initializeLifecycleManager(String appId) {
lifecycleManager = new LifecycleManager(appUniqueId, celebornConf);
lifecycleManager.applicationCount().increment();
lifecycleManager.registerCancelShuffleCallback(SparkUtils::cancelShuffle);
+ // Forward mapper metrics to the Celeborn UI extension.
+ lifecycleManager.registerMapperEndCallback(
Review Comment:
Gated in da2330f98 — the callback is now registered only when
`SparkUtils.isCelebornPluginEnabled(conf)` finds the plugin class in
`spark.plugins` or `spark.plugins.defaultList` (the same keys the listener
checks on replay), so apps that never opted in no longer post one event per map
task or flush the event log per event.
Aggregating per shuffle and posting once at StageEnd would additionally cut
the event volume for opted-in apps; I have left it as a follow-up to keep this
change small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
