yaooqinn commented on code in PR #12114:
URL: https://github.com/apache/gluten/pull/12114#discussion_r3265040395
##########
backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala:
##########
@@ -258,6 +258,18 @@ class ColumnarShuffleWriter[K, V](
dep.metrics("dataSize").add(splitResult.getRawPartitionLengths.sum)
dep.metrics("compressTime").add(splitResult.getTotalCompressTime)
dep.metrics("peakBytes").add(splitResult.getPeakBytes)
+ // Backend-specific custom metrics (see
GlutenSplitResult.getCustomMetrics).
+ // Only entries whose key is pre-registered in `dep.metrics` are recorded;
+ // unknown keys are silently dropped so that the native side can ship new
+ // metrics ahead of the Spark-side `VeloxMetricsApi` registration without
+ // breaking older builds.
+ splitResult.getCustomMetrics.forEach {
+ (key, value) =>
+ val m = dep.metrics.get(key)
+ if (m.isDefined) {
+ m.get.add(value)
Review Comment:
`splitResult.getCustomMetrics()` is consumed here but not in
`VeloxCelebornColumnarShuffleWriter.scala:101+` or
`VeloxUniffleColumnarShuffleWriter.java:232`. RSS users won't see new
`Velox.InputEncoding.*` metrics. Suggest a shared
`applyCustomMetrics(splitResult, dep)` helper called from all three writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
