cloud-fan commented on code in PR #56649:
URL: https://github.com/apache/spark/pull/56649#discussion_r3456930272
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonSQLMetrics.scala:
##########
@@ -28,6 +28,8 @@ trait PythonSQLMetrics { self: SparkPlan =>
k -> SQLMetrics.createTimingMetric(sparkContext, v)
} ++ PythonSQLMetrics.pythonOtherMetricsDesc.map { case (k, v) =>
k -> SQLMetrics.createMetric(sparkContext, v)
+ } ++ PythonSQLMetrics.pythonBatchSizeMetricsDesc.map { case (k, v) =>
Review Comment:
These three batch metrics live in the shared `PythonSQLMetrics` trait, so
they surface (as `0` / `0.0 B`) on every operator that mixes it in — the Arrow
eval/agg/window/cogroup execs, `MapInBatchExec`, and the streaming Python
stateful operators — none of which pickle, so the values stay zero. The other
shared metrics are populated on all paths, so this is the first pickle-specific
one to leak onto those nodes. Non-blocking, and a fair tradeoff against keeping
a single metrics map — but if the always-zero rows bother you, a small
`PythonPickleBatchMetrics` trait on just
`BatchEvalPythonExec`/`BatchEvalPythonUDTFExec` would scope them, and
`getInputIterator` (which looks them up by key) wouldn't change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
