Haotian Sun created SPARK-59773:
-----------------------------------
Summary: Add versioned Python worker metrics reports
Key: SPARK-59773
URL: https://issues.apache.org/jira/browse/SPARK-59773
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
Introduce a bounded, length-prefixed JSON report over the Python worker socket
with named metric values and units. The JVM advertises v1 support per task, and
workers retain the legacy TIMING_DATA report when that support is absent.
Decode generic metric entries on the JVM while the timing consumer validates
the four existing timing values. Preserve the spill-count and accumulator
trailer, and cover the format with Python, Scala, and live RDD/Arrow worker
tests.
Draft PR: https://github.com/apache/spark/pull/59019
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
