Haotian Sun created SPARK-59772:
-----------------------------------
Summary: Improve Python UDF worker metrics collection and reporting
Key: SPARK-59772
URL: https://issues.apache.org/jira/browse/SPARK-59772
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
Improve collection and reporting of Python UDF worker metrics, building on the
existing SQL metrics in SPARK-34265.
Planned OSS work:
1. Add a versioned, extensible Python worker-to-JVM metrics report while
preserving the legacy timing protocol.
2. Collect additional low-overhead worker measurements at defined execution
boundaries.
3. Register and aggregate selected measurements as Spark SQL metrics and expose
them through existing Spark observability paths.
The protocol change is the first subtask. Additional measurements and SQL
metric registration will be separate subtasks once their semantics are defined.
Related history: SPARK-30306 proposed counters through the Spark Metrics
System; SPARK-34265 added the current SQL timing metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
