[
https://issues.apache.org/jira/browse/SPARK-59772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haotian Sun updated SPARK-59772:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Improve Python UDF worker metrics collection and reporting
> ----------------------------------------------------------
>
> Key: SPARK-59772
> URL: https://issues.apache.org/jira/browse/SPARK-59772
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Haotian Sun
> Priority: Major
>
> Improve collection and reporting of Python UDF worker metrics, building on
> the existing SQL metrics in SPARK-34265.
> Planned OSS work:
> 1. Add a versioned, extensible Python worker-to-JVM metrics report while
> preserving the legacy timing protocol.
> 2. Collect additional low-overhead worker measurements at defined execution
> boundaries.
> 3. Register and aggregate selected measurements as Spark SQL metrics and
> expose them through existing Spark observability paths.
> The protocol change is the first subtask. Additional measurements and SQL
> metric registration will be separate subtasks once their semantics are
> defined.
> Related history: SPARK-30306 proposed counters through the Spark Metrics
> System; SPARK-34265 added the current SQL timing metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
