[
https://issues.apache.org/jira/browse/SPARK-59993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haotian Sun updated SPARK-59993:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Collect scalar pandas UDF phase timing metrics
> ----------------------------------------------
>
> Key: SPARK-59993
> URL: https://issues.apache.org/jira/browse/SPARK-59993
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Haotian Sun
> Priority: Major
>
> Add task-local timing accumulation in the Python worker and report input
> conversion, UDF execution, and output conversion durations for scalar pandas
> UDFs, together with task and batch coverage counters.
> Decode numeric worker reports into a generic map and register the new SQL
> metrics so their values are available in the Spark SQL UI. Keep lifecycle
> timing calculations and metric updates in the Python runner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
