[
https://issues.apache.org/jira/browse/SPARK-59993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tian Gao resolved SPARK-59993.
------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59246
[https://github.com/apache/spark/pull/59246]
> Collect worker and Arrow-optimized Python UDF timing metrics
> ------------------------------------------------------------
>
> Key: SPARK-59993
> URL: https://issues.apache.org/jira/browse/SPARK-59993
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Haotian Sun
> Assignee: Haotian Sun
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Collect Python worker metrics with a collector shared within each worker
> process and reset at task entry. Decode numeric worker reports through the
> generic Scala reader while preserving lifecycle timing and spill accounting.
> For modern Arrow-optimized scalar Python UDFs (SQL_ARROW_BATCHED_UDF with
> non-legacy conversion), record data read, input preparation, UDF execution,
> output preparation, and data write durations, plus timed task and batch
> counts. Record the count of pipelined Python worker tasks so analyses can
> identify operators whose read and UDF intervals may overlap. Register the new
> values as SQL metrics for the Spark UI and QPL pipeline. These are internal
> metrics with no user-facing behavior change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
