[
https://issues.apache.org/jira/browse/SPARK-62067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haotian Sun updated SPARK-62067:
--------------------------------
Description: Python worker timing metrics are floored to whole milliseconds
per task, so sub-millisecond work reports 0 and sums over many tasks
undercount. Report the worker's timestamps and durations in nanoseconds, and
expose the Python timing SQL metrics as nanosecond timing metrics whose keys
and names include the unit. (was: Python worker timing metrics are floored to
whole milliseconds per task, so sub-millisecond work reports 0 and sums over
many tasks undercount. Report the worker's timestamps and durations in
microseconds, and expose the Python timing SQL metrics as microsecond sums
whose keys and names include the unit.)
> Report Python worker timing metrics in nanoseconds
> --------------------------------------------------
>
> Key: SPARK-62067
> URL: https://issues.apache.org/jira/browse/SPARK-62067
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Haotian Sun
> Priority: Major
> Labels: pull-request-available
>
> Python worker timing metrics are floored to whole milliseconds per task, so
> sub-millisecond work reports 0 and sums over many tasks undercount. Report
> the worker's timestamps and durations in nanoseconds, and expose the Python
> timing SQL metrics as nanosecond timing metrics whose keys and names include
> the unit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]