[
https://issues.apache.org/jira/browse/SPARK-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kay Ousterhout updated SPARK-3983:
--
Description:
The reported scheduler delay includes time to get a new thread (from a
threadpool) in order to start the task, time to deserialize the task, and time
to serialize the result. None of these things are delay caused by the
scheduler; including them as such is misleading.
This is especially problematic when debugging performance of short tasks (that
run in 10s of milliseconds), when the scheduler delay can be very large
relative to the task duration.
cc [~sparks] [~shivaram]
was:
The reported scheduler delay includes time to get a new thread (from a
threadpool) in order to start the task, time to deserialize the task, and time
to serialize the result. None of these things are delay caused by the
scheduler; including them as such is misleading.
cc [~sparks] [~shivaram]
Scheduler delay (shown in the UI) is incorrect
--
Key: SPARK-3983
URL: https://issues.apache.org/jira/browse/SPARK-3983
Project: Spark
Issue Type: Bug
Reporter: Kay Ousterhout
Assignee: Kay Ousterhout
Fix For: 1.2.0
The reported scheduler delay includes time to get a new thread (from a
threadpool) in order to start the task, time to deserialize the task, and
time to serialize the result. None of these things are delay caused by the
scheduler; including them as such is misleading.
This is especially problematic when debugging performance of short tasks
(that run in 10s of milliseconds), when the scheduler delay can be very large
relative to the task duration.
cc [~sparks] [~shivaram]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
