[
https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417929#comment-15417929
]
Kay Ousterhout commented on SPARK-3577:
---------------------------------------
I believe spill time will currently be displayed as part of the task runtime,
but not as part of scheduler delay.
The scheduler delay is calculated by looking at the difference between two
values:
(1) The time that the task was running on the executor
(2) The time from when the scheduler sent information about the task to the
executor (so the executor could run the task) until the scheduler received a
message that the task completed.
Scheduler delay is (2) - (1). Usually when it's high, it's because of queueing
delays in the scheduler that are either delaying the task getting sent to the
executor (e.g., because the scheduler has a long queue of other tasks that need
to be launched, or because tasks are large so take a while to send over the
network) or that are delaying the task completion message getting back to the
scheduler (which can happen when the rate of task launch is high -- greater
than 1K or so task launches / second).
> Add task metric to report spill time
> ------------------------------------
>
> Key: SPARK-3577
> URL: https://issues.apache.org/jira/browse/SPARK-3577
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, Spark Core
> Affects Versions: 1.1.0
> Reporter: Kay Ousterhout
> Priority: Minor
>
> The {{ExternalSorter}} passes its own {{ShuffleWriteMetrics}} into
> {{ExternalSorter}}. The write time recorded in those metrics is never used.
> We should probably add task metrics to report this spill time, since for
> shuffles, this would have previously been reported as part of shuffle write
> time (with the original hash-based sorter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
