[
https://issues.apache.org/jira/browse/SPARK-20219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957253#comment-15957253
]
Kay Ousterhout commented on SPARK-20219:
----------------------------------------
Like [~mridulm80] (as mentioned on the PR) I'm hesitant about this idea because
of the added complexity and information "leakage" from the TaskScheduler back
to the DAGScheduler (in general, we should be making this interface between
these components smaller, to make the code easier to reason about -- not
larger). [[email protected]] you mentioned some use cases when this is
helpful; can you post some concrete performance numbers about difference in
runtimes?
cc [~imranr]-- thoughts here about whether the performance improvement is worth
the added complexity?
> Schedule tasks based on size of input from ScheduledRDD
> -------------------------------------------------------
>
> Key: SPARK-20219
> URL: https://issues.apache.org/jira/browse/SPARK-20219
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: jin xing
>
> When data is highly skewed on ShuffledRDD, it make sense to launch those
> tasks which process much more input as soon as possible. The current
> scheduling mechanism in *TaskSetManager* is quite simple:
> {code}
> for (i <- (0 until numTasks).reverse) {
> addPendingTask(i)
> }
> {code}
> In scenario that "large tasks" locate at bottom half of tasks array, if tasks
> with much more input are launched early, we can significantly reduce the time
> cost and save resource when *"dynamic allocation"* is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
