[
https://issues.apache.org/jira/browse/SPARK-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279863#comment-14279863
]
Patrick Wendell commented on SPARK-5216:
----------------------------------------
This has been proposed before, but in the past we decided not to do it. Trying
to extrapolate the finish time of a stage accurately is basically impossible
since in many workloads stragglers dominate the total response time. The
conclusion was that it was better to give no estimate rather than one which is
likely to be misleading.
> Spark Ui should report estimated time remaining for each stage.
> ---------------------------------------------------------------
>
> Key: SPARK-5216
> URL: https://issues.apache.org/jira/browse/SPARK-5216
> Project: Spark
> Issue Type: Wish
> Components: Spark Core, Web UI
> Affects Versions: 1.3.0
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
>
> Per stage feedback on estimated remaining time can help user get a grasp on
> how much time the job is going to take. This will only require changes on the
> UI/JobProgressListener side of code since we already have most of the
> information needed.
> In the initial cut, plan is to estimate time based on statistics of running
> job i.e. average time taken by each task and number of task per stage. This
> will makes sense when jobs are long. And then if this makes sense, then more
> heuristics can be added like projected time saved if the rdd is cached and so
> on.
> More precise details will come as this evolves. In the meantime thoughts on
> alternate ways and suggestion on usefulness are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
