[
https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384571#comment-15384571
]
Thomas Graves commented on SPARK-15703:
---------------------------------------
I think we should do both. event queue size configurable can always be a
backup plan and if someone goes really large scale it might need to be bigger.
As long as you increase memory to match there shouldn't be any issues with
making it larger. So configurable and we can leave it undocumented for now
(internal config) and if people run into the issue we can just document it.
Then we obviously need to fix the issue itself. SPARK-16441 seems to be the
same thing just manifesting slightly differently.
> Spark UI doesn't show all tasks as completed when it should
> -----------------------------------------------------------
>
> Key: SPARK-15703
> URL: https://issues.apache.org/jira/browse/SPARK-15703
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Web UI
> Affects Versions: 2.0.0
> Reporter: Thomas Graves
> Priority: Critical
> Attachments: Screen Shot 2016-06-01 at 11.21.32 AM.png, Screen Shot
> 2016-06-01 at 11.23.48 AM.png, SparkListenerBus .png,
> spark-dynamic-executor-allocation.png
>
>
> The Spark UI doesn't seem to be showing all the tasks and metrics.
> I ran a job with 100000 tasks but Detail stage page says it completed 93029:
> Summary Metrics for 93029 Completed Tasks
> The Stages for all jobs pages list that only 89519/100000 tasks finished but
> its completed. The metrics for shuffled write and input are also incorrect.
> I will attach screen shots.
> I checked the logs and it does show that all the tasks actually finished.
> 16/06/01 16:15:42 INFO TaskSetManager: Finished task 59880.0 in stage 2.0
> (TID 54038) in 265309 ms on 10.213.45.51 (100000/100000)
> 16/06/01 16:15:42 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks
> have all completed, from pool
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
