[
https://issues.apache.org/jira/browse/SPARK-20139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014391#comment-16014391
]
Ivan Gozali commented on SPARK-20139:
-------------------------------------
Facing a similar issue with 8 [AWS
m2.4xlarge|https://aws.amazon.com/ec2/previous-generation/] worker nodes and
the following extra options.
{code}
--conf,spark.driver.maxResultSize=8G,--conf,spark.yarn.executor.memoryOverhead=2400,--conf,spark.yarn.driver.memoryOverhead=2400,--conf,spark.yarn.maxAppAttempts=1,--conf,spark.dynamicAllocation.maxExecutors=40,--driver-memory,9625M,--executor-memory,9625M,--executor-cores,12
{code}
Spark version:
{code}
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_121
Branch HEAD
Compiled by user ec2-user on 2017-02-17T19:03:33Z
Revision 30eccced8ce8c483445f0aa3175ce725831ff06b
Url [email protected]:/pkg/Aws157BigTop
Type --help for more information.
{code}
I also provided a screenshot (screenshot-2.png).
> Spark UI reports partial success for completed stage while log shows all
> tasks are finished
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-20139
> URL: https://issues.apache.org/jira/browse/SPARK-20139
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.1.0
> Reporter: Etti Gur
> Attachments: screenshot-1.png
>
>
> Spark UI reports partial success for completed stage while log shows all
> tasks are finished - i.e.:
> We have a stage that is presented under completed stages on spark UI,
> but the successful tasks are shown like so: (146372/524964) not as you'd
> expect (524964/524964)
> Looking at the application master log shows all tasks in that stage are
> successful:
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 522973.0 in stage 0.0
> (TID 522973) in 1163910 ms on ip-10-1-15-34.ec2.internal (executor 116)
> (524963/524964)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 12508.0 in stage 2.0
> (TID 537472) in 241250 ms on ip-10-1-15-14.ec2.internal (executor 38)
> (20234/20262)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 12465.0 in stage 2.0
> (TID 537429) in 241994 ms on ip-10-1-15-106.ec2.internal (executor 133)
> (20235/20262)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 15079.0 in stage 2.0
> (TID 540043) in 202889 ms on ip-10-1-15-173.ec2.internal (executor 295)
> (20236/20262)
> 17/03/29 09:45:49 INFO TaskSetManager: Finished task 19828.0 in stage 2.0
> (TID 544792) in 137845 ms on ip-10-1-15-147.ec2.internal (executor 43)
> (20237/20262)
> 17/03/29 09:45:50 INFO TaskSetManager: Finished task 19072.0 in stage 2.0
> (TID 544036) in 147363 ms on ip-10-1-15-19.ec2.internal (executor 175)
> (20238/20262)
> 17/03/29 09:45:50 INFO TaskSetManager: Finished task 524146.0 in stage 0.0
> (TID 524146) in 889950 ms on ip-10-1-15-72.ec2.internal (executor 74)
> (524964/524964)
> Also in the log we get an error:
> 17/03/29 08:24:16 ERROR LiveListenerBus: Dropping SparkListenerEvent because
> no remaining room in event queue. This likely means one of the SparkListeners
> is too slow and cannot keep up with the rate at which tasks are being started
> by the scheduler.
> This looks like the stage is indeed completed with all its tasks but UI shows
> like not all tasks really finished.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
