[
https://issues.apache.org/jira/browse/SPARK-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224185#comment-14224185
]
Sandy Ryza commented on SPARK-4584:
-----------------------------------
I took a look at the jobs Nishkam ran before and after that commit. The second
stage in the "before" job takes 69 seconds and the second stage in the "after"
job takes 158 seconds. This seems to be caused by the individual tasks taking
longer.
Totally confused about how that commit could have caused the regression.
> 2x Performance regression for Spark-on-YARN
> -------------------------------------------
>
> Key: SPARK-4584
> URL: https://issues.apache.org/jira/browse/SPARK-4584
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.2.0
> Reporter: Nishkam Ravi
> Priority: Blocker
>
> Significant performance regression observed for Spark-on-YARN (upto 2x) after
> 1.2 rebase. The offending commit is: 70e824f750aa8ed446eec104ba158b0503ba58a9
> from Oct 7th. Problem can be reproduced with JavaWordCount against a large
> enough input dataset in YARN cluster mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
