[
https://issues.apache.org/jira/browse/MAPREDUCE-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164900#comment-13164900
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3402:
----------------------------------------------------
[~karams] had been extremely helpful in running various tests to hunt this
down. And we finally got some results after a couple of weeks of hard work.
Turns out that most of the issues are because we made a switch from 32 bit JVMs
to 64 bit. Using compressed references dramatically increased the AMs speed,
and the job finishes in around 30-35 mins. That is still a regression, but
atleast the job finishes after the compressed-ops setting and/or changing the
jvm back to 32 bit.
Giving more heap to the 32 bit JVM, around 3GB, helps to finish the job in
around 7-8 mins. But that isn't something we want to do for all jobs. Reverting
back to original speed definitely means that AM is wasting away time in GCs.
Some of the observations Sid made above may hint at the root culprit.
Will file separate tickets to fix the inefficiencies.
> AMScalability test of Sleep job with 100K 1-sec maps regressed into running
> very slowly
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3402
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3402
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.1
>
>
> The world was rosier before October 19-25, [~karams] says.
> The 100K 1 second sleep job used to take around 800mins or 13-14 mins. It now
> runs till 45 mins and still manages to complete only about 45K tasks.
> One/more of the flurry of commits for 0.23.0 deserve(s) the blame.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
