Kuhu Shukla updated YARN-2410:
    Attachment: YARN-2410-v6.patch

Thank you so much [~jlowe] for the detailed feedback. I have made all but 2 
changes and request your further comments on that.

Actually I'm not really sure why SendMapOutputParams exists separate from 
ReduceContext. There should be a one-to-one relationship there. 

I totally agree. The only reason was findbugs which does not allow more than 7 
parameters in a function call( or the constructor that would populate these 
values). If this is not an issue, I can move them into a single class. For now 
I have made SendMapOutputParams an inner class to ReduceContext.

Why was reduceContext added as a TestShuffleHandler instance variable? It's 
specific to the new test.

The reduceContext is a variable holds the value set by the setAttachment() 
method and is used by the getAttachment() answer. If I declare it in the test 
method, it needs be final which cannot be done due to it being used by the 
setter. I am looking for another way. Let me know what you think.

All other items have been done. 

> Nodemanager ShuffleHandler can possible exhaust file descriptors
> ----------------------------------------------------------------
>                 Key: YARN-2410
>                 URL: https://issues.apache.org/jira/browse/YARN-2410
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.5.0
>            Reporter: Nathan Roberts
>            Assignee: Kuhu Shukla
>         Attachments: YARN-2410-v1.patch, YARN-2410-v2.patch, 
> YARN-2410-v3.patch, YARN-2410-v4.patch, YARN-2410-v5.patch, YARN-2410-v6.patch
> The async nature of the shufflehandler can cause it to open a huge number of
> file descriptors, when it runs out it crashes.
> Scenario:
> Job with 6K reduces, slow start set to 0.95, about 40 map outputs per node.
> Let's say all 6K reduces hit a node at about same time asking for their
> outputs. Each reducer will ask for all 40 map outputs over a single socket in 
> a
> single request (not necessarily all 40 at once, but with coalescing it is
> likely to be a large number).
> sendMapOutput() will open the file for random reading and then perform an 
> async transfer of the particular portion of this file(). This will 
> theoretically
> happen 6000*40=240000 times which will run the NM out of file descriptors and 
> cause it to crash.
> The algorithm should be refactored a little to not open the fds until they're
> actually needed. 

This message was sent by Atlassian JIRA

Reply via email to