[
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182450#comment-16182450
]
Kuhu Shukla commented on YARN-7244:
-----------------------------------
Thank you [~sunilg] for the review comments!
bq. We could push this config name to LocalDirAllocator and then read from NM
end
I am not sure how we can initialize the config specifically for
LocalDirAllocator (may be add a constructor?) and what would reading it from NM
end mean. Agreed that code separation is important here. May be not having this
as a config might help?
bq. Do you think, we can improve this to skip as default behavior itself?
I did not fully get what you have in mind here. Could you help me understand
and elaborate a bit. Thanks a lot!
> ShuffleHandler is not aware of disks that are added
> ---------------------------------------------------
>
> Key: YARN-7244
> URL: https://issues.apache.org/jira/browse/YARN-7244
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: YARN-7244.001.patch, YARN-7244.002.patch
>
>
> The ShuffleHandler permanently remembers the list of "good" disks on NM
> startup. If disks later are added to the node then map tasks will start using
> them but the ShuffleHandler will not be aware of them. The end result is that
> the data cannot be shuffled from the node leading to fetch failures and
> re-runs of the map tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
