[
https://issues.apache.org/jira/browse/MAPREDUCE-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15162887#comment-15162887
]
Srikanth Sampath commented on MAPREDUCE-6608:
---------------------------------------------
Thanks much [~djp] for your review and comments. Appreciate it very much.
*Issue 1*
{quote}+1 on Vinod's proposal of separating write and read path.{quote}
I agree and will log a separate YARN JIRA. Do you think that effort should be
linked to this work or can be done separately and later incorporated. Given
your suggestion for optimizing - using the service record for other attempts
(not the first one) the read paths will be much fewer.
*Issue 2*
{quote} We can involve a new MR config to switch on/off this feature (off by
default). However, I didn't see any implementation on this in demo patch {quote}
Yes, not in the demo patch. I will add it in the next version and also
maintain the old code path when the configuration is off (the default).
{quote} Beside we need to replace the read path of registry service, another
point is we don't necessary to keep the first attempt AM info which could
saving most of overhead we are adding here as most applications are expected to
end with single attempt. Isn't it? {quote}
Yes. That's correct. Very good suggestion.
{quote}Agree that named argument sounds better. However, this way has work for
a long time for MapReduce project and we won't prefer to change unless we find
some issue or bug. For path to service record, we need keep consistent with our
decision on read path. {quote}
I think named arguments are better. If we end up changing the interface of
YarnChild, I think we should do it. It depends on what we decide on *Issue 1*
{quote}UmbilicalWithRetries should follow other existing practice (for RPC
client retry during service down time) that to create a RetryProxy with
FailoverProxyProvider (may be call it as MRAMProxy) for task attempt to contact
with new attempt instance for AM.{quote}
Thanks much for this very useful suggestion. I will incorporate it.
Please let me know your recommendation on *Issue 1*
> Work Preserving AM Restart for MapReduce
> ----------------------------------------
>
> Key: MAPREDUCE-6608
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6608
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Srikanth Sampath
> Assignee: Srikanth Sampath
> Attachments: Patch1.patch, WorkPreservingMRAppMaster-1.pdf,
> WorkPreservingMRAppMaster-2.pdf, WorkPreservingMRAppMaster.pdf
>
>
> Providing a framework for work preserving AM is achieved in
> [YARN-1489|https://issues.apache.org/jira/browse/YARN-1489]. We would like
> to take advantage of this for MapReduce(MR) applications. There are some
> challenges which have been described in the attached document and few options
> discussed. We solicit feedback from the community.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
