[
https://issues.apache.org/jira/browse/YARN-802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13685110#comment-13685110
]
Siddharth Seth commented on YARN-802:
-------------------------------------
bq. If I understand correctly, in order to simultaneously support multiple jobs
of multiple users that each can contact different Shuffle provider we must have
all providers in the air in parallel.
Multiple providers can be run by the NodeManager in parallel. An application
chooses which provider(s) it wants to use when it starts a container on a
NodeManager.
bq. This data for this map arrive from the APPLICATION_INIT event. Hence, all
AuxServices that serve as ShuffleProviders need to get APPLICATION_INIT events.
The data in the APPLICAITON_INIT event is from the startContainer request (the
serviceData in the ContainerLaunchConetxt). If the application wants the INIT
event to go to multiple providers, it can set the service data accordingly. The
MapReduce AM hardcodes this to the default SHUFFLE_PROVIDER which is why only
that one gets the init event.
There may be auxillary services which are not responsible for shuffle, or are
in general incompatible with the shuffle consumer configured by the job. I
don't think they need to get an INIT event.
> APPLICATION_INIT is never sent to AuxServices other than the builtin
> ShuffleHandler
> -----------------------------------------------------------------------------------
>
> Key: YARN-802
> URL: https://issues.apache.org/jira/browse/YARN-802
> Project: Hadoop YARN
> Issue Type: Bug
> Components: applications, nodemanager
> Affects Versions: 2.0.4-alpha
> Reporter: Avner BenHanoch
>
> APPLICATION_INIT is never sent to AuxServices other than the built-in
> ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be
> able to function, because APPLICATION_INIT enables the AuxiliaryService to
> map jobId->userId. This is needed for properly finding the MOFs of a job per
> reducers' requests.
> NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to
> hard-coded expression in hadoop code. The current TaskAttemptImpl.java code
> explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID,
> ...) and ignores any additional AuxiliaryService. As a result, only the
> built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party
> AuxillaryService will never get APPLICATION_INIT events.
> I think a solution can be in one of two ways:
> 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register
> each of them, by calling serviceData.put (…) in loop.
> 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668
> "APPLICATION_STOP is never sent to AuxServices". This means that in case the
> 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux
> Services regardless of the value in event.getServiceID().
> I prefer the 2nd solution. I am welcoming any ideas. I can provide the
> needed patch for any option that people like.
> See [Pluggable Shuffle in Hadoop
> documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
