[
https://issues.apache.org/jira/browse/MAPREDUCE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720131#comment-13720131
]
Jason Lowe commented on MAPREDUCE-4421:
---------------------------------------
bq. Also, what would it take to make this work easily for non-MR frameworks?
Other frameworks can do a similar trick, and note that I didn't have to make
any YARN changes for it to work. Well, there is the aux service issue as I
mentioned, but otherwise it can be done in a similar fashion. All it's
basically doing from a YARN standpoint is having the client automatically
bundle an archive as a LocalResource and doctoring the container environment
accordingly. I thought I heard Tez was being deployed this way, but I haven't
verified that.
At the last Hadoop Summit, [~tucu00] had what I thought was a brilliant idea.
Not only the idea of grabbing the framework support code for containers via
HDFS, but having the *client* code come from an HDFS blob as well. There would
be some yarn command to launch an application for a particular version of a
framework, and that command would look in a configured place where frameworks
are stored, pick out the appropriate version of the named framework, download
the client code, and invoke the client to complete the rest of the app
submission. The client could then bundle the rest of the framework in a
similar fashion to how it's being done for MapReduce here.
In essence, it would be a one-step deploy for app frameworks on YARN. Drop a
blob in HDFS, and suddenly users can start using that framework even though
they don't have any of the framework code installed at the time. There's still
some big issues to work out, e.g.: how to download the client code efficiently
(it becomes much like a localization issue with managing a cache of clients
already downloaded, etc.), and I'm sure there's plenty of other devils in the
details. But if accomplished, this would allow one-step deploys for
application frameworks in YARN which I think would be a great feature.
> Remove dependency on deployed MR jars
> -------------------------------------
>
> Key: MAPREDUCE-4421
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-4421.patch, MAPREDUCE-4421.patch
>
>
> Currently MR AM depends on MR jars being deployed on all nodes via implicit
> dependency on YARN_APPLICATION_CLASSPATH.
> We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and,
> probably, just rely on adding a shaded MR jar along with job.jar to the
> dist-cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
