[
https://issues.apache.org/jira/browse/OOZIE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-1770:
---------------------------------
Attachment: Prelim OYA Scoping Doc 001.pdf
At around the same time as [~jaydeepvishwakarma] posted his document for the
third option, I also wrote a prelim scoping doc for the second option. I'm
attaching it now for reference, but I had sent it to him and a few others for
discussion.
[~jaydeepvishwakarma], [~sriksun], and I had a call where we discussed how to
move forward with OYA. Here's some nodes:
- We'll start with the second option as phase 1 and do the third option as
phase 2. The action code changes should be mostly the same between them, with
the different being more in the AM part. This gives us a good usable and
deliverable milestone in the middle of the project.
- We'll deprecate the MR launching code and remove it at a future point. This
way users don't have to make a major switch right away, and should mitigate any
problems that may arise. Though I think if this gets too complicated, we
should drop it.
- We should look into including some options to have the new launcher pull in
all of the Hadoop jars, as a compatibility option for users whose jobs depend
on having them in the classpath
- My document talks about having an AM pool and using them to create containers
to run the launchers. Instead, we talked about simplifying this to have an AM
per action. Essentially, it would be like uber mode, but without the MR stuff.
However, I was discussing this with [~kasha] afterwards, and I think we might
be better off not doing this.
-- We'd still have to start a JVM for each Action (and then use
{{ProcessBuilder}} to get a shell), which adds overhead, and is unnecessary in
a lot of cases
-- The AM pool idea lets us simply get a container (which is already a shell)
and run the action there
-- Using the pool should be easier to switch to phase 2 because the Workflow AM
would launch actions in containers in the same way as the AM pool. With the AM
per Action, we'd have to do more refactoring between phases here. Essentially,
the AM pool is like a proxy to what will eventually be phase 2.
- We should use a feature branch.
[~jaydeepvishwakarma] and [~sriksun], what do you think about the AM pool vs
Action AM given the points above?
> Create Oozie Application Master for YARN
> ----------------------------------------
>
> Key: OOZIE-1770
> URL: https://issues.apache.org/jira/browse/OOZIE-1770
> Project: Oozie
> Issue Type: New Feature
> Reporter: Bowen Zhang
> Assignee: Bowen Zhang
> Attachments: OozieYarnAM.pdf, Prelim OYA Scoping Doc 001.pdf,
> oya-rm-screenshot.jpg, oya.patch
>
>
> After the first release of oozie on hadoop 2, it will be good if users can
> set execution engine in oozie conf, be it YARN AM or traditional MR. We can
> target this for post oozie 4.1 release.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
