+1, >From user's point of view, the high-level resource management API without any additional dependency is much comfortable than using low-level APIs. I believe the way should increase YARN applications including long-running services. From developer's point of view, the integration make YARN more stable and feature-rich since we can get feedback about YARN low-level APIs including backward compatibility, features we need, and so on.
BTW, I have a question about language-bindings: can we prepare python, C++, or another language APIs over Slider? Is it out of focus of this discussion? Thanks, - Tsuyoshi On Fri, Jul 15, 2016 at 11:36 AM, Vinod Kumar Vavilapalli <vino...@apache.org> wrote: > Hi, Hadoop YARN community! > > (Cross-posting across Hadoop and Slider communities) > > I opened a JIRA a little while ago to pursue a native YARN framework for > services: https://issues.apache.org/jira/browse/YARN-5079. > > It is part of a bigger effort that a bunch of us YARN community members are > interested in making progress on: YARN-4692 - [Umbrella] Simplified and > first-class support for services in YARN. > > The idea is that with our current attention on making services first-class, > it's time to take a fresh look at how we can make Apache Hadoop YARN support > services well out of the box. I’ve been looking at various possibilities - > ranging from a custom new framework room scratch to using one of the existing > projects - and stopped at Apache Slider (http://slider.incubator.apache.org) > given its association with some of the YARN community members (Steve > Loughran, Devaraj Das, Arun C Murthy, myself etc.). > > Slider client & AM already handles a great deal of the functionality that we > need. I posit that assimilating the client, ApplicationMaster etc of an > existing framework like Apache Slider can serve our purpose really well. My > early informal discussions about this with few Hadoop and Slider community > members yielded generally favourable feedback. > > The Apache Slider incubator community also discussed this and expressed > generally positive interest in YARN taking up Slider’s key pieces, you can > see that discussion here: https://s.apache.org/0hoh. > > So in summary, we are looking to the following > > - Code > — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into Apache > Hadoop for providing a native services experiences in YARN > — Leave for now some of the pieces behind in Apache Slider - (a) Slider > agent as we won’t need it, (b) Slider packages that need more deliberation in > terms of where they will live in the long term. > — Create a branch in YARN, copy this code over into a new module(s), and > work towards completing a functioning app running on YARN. > > - Communities & releases > — Good thing is that many of Apache Slider community members are already > seasoned folk in the Apache Hadoop ecosystem projects. For those committers & > PMC in Slider that are not yet Hadoop committers / PMC, without complicating > things much, a proposed path forward is active participation in the branch > (as branch committers?) and eventually in mainline YARN and thus go through a > natural progression to committership / PMC. Given that most of the members > are stalwarts in the Apache communities, this should be a cinch IMO. > — The work on this new code can start, and depending on its state, and > assuming that the experiment succeeds, can be merged into trunk and later > picked up in the next nearest & feasible Apache Hadoop release. > — While the work on forked-over-code goes on till we have a functioning > app, the current Apache Slider project continues to live, with supporting > releases etc. At some point in the future, when things become clearer, more > decisions can be taken on the (parts left behind in the) incubating project’s > future. > > Thoughts? > > Thanks > +Vinod > --------------------------------------------------------------------- > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
