Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

Thanks for the discussion and the feedback, everyone! I sense a general consensus, so am going to go ahead with the branch proposal. Please regard this as the conclusion of the discussion here. If you have more inputs, feel free to jump on to the JIRA YARN-5079! Thanks again, +Vinod > On Jul

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

An interesting proposal, we can continue this discussion on the Slider dev lists. Thanks +Vinod > On Jul 19, 2016, at 4:51 PM, jonma...@gmail.com wrote: > > +1 - this approach definitely addresses an important need. > > The project has gone through at least a couple of development/repo >

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

If you see comments from Gour (of Apache Slider), that seems like the general direction: "All that would be expected is to migrate the app-packages to the agent-less version. Of course the Slider community will do the migration for the current app-packages in the Slider codebase.” Whether this

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

Yes, that’s the general goal for YARN-4692 also! Thanks +Vinod > On Jul 18, 2016, at 8:14 PM, Arun Suresh wrote: > > I would also like to see this effort encompass not just services but also > general long-lived applications

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

This sounds like a good plan for ongoing existence of the code in both the places, Gour! Thanks +Vinod > On Jul 18, 2016, at 12:23 PM, Gour Saha > wrote: > > Note: On the Slider side, we plan to create a branch corresponding to this >

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1 on the proposal as: 1) We definitely need first class support for long running services on YARN. 2) We need to surface the support we are building in an easy to consume format to app developers as YARN APIs IMHO is too low-level for direct end-developer consumption. 3) Slider is the obvious

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1 - this approach definitely addresses an important need. The project has gone through at least a couple of development/repo approaches: 1) git-flow with master/develop branches 2) abandonment of the master branch and work/releases based on the develop branch. I have no strong objections to

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1 as a step towards better service support in YARN. In terms of compatibility and Slider fixes - is the intent to allow existing slider packages to work without migrating over to the new agent-less, etc implementation in Hadoop. e.g. Assuming the slider integration work is released in Hadoop

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1. Merging the core parts of Slider into YARN should help with faster development on service-oriented features. On Tue, Jul 19, 2016 at 12:38 PM, Wangda Tan wrote: > +1, this can avoid lots efforts for end user to run services on YARN. > > On Tue, Jul 19, 2016 at 7:12

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1, this can avoid lots efforts for end user to run services on YARN. On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi wrote: > +1. I think making Slider the basis of a services framework in YARN will > allow us to significantly improve YARN's support for services. > >

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1. Looking forward to this new feature,this will open a door for Yarn to support more use cases. zhihai On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi wrote: > +1. I think making Slider the basis of a services framework in YARN will > allow us to significantly

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1. I think making Slider the basis of a services framework in YARN will allow us to significantly improve YARN's support for services. On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli wrote: > Hi, Hadoop YARN community! > > (Cross-posting across Hadoop and Slider

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1. This should go a long way to help running services on YARN. -Varun On 7/19/16, 12:53 AM, "Gour Saha" wrote: >+1 forwarding from the discussion in Slider DL > >Note: On the Slider side, we plan to create a branch corresponding to this >YARN branch. In this, we intend

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

The API layer effort as in https://issues.apache.org/jira/browse/YARN-4793 is REST, defined in swagger format (the first draft version is uploaded to YARN-4793). Proposal is to source control this as the entry point for the APIs. The server side stubs (Java) will be generated from this definition,

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1, >From user's point of view, the high-level resource management API without any additional dependency is much comfortable than using low-level APIs. I believe the way should increase YARN applications including long-running services. From developer's point of view, the integration make YARN

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1 It is definitely a step in the right direction and it's about time YARN recognizes services as first-class citizens. I would also like to see this effort encompass not just services but also general long-lived applications such as say storm/heron topologies, samza and possibly support for

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1. It would be great if it becomes the default way for services to run on YARN. That would also make it easier to make YARN features support services better. As you mentioned, there would be a lot of logistics to work out, and we'll need to talk about them in more detail as they come up.

Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

+1 forwarding from the discussion in Slider DL Note: On the Slider side, we plan to create a branch corresponding to this YARN branch. In this, we intend to remove all the pieces which will move to the YARN branch (Slider core, AM, client). We will retain the Agent and app-packages which will

[DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider

Hi, Hadoop YARN community! (Cross-posting across Hadoop and Slider communities) I opened a JIRA a little while ago to pursue a native YARN framework for services: https://issues.apache.org/jira/browse/YARN-5079. It is part of a bigger effort that a bunch of us YARN community members are