RE: [VOTE] Merge yarn-native-services branch into trunk
Cool to have this feature! Thanks Jian and all. Regards, Kai -Original Message- From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] Sent: Tuesday, November 07, 2017 7:20 AM To: Jian He <j...@hortonworks.com> Cc: yarn-...@hadoop.apache.org; common-...@hadoop.apache.org; Hdfs-dev <hdfs-dev@hadoop.apache.org>; mapreduce-...@hadoop.apache.org Subject: Re: [VOTE] Merge yarn-native-services branch into trunk Congratulations to all the contributors involved, this is a great step forward! +Vinod > On Nov 6, 2017, at 2:40 PM, Jian He <j...@hortonworks.com> wrote: > > Okay, I just merged the branch to trunk (108 commits in total !) > Again, thanks for all who contributed to this feature! > > Jian > > On Nov 6, 2017, at 1:26 PM, Jian He > <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote: > > Here’s +1 from myself. > The vote passes with 7 (+1) bindings and 2 (+1) non-bindings. > > Thanks for all who voted. I’ll merge to trunk by the end of today. > > Jian > > On Nov 6, 2017, at 8:38 AM, Billie Rinaldi > <billie.rina...@gmail.com<mailto:billie.rina...@gmail.com>> wrote: > > +1 (binding) > > On Mon, Oct 30, 2017 at 1:06 PM, Jian He > <j...@hortonworks.com<mailto:j...@hortonworks.com>> wrote: > Hi All, > > I would like to restart the vote for merging yarn-native-services to trunk. > Since last vote, we have been working on several issues in documentation, > DNS, CLI modifications etc. We believe now the feature is in a much better > shape. > > Some back ground: > At a high level, the following are the key feautres implemented. > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate > existing services to YARN either docker or non-docker based. > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user > to deploy a service via a simple JSON spec > - YARN-4757[3]. Extending today's service registry with a simple DNS > service to enable users to discover services deployed on YARN via > standard DNS lookup > - YARN-6419[4]. UI support for native-services on the new YARN UI All > these new services are optional and are sitting outside of the existing > system, and have no impact on existing system if disabled. > > Special thanks to a team of folks who worked hard towards this: Billie > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K > S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without > their ideas and hard work. > Also thanks Allen for some review and verifications. > > Thanks, > Jian > > [1] https://issues.apache.org/jira/browse/YARN-5079 > [2] https://issues.apache.org/jira/browse/YARN-4793 > [3] https://issues.apache.org/jira/browse/YARN-4757 > [4] https://issues.apache.org/jira/browse/YARN-6419 > > > - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Congratulations to all the contributors involved, this is a great step forward! +Vinod > On Nov 6, 2017, at 2:40 PM, Jian Hewrote: > > Okay, I just merged the branch to trunk (108 commits in total !) > Again, thanks for all who contributed to this feature! > > Jian > > On Nov 6, 2017, at 1:26 PM, Jian He > > wrote: > > Here’s +1 from myself. > The vote passes with 7 (+1) bindings and 2 (+1) non-bindings. > > Thanks for all who voted. I’ll merge to trunk by the end of today. > > Jian > > On Nov 6, 2017, at 8:38 AM, Billie Rinaldi > > wrote: > > +1 (binding) > > On Mon, Oct 30, 2017 at 1:06 PM, Jian He > > wrote: > Hi All, > > I would like to restart the vote for merging yarn-native-services to trunk. > Since last vote, we have been working on several issues in documentation, > DNS, CLI modifications etc. We believe now the feature is in a much better > shape. > > Some back ground: > At a high level, the following are the key feautres implemented. > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate > existing services to YARN either docker or non-docker based. > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to > deploy a service via a simple JSON spec > - YARN-4757[3]. Extending today's service registry with a simple DNS service > to enable users to discover services deployed on YARN via standard DNS lookup > - YARN-6419[4]. UI support for native-services on the new YARN UI > All these new services are optional and are sitting outside of the existing > system, and have no impact on existing system if disabled. > > Special thanks to a team of folks who worked hard towards this: Billie > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma K > S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible without > their ideas and hard work. > Also thanks Allen for some review and verifications. > > Thanks, > Jian > > [1] https://issues.apache.org/jira/browse/YARN-5079 > [2] https://issues.apache.org/jira/browse/YARN-4793 > [3] https://issues.apache.org/jira/browse/YARN-4757 > [4] https://issues.apache.org/jira/browse/YARN-6419 > > > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (binding) On Mon, Oct 30, 2017 at 1:06 PM, Jian Hewrote: > Hi All, > > I would like to restart the vote for merging yarn-native-services to trunk. > Since last vote, we have been working on several issues in documentation, > DNS, CLI modifications etc. We believe now the feature is in a much better > shape. > > Some back ground: > At a high level, the following are the key feautres implemented. > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate > existing services to YARN either docker or non-docker based. > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to > deploy a service via a simple JSON spec > - YARN-4757[3]. Extending today's service registry with a simple DNS > service to enable users to discover services deployed on YARN via standard > DNS lookup > - YARN-6419[4]. UI support for native-services on the new YARN UI > All these new services are optional and are sitting outside of the > existing system, and have no impact on existing system if disabled. > > Special thanks to a team of folks who worked hard towards this: Billie > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible > without their ideas and hard work. > Also thanks Allen for some review and verifications. > > Thanks, > Jian > > [1] https://issues.apache.org/jira/browse/YARN-5079 > [2] https://issues.apache.org/jira/browse/YARN-4793 > [3] https://issues.apache.org/jira/browse/YARN-4757 > [4] https://issues.apache.org/jira/browse/YARN-6419 >
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (non-binding) -Gour On 10/30/17, 1:49 PM, "Jian He"wrote: >Few more things: > >This is the document for trying a non-docker service on YARN. >https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-pro >ject/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/QuickStar >t.md > >And the document for a docker based service >https://github.com/apache/hadoop/blob/yarn-native-services/hadoop-yarn-pro >ject/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Examples. >md > >And the vote lasts 7 days as usual. > >Thanks, >Jian > >On Oct 30, 2017, at 1:06 PM, Jian He > > wrote: > >Hi All, > >I would like to restart the vote for merging yarn-native-services to >trunk. >Since last vote, we have been working on several issues in documentation, >DNS, CLI modifications etc. We believe now the feature is in a much >better shape. > >Some back ground: >At a high level, the following are the key feautres implemented. >- YARN-5079[1]. A native YARN framework (ApplicationMaster) to >orchestrate existing services to YARN either docker or non-docker based. >- YARN-4793[2]. A Rest API service embeded in RM (optional) for user to >deploy a service via a simple JSON spec >- YARN-4757[3]. Extending today's service registry with a simple DNS >service to enable users to discover services deployed on YARN via >standard DNS lookup >- YARN-6419[4]. UI support for native-services on the new YARN UI >All these new services are optional and are sitting outside of the >existing system, and have no impact on existing system if disabled. > >Special thanks to a team of folks who worked hard towards this: Billie >Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith >Sharma K S, Sunil G, Akhil PB, Eric Yang. This effort could not be >possible without their ideas and hard work. >Also thanks Allen for some review and verifications. > >Thanks, >Jian > >[1] https://issues.apache.org/jira/browse/YARN-5079 >[2] https://issues.apache.org/jira/browse/YARN-4793 >[3] https://issues.apache.org/jira/browse/YARN-4757 >[4] https://issues.apache.org/jira/browse/YARN-6419 > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (binding) thanks Jian for all the great work! Built from branch and deployed, able to bring up services along with atsv2 enabled and new YARN UI integration. Tried flexing, start, stop operations using REST api's. - Rohith Sharma K S On 31 October 2017 at 01:36, Jian Hewrote: > Hi All, > > I would like to restart the vote for merging yarn-native-services to trunk. > Since last vote, we have been working on several issues in documentation, > DNS, CLI modifications etc. We believe now the feature is in a much better > shape. > > Some back ground: > At a high level, the following are the key feautres implemented. > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate > existing services to YARN either docker or non-docker based. > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to > deploy a service via a simple JSON spec > - YARN-4757[3]. Extending today's service registry with a simple DNS > service to enable users to discover services deployed on YARN via standard > DNS lookup > - YARN-6419[4]. UI support for native-services on the new YARN UI > All these new services are optional and are sitting outside of the > existing system, and have no impact on existing system if disabled. > > Special thanks to a team of folks who worked hard towards this: Billie > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible > without their ideas and hard work. > Also thanks Allen for some review and verifications. > > Thanks, > Jian > > [1] https://issues.apache.org/jira/browse/YARN-5079 > [2] https://issues.apache.org/jira/browse/YARN-4793 > [3] https://issues.apache.org/jira/browse/YARN-4757 > [4] https://issues.apache.org/jira/browse/YARN-6419 >
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (binding) Tested the branch code and brought up services like sleep and httpd. Also verified UI as well. - Sunil On Tue, Oct 31, 2017 at 1:36 AM Jian Hewrote: > Hi All, > > I would like to restart the vote for merging yarn-native-services to trunk. > Since last vote, we have been working on several issues in documentation, > DNS, CLI modifications etc. We believe now the feature is in a much better > shape. > > Some back ground: > At a high level, the following are the key feautres implemented. > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate > existing services to YARN either docker or non-docker based. > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to > deploy a service via a simple JSON spec > - YARN-4757[3]. Extending today's service registry with a simple DNS > service to enable users to discover services deployed on YARN via standard > DNS lookup > - YARN-6419[4]. UI support for native-services on the new YARN UI > All these new services are optional and are sitting outside of the > existing system, and have no impact on existing system if disabled. > > Special thanks to a team of folks who worked hard towards this: Billie > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible > without their ideas and hard work. > Also thanks Allen for some review and verifications. > > Thanks, > Jian > > [1] https://issues.apache.org/jira/browse/YARN-5079 > [2] https://issues.apache.org/jira/browse/YARN-4793 > [3] https://issues.apache.org/jira/browse/YARN-4757 > [4] https://issues.apache.org/jira/browse/YARN-6419 >
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (binding) for branch merge. I couldn’t get the patch to work properly though. Yarn-native-services branch works fine. If bugs surface in trunk, we will fix it. Regards, Eric On 11/3/17, 2:45 PM, "Wangda Tan" <wheele...@gmail.com> wrote: Thanks all for the great works! +1 (Binding). Tried to use native services to build and run applications successfully. - Wangda On Fri, Nov 3, 2017 at 11:50 AM, Arun Suresh <asur...@apache.org> wrote: > +1 (binding) > > Cheers > -Arun > > On Nov 3, 2017 11:44 AM, "Chandni Singh" <csi...@hortonworks.com> wrote: > > > +1 > > Thanks, > > Chandni > > > > From: Jian He <j...@hortonworks.com> > > Sent: Monday, October 30, 2017 1:49 PM > > To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common; > > mapreduce-...@hadoop.apache.org > > Subject: Re: [VOTE] Merge yarn-native-services branch into trunk > > > > Few more things: > > > > This is the document for trying a non-docker service on YARN. > > https://github.com/apache/hadoop/blob/yarn-native- > > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/ > > src/site/markdown/yarn-service/QuickStart.md > > > > And the document for a docker based service > > https://github.com/apache/hadoop/blob/yarn-native- > > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/ > > src/site/markdown/yarn-service/Examples.md > > > > And the vote lasts 7 days as usual. > > > > Thanks, > > Jian > > > > On Oct 30, 2017, at 1:06 PM, Jian He <j...@hortonworks.com<mailto:jh > > e...@hortonworks.com>> wrote: > > > > Hi All, > > > > I would like to restart the vote for merging yarn-native-services to > trunk. > > Since last vote, we have been working on several issues in documentation, > > DNS, CLI modifications etc. We believe now the feature is in a much > better > > shape. > > > > Some back ground: > > At a high level, the following are the key feautres implemented. > > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to > orchestrate > > existing services to YARN either docker or non-docker based. > > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to > > deploy a service via a simple JSON spec > > - YARN-4757[3]. Extending today's service registry with a simple DNS > > service to enable users to discover services deployed on YARN via > standard > > DNS lookup > > - YARN-6419[4]. UI support for native-services on the new YARN UI > > All these new services are optional and are sitting outside of the > > existing system, and have no impact on existing system if disabled. > > > > Special thanks to a team of folks who worked hard towards this: Billie > > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith > Sharma > > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible > > without their ideas and hard work. > > Also thanks Allen for some review and verifications. > > > > Thanks, > > Jian > > > > [1] https://issues.apache.org/jira/browse/YARN-5079 > > [2] https://issues.apache.org/jira/browse/YARN-4793 > > [3] https://issues.apache.org/jira/browse/YARN-4757 > > [4] https://issues.apache.org/jira/browse/YARN-6419 > > > > > > > > > > - > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > > > >
Re: [VOTE] Merge yarn-native-services branch into trunk
Thanks all for the great works! +1 (Binding). Tried to use native services to build and run applications successfully. - Wangda On Fri, Nov 3, 2017 at 11:50 AM, Arun Suresh <asur...@apache.org> wrote: > +1 (binding) > > Cheers > -Arun > > On Nov 3, 2017 11:44 AM, "Chandni Singh" <csi...@hortonworks.com> wrote: > > > +1 > > Thanks, > > Chandni > > > > From: Jian He <j...@hortonworks.com> > > Sent: Monday, October 30, 2017 1:49 PM > > To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common; > > mapreduce-...@hadoop.apache.org > > Subject: Re: [VOTE] Merge yarn-native-services branch into trunk > > > > Few more things: > > > > This is the document for trying a non-docker service on YARN. > > https://github.com/apache/hadoop/blob/yarn-native- > > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/ > > src/site/markdown/yarn-service/QuickStart.md > > > > And the document for a docker based service > > https://github.com/apache/hadoop/blob/yarn-native- > > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/ > > src/site/markdown/yarn-service/Examples.md > > > > And the vote lasts 7 days as usual. > > > > Thanks, > > Jian > > > > On Oct 30, 2017, at 1:06 PM, Jian He <j...@hortonworks.com<mailto:jh > > e...@hortonworks.com>> wrote: > > > > Hi All, > > > > I would like to restart the vote for merging yarn-native-services to > trunk. > > Since last vote, we have been working on several issues in documentation, > > DNS, CLI modifications etc. We believe now the feature is in a much > better > > shape. > > > > Some back ground: > > At a high level, the following are the key feautres implemented. > > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to > orchestrate > > existing services to YARN either docker or non-docker based. > > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to > > deploy a service via a simple JSON spec > > - YARN-4757[3]. Extending today's service registry with a simple DNS > > service to enable users to discover services deployed on YARN via > standard > > DNS lookup > > - YARN-6419[4]. UI support for native-services on the new YARN UI > > All these new services are optional and are sitting outside of the > > existing system, and have no impact on existing system if disabled. > > > > Special thanks to a team of folks who worked hard towards this: Billie > > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith > Sharma > > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible > > without their ideas and hard work. > > Also thanks Allen for some review and verifications. > > > > Thanks, > > Jian > > > > [1] https://issues.apache.org/jira/browse/YARN-5079 > > [2] https://issues.apache.org/jira/browse/YARN-4793 > > [3] https://issues.apache.org/jira/browse/YARN-4757 > > [4] https://issues.apache.org/jira/browse/YARN-6419 > > > > > > > > > > - > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > > > >
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (binding) Cheers -Arun On Nov 3, 2017 11:44 AM, "Chandni Singh" <csi...@hortonworks.com> wrote: > +1 > Thanks, > Chandni > > From: Jian He <j...@hortonworks.com> > Sent: Monday, October 30, 2017 1:49 PM > To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common; > mapreduce-...@hadoop.apache.org > Subject: Re: [VOTE] Merge yarn-native-services branch into trunk > > Few more things: > > This is the document for trying a non-docker service on YARN. > https://github.com/apache/hadoop/blob/yarn-native- > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/ > src/site/markdown/yarn-service/QuickStart.md > > And the document for a docker based service > https://github.com/apache/hadoop/blob/yarn-native- > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/ > src/site/markdown/yarn-service/Examples.md > > And the vote lasts 7 days as usual. > > Thanks, > Jian > > On Oct 30, 2017, at 1:06 PM, Jian He <j...@hortonworks.com<mailto:jh > e...@hortonworks.com>> wrote: > > Hi All, > > I would like to restart the vote for merging yarn-native-services to trunk. > Since last vote, we have been working on several issues in documentation, > DNS, CLI modifications etc. We believe now the feature is in a much better > shape. > > Some back ground: > At a high level, the following are the key feautres implemented. > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate > existing services to YARN either docker or non-docker based. > - YARN-4793[2]. A Rest API service embeded in RM (optional) for user to > deploy a service via a simple JSON spec > - YARN-4757[3]. Extending today's service registry with a simple DNS > service to enable users to discover services deployed on YARN via standard > DNS lookup > - YARN-6419[4]. UI support for native-services on the new YARN UI > All these new services are optional and are sitting outside of the > existing system, and have no impact on existing system if disabled. > > Special thanks to a team of folks who worked hard towards this: Billie > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible > without their ideas and hard work. > Also thanks Allen for some review and verifications. > > Thanks, > Jian > > [1] https://issues.apache.org/jira/browse/YARN-5079 > [2] https://issues.apache.org/jira/browse/YARN-4793 > [3] https://issues.apache.org/jira/browse/YARN-4757 > [4] https://issues.apache.org/jira/browse/YARN-6419 > > > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
Re: [VOTE] Merge yarn-native-services branch into trunk
> On Sep 8, 2017, at 9:25 AM, Jian Hewrote: > > Hi Allen, > The documentations are committed. Please check QuickStart.md and others in > the same folder. > YarnCommands.md doc is updated to include new commands. > DNS default port is also documented. > Would you like to give a look and see if it address your concerns ? Somewhat. Greatly improved, but there’s still way too much “we’re working on this” and “here’s a link to a JIRA” and just general brokenness going on. Here’s some examples from concepts. Concepts! The document I’d expect to give me very basic “when we talk about X, we mean Y” definitions: "A host of scheduling features are being developed to support long running services.” Yeah, ok? How is this a concept? or "[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) implements a retry-policy to let NM re-launch a service container when it fails.” The patch itself went through nine revisions and a long discussion. Would an end user care about the details in that JIRA? If the answer to the last question is YES, then the documentation has failed. The whole point of documentation is so they don’t have to go digging into the details of the implementation, the decision process that got us there, etc. If they care enough about the details, they’ll run through the changelog and click on the JIRA link there. If the summary line of the changelog isn’t obvious, well… then we need better summaries. etc, etc. ... The sleep example is nice. Now, let’s see a non-toy example: multiple instances of Apache httpd or MariaDB or something real and not from the Hadoop echo chamber (e.g., non-JVM-based). If this is for “native” services, this shouldn’t be a problem, right? Give a real example and users will buy what you’re selling. I also think writing the docs and providing an example of doing something big and outside the team’s comfort zone will clarify where end users are going to need more help than what’s being provided. Getting a MariaDB instance or three up will help tremendously here. Which reminds me: something the documentation doesn’t cover is storage. What happens to it, where does it come from, etc, etc. That’s an important detail that I didn’t see covered. (I may have missed it.) … Why are there directions to enable other, partially unrelated services in here? Shouldn’t there be pointers to their specific documentation? Is the expectation that if the requirements for those other services change that contributors will need to update multiple documents? "Start the DNS server” Just… yikes. a) yarn classname … This is not how we do user-facing things. The fact it’s not really possible for a *daemon* to be put in the YarnCommands.md doc should be a giant red flag that something isn’t going correctly here. b) no jsvc support for something that it’s strongly hinted at wanting to run privileged = an instant -1 for failing basic security practices. There’s zero reason for it to be running continually as root. c) If this would have been hooked into the shell scripts appropriately, logs, user switching, etc would have been had for free. d) Where’s stop? Right. Since it’s outside the scripts, there is no pid support so one has to do all of that manually…. Given: "3. Supports reverse lookups (name based on IP). Note, this works only for Docker containers.” then: "It should not be used as a fully-functional corporate DNS.” Scratch corporate. It’s not a fully functional DNS server if it can’t do reverse lookups. (Which, ironically, means it’s not suitable for use with Apache Hadoop, given it requires both fwd and rev DNS ...) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Thanks for your consideration Jian, let's track this for GA then. Best, Andrew On Fri, Sep 8, 2017 at 3:02 PM, Jian Hewrote: > Hi Andrew, > > At this point, there are no more release blockers including documentations > from our side - all work done. > But I agree it is too close to the release, after talking with other team > members, we are fine to drop this from beta, > > And we want to target this for GA. > I’m withdrawing this vote and will start afresh vote later for GA. > Thanks all who voted this effort ! > > Thanks, > Jian > > > > On Sep 7, 2017, at 3:59 PM, Andrew Wang > wrote: > > > > Hi folks, > > > > This vote closes today. I see a -1 from Allen on inclusion in beta1. I > see > > there's active fixing going on, but given that we're one week out from > RC0, > > I think we should drop this from beta1. > > > > Allen, Jian, others, is this reasonable? What release should we retarget > > this for? I don't have a sense for how much work there is left to do, but > > as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January. > > > > Best, > > Andrew > > > > On Wed, Sep 6, 2017 at 10:19 AM, Jian He wrote: > > > >>> Please correct me if I’m wrong, but the current summary of the > >> branch, post these changes, looks like: > >> Sorry for confusion, I was actively writing the formal documentation for > >> how to use/how it works etc. and will post soon in a few hours. > >> > >> > >>> On Sep 6, 2017, at 10:15 AM, Allen Wittenauer < > a...@effectivemachines.com> > >> wrote: > >>> > >>> > On Sep 5, 2017, at 6:23 PM, Jian He wrote: > > >If it doesn’t have all the bells and whistles, then it shouldn’t > >> be on port 53 by default. > Sure, I’ll change the default port to not use 53 and document it. > >*how* is it getting launched on a privileged port? It sounds like > >> the expectation is to run “command” as root. *ALL* of the previous > >> daemons in Hadoop that needed a privileged port used jsvc. Why isn’t > this > >> one? These questions matter from a security standpoint. > Yes, it is running as “root” to be able to use the privileged port. > The > >> DNS server is not yet integrated with the hadoop script. > > > Check the output. It’s pretty obviously borked: > Thanks for pointing out. Missed this when rebasing onto trunk. > >>> > >>> > >>> Please correct me if I’m wrong, but the current summary of the > >> branch, post these changes, looks like: > >>> > >>> * A bunch of mostly new Java code that may or may not have > >> javadocs (post-revert YARN-6877, still working out HADOOP-14835) > >>> * ~1/3 of the docs are roadmap/TBD > >>> * ~1/3 of the docs are for an optional DNS daemon that has > >> no end user hook to start it > >>> * ~1/3 of the docs are for a REST API that comes from some > >> undefined daemon (apiserver?) > >>> * Two new, but undocumented, subcommands to yarn > >>> * There are no docs for admins or users on how to actually > >> start or use this completely new/separate/optional feature > >>> > >>> How are outside people (e.g., non-branch committers) supposed to > >> test this new feature under these conditions? > >>> > >> > >> > >> - > >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >> > >> > >
Re: [VOTE] Merge yarn-native-services branch into trunk
Hi folks, This vote closes today. I see a -1 from Allen on inclusion in beta1. I see there's active fixing going on, but given that we're one week out from RC0, I think we should drop this from beta1. Allen, Jian, others, is this reasonable? What release should we retarget this for? I don't have a sense for how much work there is left to do, but as a reminder, we're planning GA for Nov 1st, and 3.1.0 for January. Best, Andrew On Wed, Sep 6, 2017 at 10:19 AM, Jian Hewrote: > > Please correct me if I’m wrong, but the current summary of the > branch, post these changes, looks like: > Sorry for confusion, I was actively writing the formal documentation for > how to use/how it works etc. and will post soon in a few hours. > > > > On Sep 6, 2017, at 10:15 AM, Allen Wittenauer > wrote: > > > > > >> On Sep 5, 2017, at 6:23 PM, Jian He wrote: > >> > >>> If it doesn’t have all the bells and whistles, then it shouldn’t > be on port 53 by default. > >> Sure, I’ll change the default port to not use 53 and document it. > >>> *how* is it getting launched on a privileged port? It sounds like > the expectation is to run “command” as root. *ALL* of the previous > daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this > one? These questions matter from a security standpoint. > >> Yes, it is running as “root” to be able to use the privileged port. The > DNS server is not yet integrated with the hadoop script. > >> > >>> Check the output. It’s pretty obviously borked: > >> Thanks for pointing out. Missed this when rebasing onto trunk. > > > > > > Please correct me if I’m wrong, but the current summary of the > branch, post these changes, looks like: > > > > * A bunch of mostly new Java code that may or may not have > javadocs (post-revert YARN-6877, still working out HADOOP-14835) > > * ~1/3 of the docs are roadmap/TBD > > * ~1/3 of the docs are for an optional DNS daemon that has > no end user hook to start it > > * ~1/3 of the docs are for a REST API that comes from some > undefined daemon (apiserver?) > > * Two new, but undocumented, subcommands to yarn > > * There are no docs for admins or users on how to actually > start or use this completely new/separate/optional feature > > > > How are outside people (e.g., non-branch committers) supposed to > test this new feature under these conditions? > > > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
Re: [VOTE] Merge yarn-native-services branch into trunk
> On Sep 5, 2017, at 6:23 PM, Jian Hewrote: > >> If it doesn’t have all the bells and whistles, then it shouldn’t be on >> port 53 by default. > Sure, I’ll change the default port to not use 53 and document it. >> *how* is it getting launched on a privileged port? It sounds like the >> expectation is to run “command” as root. *ALL* of the previous daemons in >> Hadoop that needed a privileged port used jsvc. Why isn’t this one? These >> questions matter from a security standpoint. > Yes, it is running as “root” to be able to use the privileged port. The DNS > server is not yet integrated with the hadoop script. > >> Check the output. It’s pretty obviously borked: > Thanks for pointing out. Missed this when rebasing onto trunk. Please correct me if I’m wrong, but the current summary of the branch, post these changes, looks like: * A bunch of mostly new Java code that may or may not have javadocs (post-revert YARN-6877, still working out HADOOP-14835) * ~1/3 of the docs are roadmap/TBD * ~1/3 of the docs are for an optional DNS daemon that has no end user hook to start it * ~1/3 of the docs are for a REST API that comes from some undefined daemon (apiserver?) * Two new, but undocumented, subcommands to yarn * There are no docs for admins or users on how to actually start or use this completely new/separate/optional feature How are outside people (e.g., non-branch committers) supposed to test this new feature under these conditions? - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
Thanks Allen. You are right, the github renderer does have trouble rendering the headers. I was only looking at the html generated by mvn site, which did not have trouble rendering them. Anyway I added a space after all the hashes and it looks ok through github now. -Gour On 9/5/17, 3:20 PM, "Allen Wittenauer"wrote: > >> On Sep 5, 2017, at 3:12 PM, Gour Saha wrote: >> >> 2) Lots of markdown problems in the NativeServicesDiscovery.md document. >> This includes things like Œyarnsite.xml¹ (missing a dash.) >> >> The md patch uploaded to YARN-5244 had some special chars. I fixed those >> in YARN-7161. > > > It’s a lot more than just special chars I think. Even github (which has >a way better markdown processor than what we’re using for the site docs) >is having trouble rendering it: > >https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c64 >06ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/na >tive-services/NativeServicesDiscovery.md > >e.g., all of those ‘###’ are likely missing a space. >
Re: [VOTE] Merge yarn-native-services branch into trunk
> On Sep 5, 2017, at 3:12 PM, Gour Sahawrote: > > 2) Lots of markdown problems in the NativeServicesDiscovery.md document. > This includes things like Œyarnsite.xml¹ (missing a dash.) > > The md patch uploaded to YARN-5244 had some special chars. I fixed those > in YARN-7161. It’s a lot more than just special chars I think. Even github (which has a way better markdown processor than what we’re using for the site docs) is having trouble rendering it: https://github.com/apache/hadoop/blob/51c39c4261236ab714fe0ec8d00753dc4c6406ee/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesDiscovery.md e.g., all of those ‘###’ are likely missing a space. - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like Œyarnsite.xml¹ (missing a dash.) The md patch uploaded to YARN-5244 had some special chars. I fixed those in YARN-7161. > - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
> On Sep 5, 2017, at 2:53 PM, Jian Hewrote: > >> Based on the documentation, this doesn’t appear to be a fully function DNS >> server as an admin would expect (e.g., BIND, Knot, whatever). Where’s >> forwarding? How do I setup notify? Are secondaries even supported? etc, etc. > > It seems like this is a rehash of some of the discussion you and others had > on the JIRA. The DNS here is a thin layer backed by service registry. My > understanding from the JIRA is that there are no claims that this is already > a DNS with all the bells and whistles - its goal is mainly to expose dynamic > services running on YARN as end-points. Clearly, this is an optional daemon, > if the provided feature set is deemed insufficient, an alternative solution > can be plugged in by specific admins because the DNS piece is completely > decoupled from the rest of native-services. If it doesn’t have all the bells and whistles, then it shouldn’t be on port 53 by default. It should also be documented that one *can’t* do these things. If the standard config is likely to be a “real” server on port 53 either acting as a secondary to the YARN one or at least able to forward queries to it, then these need to get documented. As it stands, operations folks are going to be taken completely by surprise by some relatively random process sitting on a very well established port. >> In fact: was this even tested on port 53? How does this get launched such >> that it even has access to open port 53? I don’t see any calls to use the >> secure daemon code in the shell scripts. Is there any jsvc voodoo or is it >> just “run X as root”? > > Yes, we have tested this DNS server on port 53 on a cluster by running the > DNS server as root user. The port is clearly configurable, so the admin has > two options. Run as root + port 53. Run as non-root + non-privileged port. We > tested and left it as port 53 to keep it on a standard DNS port. It is > already documented as such though I can see that part can be improved a > little. *how* is it getting launched on a privileged port? It sounds like the expectation is to run “command” as root. *ALL* of the previous daemons in Hadoop that needed a privileged port used jsvc. Why isn’t this one? These questions matter from a security standpoint. >> 4) Post-merge, yarn usage information is broken. This is especially >> bad since it doesn’t appear that YarnCommands was ever updated to include >> the new sub-commands. > > The “yarn” usage command is working for me. what do you mean ? Check the output. It’s pretty obviously borked: ===snip Daemon Commands: nodemanager run a nodemanager on each worker proxyserver run the web app proxy server resourcemanager run the ResourceManager router run the Router daemon timelineserver run the timeline server Run a service Commands: service run a service Run yarn-native-service rest server Commands: apiserverrun yarn-native-service rest server ===snip=== > Yeah, looks like some previous features also forgot to update YarnCommands.md > for the new sub commands Likely. But I was actually interested in playing with this one to compare it to the competition. [Lucky you. ;) ] But with pretty much zero documentation…. - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
> On Aug 31, 2017, at 8:33 PM, Jian Hewrote: > I would like to call a vote for merging yarn-native-services to trunk. 1) Did I miss it or is there no actual end-user documentation on how to use this? I see hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/native-services/NativeServicesIntro.md, but that’s not particularly useful. It looks like there are daemons that need to get started, based on other documentation? How? What do I configure? Is there a command to use to say “go do native for this job”? I honestly have no idea how to make this do anything because most of the docs appear to be either TBD or expect me to read through a ton of JIRAs. 2) Lots of markdown problems in the NativeServicesDiscovery.md document. This includes things like ‘yarnsite.xml’ (missing a dash.) Also, I’m also confused why it’s called that when the title is YARN DNS, but whatever. 3) The default port for the DNS server should NOT be 53 if typical deployments need to specify an alternate port. Based on the documentation, this doesn’t appear to be a fully function DNS server as an admin would expect (e.g., BIND, Knot, whatever). Where’s forwarding? How do I setup notify? Are secondaries even supported? etc, etc. In fact: was this even tested on port 53? How does this get launched such that it even has access to open port 53? I don’t see any calls to use the secure daemon code in the shell scripts. Is there any jsvc voodoo or is it just “run X as root”? 4) Post-merge, yarn usage information is broken. This is especially bad since it doesn’t appear that YarnCommands was ever updated to include the new sub-commands. At this point in time: -1 on 3.0.0-beta1 -0 on trunk - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (binding). Cheers -Arun On Fri, Sep 1, 2017 at 5:24 PM, Wangda Tanwrote: > +1 (Binding), I tried to use YARN service assembly before to run different > kinds of jobs (for example, distributed Tensorflow), it is really easy for > end user to run jobs on YARN. > > Thanks to the whole team for the great job! > > Best, > Wangda > > > On Fri, Sep 1, 2017 at 3:33 PM, Gour Saha wrote: > > > +1 (non-binding) > > > > On 9/1/17, 11:58 AM, "Billie Rinaldi" wrote: > > > > >+1 (non-binding) > > > > > >On Thu, Aug 31, 2017 at 8:33 PM, Jian He wrote: > > > > > >> Hi All, > > >> > > >> I would like to call a vote for merging yarn-native-services to trunk. > > >>The > > >> vote will run for 7 days as usual. > > >> > > >> At a high level, the following are the key feautres implemented. > > >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate > > >>and > > >> orchestrate existing services to YARN either docker or non-docker > based. > > >> - YARN-4793[2]. A Rest API server for user to deploy a service via a > > >> simple JSON spec > > >> - YARN-4757[3]. Extending today's service registry with a simple DNS > > >> service to enable users to discover services deployed on YARN > > >> - YARN-6419[4]. UI support for native-services on the new YARN UI > > >> All these new services are optional and are sitting outside of the > > >> existing system, and have no impact on existing system if disabled. > > >> > > >> Special thanks to a team of folks who worked hard towards this: Billie > > >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith > > >>Sharma > > >> K S, Sunil G, Akhil PB. This effort could not be possible without > their > > >> ideas and hard work. > > >> > > >> Thanks, > > >> Jian > > >> > > >> [1] https://issues.apache.org/jira/browse/YARN-5079 > > >> [2] https://issues.apache.org/jira/browse/YARN-4793 > > >> [3] https://issues.apache.org/jira/browse/YARN-4757 > > >> [4] https://issues.apache.org/jira/browse/YARN-6419 > > >> > > >> > > >> - > > >> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > > >> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > > >> > > >> > > > > > > - > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > > > > >
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (Binding), I tried to use YARN service assembly before to run different kinds of jobs (for example, distributed Tensorflow), it is really easy for end user to run jobs on YARN. Thanks to the whole team for the great job! Best, Wangda On Fri, Sep 1, 2017 at 3:33 PM, Gour Sahawrote: > +1 (non-binding) > > On 9/1/17, 11:58 AM, "Billie Rinaldi" wrote: > > >+1 (non-binding) > > > >On Thu, Aug 31, 2017 at 8:33 PM, Jian He wrote: > > > >> Hi All, > >> > >> I would like to call a vote for merging yarn-native-services to trunk. > >>The > >> vote will run for 7 days as usual. > >> > >> At a high level, the following are the key feautres implemented. > >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate > >>and > >> orchestrate existing services to YARN either docker or non-docker based. > >> - YARN-4793[2]. A Rest API server for user to deploy a service via a > >> simple JSON spec > >> - YARN-4757[3]. Extending today's service registry with a simple DNS > >> service to enable users to discover services deployed on YARN > >> - YARN-6419[4]. UI support for native-services on the new YARN UI > >> All these new services are optional and are sitting outside of the > >> existing system, and have no impact on existing system if disabled. > >> > >> Special thanks to a team of folks who worked hard towards this: Billie > >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith > >>Sharma > >> K S, Sunil G, Akhil PB. This effort could not be possible without their > >> ideas and hard work. > >> > >> Thanks, > >> Jian > >> > >> [1] https://issues.apache.org/jira/browse/YARN-5079 > >> [2] https://issues.apache.org/jira/browse/YARN-4793 > >> [3] https://issues.apache.org/jira/browse/YARN-4757 > >> [4] https://issues.apache.org/jira/browse/YARN-6419 > >> > >> > >> - > >> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > >> > >> > > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > >
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (non-binding) On 9/1/17, 11:58 AM, "Billie Rinaldi"wrote: >+1 (non-binding) > >On Thu, Aug 31, 2017 at 8:33 PM, Jian He wrote: > >> Hi All, >> >> I would like to call a vote for merging yarn-native-services to trunk. >>The >> vote will run for 7 days as usual. >> >> At a high level, the following are the key feautres implemented. >> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate >>and >> orchestrate existing services to YARN either docker or non-docker based. >> - YARN-4793[2]. A Rest API server for user to deploy a service via a >> simple JSON spec >> - YARN-4757[3]. Extending today's service registry with a simple DNS >> service to enable users to discover services deployed on YARN >> - YARN-6419[4]. UI support for native-services on the new YARN UI >> All these new services are optional and are sitting outside of the >> existing system, and have no impact on existing system if disabled. >> >> Special thanks to a team of folks who worked hard towards this: Billie >> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith >>Sharma >> K S, Sunil G, Akhil PB. This effort could not be possible without their >> ideas and hard work. >> >> Thanks, >> Jian >> >> [1] https://issues.apache.org/jira/browse/YARN-5079 >> [2] https://issues.apache.org/jira/browse/YARN-4793 >> [3] https://issues.apache.org/jira/browse/YARN-4757 >> [4] https://issues.apache.org/jira/browse/YARN-6419 >> >> >> - >> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org >> >> - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Merge yarn-native-services branch into trunk
+1 (non-binding) On Thu, Aug 31, 2017 at 8:33 PM, Jian Hewrote: > Hi All, > > I would like to call a vote for merging yarn-native-services to trunk. The > vote will run for 7 days as usual. > > At a high level, the following are the key feautres implemented. > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to migrate and > orchestrate existing services to YARN either docker or non-docker based. > - YARN-4793[2]. A Rest API server for user to deploy a service via a > simple JSON spec > - YARN-4757[3]. Extending today's service registry with a simple DNS > service to enable users to discover services deployed on YARN > - YARN-6419[4]. UI support for native-services on the new YARN UI > All these new services are optional and are sitting outside of the > existing system, and have no impact on existing system if disabled. > > Special thanks to a team of folks who worked hard towards this: Billie > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma > K S, Sunil G, Akhil PB. This effort could not be possible without their > ideas and hard work. > > Thanks, > Jian > > [1] https://issues.apache.org/jira/browse/YARN-5079 > [2] https://issues.apache.org/jira/browse/YARN-4793 > [3] https://issues.apache.org/jira/browse/YARN-4757 > [4] https://issues.apache.org/jira/browse/YARN-6419 > > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > >