I totally agree with what Taylor said. Slider looks really cool from a generic application perspective, but to get the most out of YARN we will probably want to do some things that slider cannot do generically. If we want to have a discussion about storm on YARN being an official part of storm we can. We¹ll gladly donate the code to apache if that is what the community decides. But this thread is about storm on mesos. Mainly if there is sufficient community interest/support to pull this in as an official part of storm. There are a few groups using it right now, but I don¹t know if there are enough people who are willing and able to support it in Apache.
- Bobby On 6/9/14, 9:44 PM, "P. Taylor Goetz" <[email protected]> wrote: >Jon, > >Likely, both. But the community will ultimately drive that decision. > >Storm on YARN will be an important step toward integrating Storm into the >Hadoop 2.x ecosystem. > >Getting there obviously involve potential add-ons/modifications to both >Storm and YARN. There is already a lot of work and collaboration going on >in this respect. Keep an eye on the Apache Slider project as well as the >storm-on-yarn project from Yahoo!. Better yet, get involved if you can. > >This (resource negotiation) is a really interesting and important area. >One of the possibilities I like to point out is the concept of >auto-scaling, where Storm adjusts to demand by dynamically >requesting/allocating resources, rebalancing, etc. That won't happen >overnight, but rather over a number of iterations. The important part is >that it is done right, with community support. > >Exciting times. > >-Taylor > >> On Jun 9, 2014, at 9:52 PM, Jon Logan <[email protected]> wrote: >> >> What are the plans for the Storm-YARN work that is being worked on? They >> both seem to be solving similar problems, do we want to support both? >> sub-projects? >> >> >>> On Mon, Jun 9, 2014 at 6:01 PM, P. Taylor Goetz <[email protected]> >>>wrote: >>> >>> FYI... We would like to see the level of community interest in support >>>for >>> storm on mesos. >>> >>> -Taylor >>> >>> >>> Begin forwarded message: >>> >>>> From: "P. Taylor Goetz" <[email protected]> >>>> Date: May 20, 2014 at 2:25:53 PM EDT >>>> To: Brenden Matthews <[email protected]> >>>> Cc: Bobby Evans <[email protected]>, James Xu >>>><[email protected]>, >>> Nathan Marz <[email protected]>, Derek Dagit >>><[email protected]>, >>> David Lao <[email protected]>, Flip Kromer < >>> [email protected]>, "Michael G. Noll" <[email protected]>, >>> Benjamin Hindman <[email protected]>, "[email protected]" < >>> [email protected]>, Andy Feng <[email protected]> >>>> Subject: Re: Where should `storm-mesos` live? >>>> Can we move this discussion to the dev@ list? That way we¹ll reach a >>> wider audience and will be better able to gauge the level of interest >>>from >>> the broader community. >>>> >>>> If there¹s enough interest, I¹m sure we can work something out. >>>> >>>> - Taylor >>>> >>>> >>>>> On May 20, 2014, at 2:13 PM, Bobby Evans <[email protected]> wrote: >>>>> >>>>> I hope you find someone it would be great to see this as a part of >>> storm proper. >>>>> >>>>> - Bobby >>>>> >>>>> From: Brenden Matthews <[email protected]> >>>>> Date: Tuesday, May 20, 2014 at 1:09 PM >>>>> To: "Yahoo! Inc." <[email protected]> >>>>> Cc: James Xu <[email protected]>, Nathan Marz < >>> [email protected]>, Derek Dagit <[email protected]>, "P. Taylor >>> Goetz" <[email protected]>, David Lao >>><[email protected]>, >>> Flip Kromer <[email protected]>, "Michael G. Noll" < >>> [email protected]>, Benjamin Hindman >>><[email protected]>, >>> "[email protected]" <[email protected]>, Andrew Feng < >>> [email protected]> >>>>> Subject: Re: Where should `storm-mesos` live? >>>>> >>>>> Bump. >>>>> I'm still trying to find people interested in helping shepherd this. >>>>>So >>> far no takers. >>>>>> On Apr 30, 2014 10:42 AM, "Bobby Evans" <[email protected]> wrote: >>>>>> I still want to know who else is signing up to help maintain >>> storm-mesos. With the requirement of two +1s from committers before a >>> commit if we have only one person that is really familiar and actively >>> reviewing pull requests means nothing will ever be committed. For me >>> personally if there are other people at Twitter or AirBnb that know the >>> code well and want to help maintain it I am willing to consider them >>>for >>> commitership, but I am not the only one that votes on these things so I >>> would appreciate other opinions too. After that we need to have >>>several >>> official votes/discussions. At a minimum one per new committer, which >>> would take place on the private list, and one to officially adopt >>> storm-mesos into the storm project, which would take place on the >>>public >>> list, or as part of a pull request to the storm project. After that >>>you >>> and AirBnb would have to sign the Apache Contributor License >>>Agreement, but >>> that is all explained after/if the votes pass. >>>>>> >>>>>> -Bobby >>>>>> >>>>>> From: Brenden Matthews <[email protected]> >>>>>> Date: Wednesday, April 30, 2014 at 10:43 AM >>>>>> To: "Yahoo! Inc." <[email protected]> >>>>>> Cc: James Xu <[email protected]>, Nathan Marz < >>> [email protected]>, Derek Dagit <[email protected]>, David Lao < >>> [email protected]>, "P. Taylor Goetz" >>><[email protected]>, >>> Flip Kromer <[email protected]>, Benjamin Hindman < >>> [email protected]>, "Michael G. Noll" >>><[email protected]>, >>> Andrew Feng <[email protected]>, "[email protected]" < >>> [email protected]> >>>>>> Subject: Re: Where should `storm-mesos` live? >>>>>> >>>>>> Okay, sounds good. What do we need to do to start that process? Does >>> it require any action on my part? >>>>>>> On Apr 30, 2014 7:45 AM, "Bobby Evans" <[email protected]> wrote: >>>>>>> I did a quick read through of the code this morning. I also looked >>>>>>>at >>> the recent changes and the full change log from when it branched off >>>the >>> original release. The code is clean and easy to follow, the changes >>>made >>> recently seem to make a lot of since. I would be +1 for pulling this >>>in >>> and having Brenden be a committer, assuming that we have at least one >>>other >>> committer, new or current, that can review changes and help to maintain >>> that section of storm. I am happy to do what I can but I am not a >>>mesos >>> expert. I assume Nathan can do some reviews too, but I do not want to >>> volunteer anyone. We also would need to do a formal vote on this on >>>the >>> real mailing list. >>>>>>> >>>>>>> - Bobby >>>>>>> >>>>>>> From: Brenden Matthews <[email protected]> >>>>>>> Date: Tuesday, April 29, 2014 at 1:36 PM >>>>>>> To: "Michael G. Noll" <[email protected]> >>>>>>> Cc: "Yahoo! Inc." <[email protected]>, "P. Taylor Goetz" < >>> [email protected]>, Benjamin Hindman <[email protected]>, >>>Nathan >>> Marz <[email protected]>, "[email protected]" >>><[email protected]>, >>> Flip Kromer <[email protected]>, Andrew Feng <[email protected]>, >>> Derek Dagit <[email protected]>, David Lao < >>> [email protected]>, James Xu <[email protected]> >>>>>>> Subject: Re: Where should `storm-mesos` live? >>>>>>> >>>>>>>> On Tue, Apr 29, 2014 at 9:15 AM, Michael G. Noll < >>> [email protected]> wrote: >>>>>>>> That aside there's one further thing I'd like to mention. Apart >>>>>>>>from >>>>>>>> the theoretical question "where should storm-mesos live?" there's >>> also >>>>>>>> the practical side: Assuming we would indeed like to take >>> storm-mesos >>>>>>>> under the Apache Storm umbrella, taking advantage of Brenden's >>>>>>>>work, >>>>>>>> would we actually have e.g. the bandwidth and skills to properly >>>>>>>> maintain the project moving forward? (e.g. having the buy-in of >>> people >>>>>>>> who have a direct interest in keeping this component up to date, >>>>>>>>who >>>>>>>> can test it properly and at scale, etc.) Would you or your team >>>>>>>>at >>>>>>>> AirBnB, Brenden, be willing and able to continue to contribute to >>>>>>>>a >>>>>>>> storm-mesos component? Given my lack of direct Mesos experience I >>>>>>>> don't know about what level of effort we're talking here, so I >>>>>>>>hope >>>>>>>> this comment of mine doesn't come across as making the original >>>>>>>> question even more complicated. :-) >>>>>>> >>>>>>> I'm happy to continue maintaining it as a public project for as >>>>>>>long >>> as I can. If we were to include the project as part of Apache Storm, I >>> would presumably need to be made a committer for that project. >>>> >>>
