Jon, Likely, both. But the community will ultimately drive that decision.
Storm on YARN will be an important step toward integrating Storm into the Hadoop 2.x ecosystem. Getting there obviously involve potential add-ons/modifications to both Storm and YARN. There is already a lot of work and collaboration going on in this respect. Keep an eye on the Apache Slider project as well as the storm-on-yarn project from Yahoo!. Better yet, get involved if you can. This (resource negotiation) is a really interesting and important area. One of the possibilities I like to point out is the concept of auto-scaling, where Storm adjusts to demand by dynamically requesting/allocating resources, rebalancing, etc. That won't happen overnight, but rather over a number of iterations. The important part is that it is done right, with community support. Exciting times. -Taylor > On Jun 9, 2014, at 9:52 PM, Jon Logan <[email protected]> wrote: > > What are the plans for the Storm-YARN work that is being worked on? They > both seem to be solving similar problems, do we want to support both? > sub-projects? > > >> On Mon, Jun 9, 2014 at 6:01 PM, P. Taylor Goetz <[email protected]> wrote: >> >> FYI... We would like to see the level of community interest in support for >> storm on mesos. >> >> -Taylor >> >> >> Begin forwarded message: >> >>> From: "P. Taylor Goetz" <[email protected]> >>> Date: May 20, 2014 at 2:25:53 PM EDT >>> To: Brenden Matthews <[email protected]> >>> Cc: Bobby Evans <[email protected]>, James Xu <[email protected]>, >> Nathan Marz <[email protected]>, Derek Dagit <[email protected]>, >> David Lao <[email protected]>, Flip Kromer < >> [email protected]>, "Michael G. Noll" <[email protected]>, >> Benjamin Hindman <[email protected]>, "[email protected]" < >> [email protected]>, Andy Feng <[email protected]> >>> Subject: Re: Where should `storm-mesos` live? >>> Can we move this discussion to the dev@ list? That way we’ll reach a >> wider audience and will be better able to gauge the level of interest from >> the broader community. >>> >>> If there’s enough interest, I’m sure we can work something out. >>> >>> - Taylor >>> >>> >>>> On May 20, 2014, at 2:13 PM, Bobby Evans <[email protected]> wrote: >>>> >>>> I hope you find someone it would be great to see this as a part of >> storm proper. >>>> >>>> - Bobby >>>> >>>> From: Brenden Matthews <[email protected]> >>>> Date: Tuesday, May 20, 2014 at 1:09 PM >>>> To: "Yahoo! Inc." <[email protected]> >>>> Cc: James Xu <[email protected]>, Nathan Marz < >> [email protected]>, Derek Dagit <[email protected]>, "P. Taylor >> Goetz" <[email protected]>, David Lao <[email protected]>, >> Flip Kromer <[email protected]>, "Michael G. Noll" < >> [email protected]>, Benjamin Hindman <[email protected]>, >> "[email protected]" <[email protected]>, Andrew Feng < >> [email protected]> >>>> Subject: Re: Where should `storm-mesos` live? >>>> >>>> Bump. >>>> I'm still trying to find people interested in helping shepherd this. So >> far no takers. >>>>> On Apr 30, 2014 10:42 AM, "Bobby Evans" <[email protected]> wrote: >>>>> I still want to know who else is signing up to help maintain >> storm-mesos. With the requirement of two +1s from committers before a >> commit if we have only one person that is really familiar and actively >> reviewing pull requests means nothing will ever be committed. For me >> personally if there are other people at Twitter or AirBnb that know the >> code well and want to help maintain it I am willing to consider them for >> commitership, but I am not the only one that votes on these things so I >> would appreciate other opinions too. After that we need to have several >> official votes/discussions. At a minimum one per new committer, which >> would take place on the private list, and one to officially adopt >> storm-mesos into the storm project, which would take place on the public >> list, or as part of a pull request to the storm project. After that you >> and AirBnb would have to sign the Apache Contributor License Agreement, but >> that is all explained after/if the votes pass. >>>>> >>>>> -Bobby >>>>> >>>>> From: Brenden Matthews <[email protected]> >>>>> Date: Wednesday, April 30, 2014 at 10:43 AM >>>>> To: "Yahoo! Inc." <[email protected]> >>>>> Cc: James Xu <[email protected]>, Nathan Marz < >> [email protected]>, Derek Dagit <[email protected]>, David Lao < >> [email protected]>, "P. Taylor Goetz" <[email protected]>, >> Flip Kromer <[email protected]>, Benjamin Hindman < >> [email protected]>, "Michael G. Noll" <[email protected]>, >> Andrew Feng <[email protected]>, "[email protected]" < >> [email protected]> >>>>> Subject: Re: Where should `storm-mesos` live? >>>>> >>>>> Okay, sounds good. What do we need to do to start that process? Does >> it require any action on my part? >>>>>> On Apr 30, 2014 7:45 AM, "Bobby Evans" <[email protected]> wrote: >>>>>> I did a quick read through of the code this morning. I also looked at >> the recent changes and the full change log from when it branched off the >> original release. The code is clean and easy to follow, the changes made >> recently seem to make a lot of since. I would be +1 for pulling this in >> and having Brenden be a committer, assuming that we have at least one other >> committer, new or current, that can review changes and help to maintain >> that section of storm. I am happy to do what I can but I am not a mesos >> expert. I assume Nathan can do some reviews too, but I do not want to >> volunteer anyone. We also would need to do a formal vote on this on the >> real mailing list. >>>>>> >>>>>> - Bobby >>>>>> >>>>>> From: Brenden Matthews <[email protected]> >>>>>> Date: Tuesday, April 29, 2014 at 1:36 PM >>>>>> To: "Michael G. Noll" <[email protected]> >>>>>> Cc: "Yahoo! Inc." <[email protected]>, "P. Taylor Goetz" < >> [email protected]>, Benjamin Hindman <[email protected]>, Nathan >> Marz <[email protected]>, "[email protected]" <[email protected]>, >> Flip Kromer <[email protected]>, Andrew Feng <[email protected]>, >> Derek Dagit <[email protected]>, David Lao < >> [email protected]>, James Xu <[email protected]> >>>>>> Subject: Re: Where should `storm-mesos` live? >>>>>> >>>>>>> On Tue, Apr 29, 2014 at 9:15 AM, Michael G. Noll < >> [email protected]> wrote: >>>>>>> That aside there's one further thing I'd like to mention. Apart from >>>>>>> the theoretical question "where should storm-mesos live?" there's >> also >>>>>>> the practical side: Assuming we would indeed like to take >> storm-mesos >>>>>>> under the Apache Storm umbrella, taking advantage of Brenden's work, >>>>>>> would we actually have e.g. the bandwidth and skills to properly >>>>>>> maintain the project moving forward? (e.g. having the buy-in of >> people >>>>>>> who have a direct interest in keeping this component up to date, who >>>>>>> can test it properly and at scale, etc.) Would you or your team at >>>>>>> AirBnB, Brenden, be willing and able to continue to contribute to a >>>>>>> storm-mesos component? Given my lack of direct Mesos experience I >>>>>>> don't know about what level of effort we're talking here, so I hope >>>>>>> this comment of mine doesn't come across as making the original >>>>>>> question even more complicated. :-) >>>>>> >>>>>> I'm happy to continue maintaining it as a public project for as long >> as I can. If we were to include the project as part of Apache Storm, I >> would presumably need to be made a committer for that project. >>> >>
