Mesosphere has official support for Storm on Mesos: https://github.com/mesos/storm
On Wed, May 27, 2015 at 11:14 AM, <[email protected]> wrote: > *Dell - Internal Use - Confidential * > > Thanks Bobby, for the detailed answer. > > > > So it sounds like , it is better not to combine Storm with batch > workloads at this point (yarn, mesos or ec2), due to the network saturation > and timeout threats. > > > > Is this behavior also seen in other streaming frameworks like spark > streaming running on YARN. > > > > *From:* Bobby Evans [mailto:[email protected]] > *Sent:* Wednesday, May 27, 2015 9:07 AM > *To:* Jeffery Maass; [email protected] > *Subject:* Re: Status of running storm on yarn (the yahoo project) > > > > Mesos is very similar to YARN. It is a resource scheduler. Storm in the > past had support for mesos, through a separate repo > > > > https://github.com/nathanmarz/storm-mesos > > > > it might still work with the latest versions of storm. I don't know. The > concept here is that there was a special layer installed that would look > for when the cluster had outstanding requests and not enough resources to > meet those requests. It would then request that many resources from mesos, > launch supervisors on those nodes and let the scheduler do the rest. It > works quire well for elasticity at a small scale, or when you have a lot > more network bandwidth than you need. The problem is if mesos, or YARN, or > open-stack, or EC2, or ... collocates your storm topology with some big > batch job that suddenly saturates the network for a few seconds to a min > heartbeats could start to time out, traffic would not flow from one worker > to another, etc. For some topologies all you do is tune your timeouts so > workers don't get shot and relaunched too frequently and live with the > noise from other stuff happening on the network. For us though we have > some very tight SLAs, if the data is 5 seconds old throw it away I cannot > use it any more. > > > > My current goal with storm in this area is to have it be aware of the > resources that your topology is using, the SLAs that it has, its desired > budget for resources, how far over that budget it is willing to go, Where > it could possibly get other resources if needed (i.e. YARN, Mesos, Open > Stack), and any other constraints it might have. Storm would then take all > of this into account and adjust the scheduling of your topology so that it > can grow and shrink with the resources it needs to meet the SLAs it has, > optionally taking some of those resources from other systems if needed. > This is still a ways out, but looking at the research that is being done in > this area it should be doable in the next year or so. > > > > - Bobby > > > > > > > > On Wednesday, May 27, 2015 8:38 AM, Jeffery Maass <[email protected]> > wrote: > > > > I have heard Nathan Marz mention Mesos. > > How is yarn / storm-yarn / slider-yarn different from Mesos? > > These are the links I found to Mesos: > https://github.com/mesos/storm > https://github.com/nathanmarz/storm-mesos > http://mesos.apache.org/ > > > Thank you for your time! > > +++++++++++++++++++++ > Jeff Maass <[email protected]> > linkedin.com/in/jeffmaass > stackoverflow.com/users/373418/maassql > +++++++++++++++++++++ > > > > On Wed, May 27, 2015 at 8:28 AM, Bobby Evans <[email protected]> wrote: > > storm-yarn was originally done as a proof of concept. We had plans to > take it further, but the amount of work required to make it production > ready on a very heavily used cluster was more then we were willing to > invest at the time. Most of that work was around network scheduling, > isolation and prioritization, mainly in YARN itself. There has been some > work looking into this, but nothing much has happened with it. At the same > time http://slider.incubator.apache.org/ showed up and is now the > preferred way to run Storm on YARN. To get around the networking issues > most people will tag a subset of their cluster, a few racks, and only > schedule storm to run on those nodes. Long term I really would like to > revive storm on yarn, and integrate it directly into storm. Giving storm > and the scheduler the ability to request new resources with specific > constraints opens up a lot of new possibilities. If you want to help out, > or if anyone else wants to help out with this work, I would be very happy > to file some JIRA in open source and help direct what needs to be done. > > - Bobby > > > > > > On Wednesday, May 27, 2015 4:59 AM, Spico Florin <[email protected]> > wrote: > > > > Hello! > > I'm interesting in running the storm topologies on yarn. > > I was looking at the yahoo project https://github.com/yahoo/storm-yarn, > and I could observed that there is no activity since 7 months ago. Also, > the issues and requests lists are not updated. > > Therefore I have some questions: > > 1. Is there any plan to evolve this project? > > 2. Is there any plan to integrate this project in the main branch? > > 3. Is someone using this approach in production ready mode? > > > > I look forward for your answers. > > Regards, > > Florin > > > > > > > > > > > > > > > > > -- Twitter: @nathanmarz http://nathanmarz.com
