Re: running multiple topologies in same cluster

[Spico Florin]

Hello!
Thank you all for your answers! I guess I'll wait for adding in support for
resource aware scheduling in a multi-tenant stand alone storm cluster.
Regards,
 Florin

On Tue, Feb 2, 2016 at 1:00 AM, Erik Weathers  wrote:

> hi Spico,
>
> As Bobby said, native Storm is going to have better support for this soon.
>
> FWIW, there is also the storm-on-mesos project, which we've been running
> on for almost 2 years at Groupon.
>
>- https://github.com/mesos/storm
>
> Caveats:
>
>- storm's logviewer is unsupported
>- scheduling can be suboptimal, causing topologies of different "size"
>(resource requirements) to starve each other
>   - side effect of needing to dynamically calculate storm "slots"
>   from mesos resource offers, my team has a framework change we are 
> testing
>   that will improve that behavior.
>- it's relatively complex to operate since there are so many different
>moving parts / components
>   - mesos-master
>   - mesos-slave/agent (it's being renamed)
>   - storm nimbus (MesosNimbus -> the mesos scheduler)
>   - storm supervisor (MesosSupervisor -> the mesos executor)
>   - storm worker (the mesos task)
>   - ZooKeeper
>- it probably won't work nicely with all the fancy security stuff that
>has been added to Storm in 0.10.0+
>
> - Erik
>
> On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans  wrote:
>
>> We are currently adding in support for resource aware scheduling in a
>> multi-tenant stand alone storm cluster.  It is still alpha quality but we
>> plan on getting it into production at Yahoo this quarter.  If you can wait
>> that would be the preferred way I see to support your use case.
>>
>> - Bobby
>>
>>
>> On Monday, February 1, 2016 12:16 PM, Spico Florin 
>> wrote:
>>
>>
>> Hello!
>> I have an use case where we have to deploy many tpologies in a storm
>> cluster.
>> 1.we would like to know if running these topologies in combination with
>> apache slider over yarn would bring us some benefits in terms of resource
>> consumption?
>> 2. in such cases (running many topolgies aprox 60) what are the best
>> practices on how to run them over a cluster with a smart load balanced
>> hardware resources consumption (cpu, ram)?
>> I look forward for your answers.
>> Regards,
>> Florin
>>
>>
>>
>>
>

running multiple topologies in same cluster

[Spico Florin]

Hello!
I have an use case where we have to deploy many tpologies in a storm
cluster.
1.we would like to know if running these topologies in combination with
apache slider over yarn would bring us some benefits in terms of resource
consumption?
2. in such cases (running many topolgies aprox 60) what are the best
practices on how to run them over a cluster with a smart load balanced
hardware resources consumption (cpu, ram)?
I look forward for your answers.
Regards,
Florin