TL;DR: I want to verify that assigning one bolt tasks to a worker by implementing a scheduler prevents the EvenScheduler from assigning any other tasks to that worker slot.
I am trying to assign all tasks of a specific bolt to one machine and let the even scheduler do the rest. I have specified total topology workers to be 2 (although the available slots are more) and since one is occupied by cluster.assign, the EvenScheduler cannot assign any more tasks to it, so it assigns all the rest to the second worker. Looking at the Cluster#getAvailableSlots after the custom assignment and before the even scheduler, the occupied worker slot is still assignable but not available. I believe this works as intended. On the other hand this means that 2 powerful nodes cannot have tasks evenly distributed to them with 1 bolt being set on node 1 and while having 2 JVMs in total for optimal performance (1 per machine). Unless of course a solo custom scheduler is used. Please correct me if I'm wrong. P.S. Adding one more worker to the total topology workers (3 in total) does the job. On Mon, Sep 1, 2014 at 10:45 PM, Michael Vogiatzis < [email protected]> wrote: > Sounds good, thanks for the advice. > > Regards, > Michael > > > On Mon, Sep 1, 2014 at 9:43 PM, Nathan Marz <[email protected]> wrote: > >> You'd probably want to make use of the component-specific configuration >> on spouts and bolts to funnel the information to your scheduler as to which >> tasks should be colocated. >> >> >> On Mon, Sep 1, 2014 at 1:42 PM, Nathan Marz <[email protected]> >> wrote: >> >>> Yes, you can definitely accomplish that by writing your own scheduler. >>> >>> >>> On Mon, Sep 1, 2014 at 12:38 PM, Michael Vogiatzis < >>> [email protected]> wrote: >>> >>>> Probably what I was looking for is an example of a pluggable scheduler >>>> in order to control where particular bolt/spouts run on. Just found this >>>> useful post: >>>> http://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/ >>>> . >>>> >>>> I actually want to paralellize one of my bolts and also share the same >>>> memory between the tasks without using any external memcached etc because >>>> read latency really maters here. Is a pluggable scheduler (all bolt tasks >>>> in a strong node) and a static concurrent hashmap a good approach to it? >>>> >>>> Thanks, >>>> Michael >>>> >>>> >>>> >>>> On Mon, Sep 1, 2014 at 8:10 PM, Nathan Marz <[email protected]> >>>> wrote: >>>> >>>>> There's not much more to it than what was documented on the blog post >>>>> announcing it: >>>>> http://storm.incubator.apache.org/2013/01/11/storm082-released.html >>>>> >>>>> What are your questions about it? >>>>> >>>>> >>>>> >>>>> On Mon, Sep 1, 2014 at 12:04 PM, Michael Vogiatzis <[email protected] >>>>> > wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Does anyone know of any good resources/how-tos on the Isolation >>>>>> Scheduler? >>>>>> >>>>>> Thanks, >>>>>> Michael >>>>>> >>>>>> -- >>>>>> Michael Vogiatzis >>>>>> Twitter: @mvogiatzis <https://twitter.com/mvogiatzis> >>>>>> http://micvog.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Twitter: @nathanmarz >>>>> http://nathanmarz.com >>>>> >>>> >>>> >>>> >>>> -- >>>> Michael Vogiatzis >>>> Twitter: @mvogiatzis <https://twitter.com/mvogiatzis> >>>> http://micvog.com/ >>>> >>> >>> >>> >>> -- >>> Twitter: @nathanmarz >>> http://nathanmarz.com >>> >> >> >> >> -- >> Twitter: @nathanmarz >> http://nathanmarz.com >> > > > > -- > Michael Vogiatzis > Twitter: @mvogiatzis <https://twitter.com/mvogiatzis> > http://micvog.com/ > -- Michael Vogiatzis Twitter: @mvogiatzis <https://twitter.com/mvogiatzis> http://micvog.com/
