Any Storm Streaming job runs in its own space and doesn't interact with other topology. Your tuple distribution will be across the topology within the number of workers on the number of bolts defined, so for instance if you have shuffle grouping enabled and specific data of your interest
0 1 - Kafka Partition s1 s2 - Subscribed Spouts b1 b2 b3 b4 - bolts available Then all the data(tuple) which is passing through s1 and s2(which indeed are subscribed to Kafka Partition 0 and 1) are going to emit to bolts b[1-4] based on the hash of the tuple key so it will be something like, for the data tuple(somefancydata1) - b1 tuple(somefancydata43) - b3 tuple(somefancydata855) - b1 and so on. . .the data(tuple) will be distinct across the bolts! Let me know if that solves your concern! Cheers! On Mon, Jun 8, 2015 at 11:05 AM, Seungtack Baek < [email protected]> wrote: > @Vineet, > > Thanks a lot for "another" timely response! > > Actually I have read that section but it wasn't still clear (to me, and I > guess to me only) whether field grouping was concerning the whole cluster > (or topology) or for the same worker only.. Maybe I am not too familiar > with the "zoo". > > > Thanks, > Baek > > > *Seungtack Baek | Precocity, LLC* > > Tel/Direct: (972) 378-1030 | Mobile: (214) 477-5715 > > *[email protected] <[email protected]>* | > www.precocityllc.com > > > This is the end of this message. > > -- > > On Mon, Jun 8, 2015 at 12:31 AM, Vineet Mishra <[email protected]> > wrote: > >> Hi Seung, >> >> You can better refer to the section Stream Groupings in the following >> link attached below >> >> https://storm.apache.org/documentation/Concepts.html >> >> It will get you better understanding of the tuple distribution in Storm, >> for clear understanding here is the pictorial representation of the same, >> >> https://blog.safaribooksonline.com/wp-content/uploads/2013/06/Grouping.png >> >> Cheers! >> >> On Mon, Jun 8, 2015 at 10:47 AM, Dima Dragan <[email protected]> >> wrote: >> >>> Hi, Seungtack! >>> >>> Distribution of messages will be depends only from grouping (in case of >>> "shuffe grouping", Tuples are randomly distributed across the all bolt's >>> tasks in a way such that each bolt is guaranteed to get an equal number of >>> tuples. >>> >>> Best regards, >>> Dmytro Dragan >>> On Jun 8, 2015 07:12, "Seungtack Baek" <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I have read from the documentation that if you have more spout tasks >>>> than kafka partition, the excessive tasks will remain idle for entire >>>> lifecycle of the topology. >>>> >>>> Now, Let's consider 4 spout tasks, 32 bolt tasks (of one class) in 4 >>>> workers (in 4 nodes) and 2 partitions in kafka. Then 2 tasks will be >>>> assigned to each partitions in kafka and the other 2 will remain idle. >>>> However, does that mean that only the bolts within the same worker will get >>>> the messages (assuming shuffle grouping)? Or, do the messages get emitted >>>> to whatever bolt taks available, regardless of which worker? >>>> >>>> Thanks, >>>> Baek >>>> >>>> >>>> *Seungtack Baek | Precocity, LLC* >>>> >>>> Tel/Direct: (972) 378-1030 | Mobile: (214) 477-5715 >>>> >>>> *[email protected] <[email protected]>* | >>>> www.precocityllc.com >>>> >>>> >>>> This is the end of this message. >>>> >>>> -- >>>> >>>> On Sun, Jun 7, 2015 at 10:12 PM, Seungtack Baek < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have read from the documentation that if you have more spout tasks >>>>> than kafka partition, the excessive tasks will remain idle for entire >>>>> lifecycle of the topology. >>>>> >>>>> Now, Let's consider 4 spout tasks, 32 bolt tasks (of one class) in 4 >>>>> workers (in 4 nodes) and 2 partitions in kafka. Then 2 tasks will be >>>>> assigned to each partitions in kafka and the other 2 will remain idle. >>>>> However, does that mean that only the bolts within the same worker will >>>>> get >>>>> the messages (assuming shuffle grouping)? Or, do the messages get emitted >>>>> to whatever bolt taks available, regardless of which worker? >>>>> >>>>> Thanks, >>>>> Baek >>>>> >>>>> >>>>> *Seungtack Baek | Precocity, LLC* >>>>> >>>>> Tel/Direct: (972) 378-1030 | Mobile: (214) 477-5715 >>>>> >>>>> *[email protected] <[email protected]>* | >>>>> www.precocityllc.com >>>>> >>>>> >>>>> This is the end of this message. >>>>> >>>>> -- >>>>> >>>> >>>> >> >
