Hao Yu, We were having the exact same configuration discussion. We also prefer your second option (dynamic configuration spout). However (@susheel) all bolts will get the same configuration from the spout... each bolt will do the filtering internally (probably will be implemented by a baseclass or wrapper bolt to abstract this design decision).
As for complexity.... Design A is more Complex since it mixes the "roles and responsibilities" of data processing and configuration processing. Design B (configuration spout) is more Compound (more moving parts, each one of them is simpler) which pushes the complexity up to the storm topology. We prefer it that way since storm topology complexity (graph) is more manageable then code complexity. Regards, Itai ________________________________________ From: Susheel Kumar Gadalay <[email protected]> Sent: Monday, December 22, 2014 12:50 PM To: [email protected] Subject: Re: Is it practical for bolt to receive data, not business data but processing configuration? If you want single spout to emit different fields, you can do so by giving a stream. By default all the fields are in default stream. So you emit a different steam like business parameters stream and let bolt1 and 2 subscribe to this stream. collector.emit("business_parameters", "param1", new value(...)) The default stream will work as it is now. On 12/22/14, 余昊 <[email protected]> wrote: > Hi, > > Considering a simple topology: > spoutD -> boltA -> boltB > where spoutD receives business data, and forwards to boltA for processing, > the results are then sent to boltB for further processing. > > In our scenario, it is required that the parameters (or configuration) of > processing logics in boltA or boltB can be modified at runtime. > > Our solution A is to send new paramter values directly to the specific > bolt. > But we think this is not practical, since in storm it seems data input (no > matter what kind of data) should be done > in spout, and bolt gets input either from the upstream bolt or from spout - > is that right? > > So we thought about solution B, which is adding a spoutC which is > specifically for receiving configuration data. > Then the topology turns to: > spoutD -> boltA -> boltB > | | > spoutC > where spoutC can dispatch the new processing parameter values to the boltA > or boltB. > Well, the cost is that as you can see the topology gets more complicating. > That's why we do not like solution B. > > If solution A can work in a practical way, we prefer it to solution B. But > is it possible? Or do you have any better suggestions? > > > > Best regards, > > Hao Yu >
