Hi kang, thanks for the response. I don't wan to drop messages, but not emit them as tuples until the throttling rates reach a certain level. But you are right that I can use a modified version of kafkaspout by maintaining offset per customer and not increment the corresponding offset until throttling rates for the customer allow it. There are some interesting problems to be solved with this approach, though. Thanks!
On Mar 9, 2014 7:57 AM, "Kang Xiao" <[email protected]> wrote: > Hi Neelesh > > Do you mean that you need a custom throttling logic by dropping some of > the large volume messages? Maybe you can implement a spout to do that. > > -- > Best Regards! > > 肖康(Kang Xiao,<[email protected]>) > Distributed Software Engineer > > 在 2014年3月6日 星期四，15:25，Neelesh 写道： > > Hi, > We're evaluating storm as our real time stream processing > infrastructure. We are a SaaS company, and the stream has messages from > different customers. Some of our customers may generate a large volume of > messages, starving messages from other customers. We've a custom throttling > logic built on top of Akka in another context, but was wondering if there > is a way of plugging in a custom throttling strategy in to Storm. topology > .max.spout.pending does not server our purpose. Any pointers are > appreciated > > Thanks > -Neelesh > > >
