One approach that I can think of would be to place an event in Kafka (or similar) every time you add or remove an IP from the DB. Then have a second spout in your topology that listens for those events and sends them along to your bolt. Your bolt would then listen to messages as normal for filtering purposes but also listen for these blacklist events to in order to update its in-memory blacklist state. You’ll need to initialize the in-memory state of the blacklist when starting the bolt (in the prepare(…) method).
You will also need to use the ALL grouping for the blacklist events to ensure that all of your bolt instances get the memo that the blacklist has changed. Because of this, this approach would only be effective if the volume of blacklist updates is significantly less than the volume of messages that you’re processing. I hope that helps. Good luck! Joe — Joseph Beard [email protected] > On Aug 6, 2015, at 4:41 AM, Ritesh Sinha <[email protected]> > wrote: > > I need to write a bolt which processes the messages on the basis of IP. > Basically , i have a list of IP's stored in mongoDB.All the messages i get > will contain IP's, So , i need to check if that particular IP is present in > the list or not if it is present then i need to ignore the message else > process it. > > The list of IP is dynamic .There can be addition or deletion of IPs after the > topology has been deployed to storm. > > The problem is either I need to keep the IP is memory(but the changes can't > be reflected once the list is updated) or read it every time from mongoDB, > which makes the processing slow. > > Is there any way to implement it efficiently ? > Is it possible to trigger any event if addition or deletion of IP has > happened in mongoDB ?
