Most likely it is not possible. You need to deploy two difference topologies for it. Or you need to have a de duplication logic implemented downstream in your database layer and s3 layer.
On Fri, Feb 26, 2016 at 12:28 PM, pradeep s <[email protected]> wrote: > Hi , > I am processing CDC messages using storm. My topology has a 2 bolts . > First one is a bolt to write data to S3 and second one is a bolt to write > to Database. > i am using anchored tuples. Now i am facing with the issue of handling > duplicate writes. > When message is successful in S3 bolt and failure happens in DB bolt, > tuple replay is happening. During replay again S3 bolt call is invoked and > data is written to S3 again. > Any way i can have the tuple replayed only for the failed bolt. > > Topology > ----------- > > topologyBuilder.setSpout("mdpSpout", new SQSMessageReaderSpout(queueUrl), > SPOUT_PARALLELISM); > > topologyBuilder.setBolt("mdpS3Bolt", new S3WriteBolt(), > BOLT_PARALLELISM).shuffleGrouping("mdpSpout"); > > topologyBuilder.setBolt("dbBolt", new DbBolt(), BOLT_PARALLELISM > ).shuffleGrouping("mdpS3Bolt"); > > > > > Regards > Pradeep S >
