Hi Pratyusha, Is it possible for you to write back to kafka topic after parent tuple is processed and other topology can read ?
On Wed, Aug 17, 2016 at 3:16 PM, Pratyusha Rasamsetty < [email protected]> wrote: > Hi all, > > *My requirement is *To process and index a huge data set to Elasticsearch. > > For each tuple that spout emits, about 100 child tuples gets emitted. Each > of them needs to be processed and indexed to Elasticsearch. Once the > children gets indexed, parent tuple that spout emitted also need to be > indexed by querying the children index which is already indexed. > > I am able to achieve the whole functionality using normal storm. But since > the processing of the tuple is too long, I had to disable guaranteed > message processing. > > Since processing tuples repeatedly is a costly operation for me, I decided > to use storm trident as it claims to be support exactly once processing. > > The problem here is I could not achieve the complete functionality with > trident. > > I have to index children and based on the bulk response that I get from > elasticsearch, I need to emit some more tuples for further processing. I > understand that we can use trident state for doing batch insert to > elasticsearch. But based on the response I could not emit from trident > state. > > Please help me solve this - "Batch insert and emit based on response using > Trident." > > > Thanks > Pratyusha > -- Regards Tousif Khazi
