Hi all, I'm new to this mailing list, this is my first post, hello and thanks in advance to everybody spend his/her time to read this.
I'm writing a Java application with kafka_2.11-0.10.1.0, I have just to copy few data from a database to Solr. In my application I've imagined to have two types of topics: control topic and data topics. In particular there is only one control topic but there can be many data topics. So the Producer has 3 steps: 1. create a new topic X (where X is a GUID or a timestamp) and write all data messages inside. 2. write a closure message that marks the end of data messages. 3. create a new message in the control topic declaring the creation of a new topic X On the other side there is a consumer that execute the following steps: 1. polls from the control data topics every few seconds 2. if see there is a new control message, starts to process the messages data polling from the new topic X. 3. when the closure message is found the process stops and start again from step 1 Now my idea is that: it is consumer responsibility see if a topic X is complete, check if the producer has sent everything. I suppose just to verify there is a closure message at end of topic. I'm curious to know your suggestion, if check for a special closure message at end of topic is enough. Or if there are already best practices useful to handle this type of scenario. Best regards and again thanks in advance for your time, Vincenzo -- Vincenzo D'Amore email: v.dam...@gmail.com skype: free.dev mobile: +39 349 8513251