Thanks Christian,
We would want to retry indefinitely. Or at least
for say x minutes. If we don't poll how do we keep the heart beat alive to
Kafka. We never want to loose this message and only want to commit to Kafka
when the message is in Mongo. That's either as a successful message in a
collection or an unsuccessful message in an error collection.
Right now I let the consumer die and don't create a new one for x minutes. This
causes a lot of rebalancing.
Michael
> On 9 Mar 2016, at 21:12, Christian Posta <christian.po...@gmail.com> wrote:
>
> So can you have to decide how long you're willing to "wait" for the mongo
> db to come back, and what you'd like to do with that message. So for
> example, do you just retry inserting to Mongo for a predefined period of
> time? Do you try forever? If you try forever, are you okay with the
> consumer threads blocking indefinitely? Or maybe you implement a "circuit
> breaker" to shed load to mongo? Or are you willing to stash the message
> into a DLQ and move on and try the next message?
>
> You don't need to "re-consume" the message do you? Can you just retry
> and/or backoff-retry with the message you have? And just do the "commit" of
> the offset if successfully?
>
>
>
> On Wed, Mar 9, 2016 at 2:00 PM, Michael Freeman <mikfree...@gmail.com>
> wrote:
>
>> Hey,
>> My team is new to Kafka and we are using the examples found at.
>>
>>
>> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
>>
>> We process messages from kafka and persist them to Mongo.
>> If Mongo is unavailable we are wondering how we can re-consume the messages
>> while we wait for Mongo to come back up.
>>
>> Right now we commit after the messages for each partition are processed
>> (Following the example).
>> I have tried a few approaches.
>>
>> 1. Catch the application exception and skip the kafka commit. However the
>> next poll does not re consume the messages.
>> 2. Allow the consumer to fail and restart the consumer. This works but
>> causes a rebalance.
>>
>> Should I attempt to store the offset and parition (in memory) instead and
>> attempt to reseek in order to re consume the messages?
>>
>> Whats the best practice approach in this kind of situation? My priority is
>> to never loose a message and to ensure it makes it to Mongo. (Redelivery is
>> ok)
>>
>> Thanks for any help or pointers in the right direction.
>>
>> Michael
>
>
>
> --
> *Christian Posta*
> twitter: @christianposta
> http://www.christianposta.com/blog
> http://fabric8.io
