Hi Ryan,
I have attached a sample implementation of KafkaBolt that does micro
batching on the tuples. Also it does take care of tuples that would be
acked for successful write to Kafka and fails in case of any exception.
The frequency of tuple batching is based on the tick tuples configured
in seconds or the batch size which is configurable number. For e.g. to set
the frequency of 10 seconds below is the configuration.
conf.put("topology.tick.tuple.freq.secs", 10);
The batch size is hardcoded in the attached file.
Thanks,
Raghavendra Nandagopal
On Thu, May 28, 2015 at 7:43 PM, Ryan Persaud <[email protected]> wrote:
> Hello,
>
> I was wondering if anyone is aware of an implementation of the KafkaBolt
> that uses the new Java Kafka Producer instead of the legacy Scala one. I
> have a topology with a high volume of tuples that need to be written to
> Kafka, so I would like to take advantage of the batching that the
> asynchronous mode provides. However, I also want to only ack tuples after
> I have confirmation that they have been successfully written to at least
> one Kafka broker. Since the Scala producer does not support callbacks, I
> think I would need to use the new Java producer to realize the desired
> functionality.
>
> Thoughts?
>
> Thanks,
> -Ryan
>
