Hi,

I'm new to Kafka and I'm trying to design a wrapper library in both Java
and Go (uses Confluent/Kafka-Go) for Kafka to be used internally. For my
use-case, CommitSync is a crucial step and we should do a read only after
properly committing the old one. Repeated processing is not a big issue and
our client service is idempotent enough. But data loss is a major issue and
should not occur.

I will create X number of consumers initially and will keep on polling from
them. Hence I would like to know more about the negative scenario's that
could happen here, Impact of them and how to properly handle them.

I would like to know more about:

1) Network issue during consumer processing:
     What happens when network goes of for a brief period and comes back?
Does Kafka consumer automatically handle this and becomes alive when
network comes back or do we have to reinitialise them? If they come back
alive do they resume work from where they left of?
Eg: Consumer X read 50 records from Partition Y. Now internally the
consumer offset moved to +50. But before committing network issue happens
and the comes back alive. Now will the consumer have the metadata about
what it read for last poll. Can it go on to commit +50 in offset?

2) Rebalancing in consumer groups. Impact of them on existing consumer
process - whether the existing working consumer instance will pause and
resume work during a rebalance or do we have to reinitialize them? How long
can rebalance occur? If the consumer comes back alive after rebalance, does
it have metadata about it last read?

3) What happens when a consumer joins during a rebalancing. Ideally it is
again a rebalancing scenario. What will happen now? The existing will be
discarded and the new one starts or will wait for the existing rebalance to
complete?

Kindly help me understanding these scenarios is detail and suggest
solutions if possible. Also it would be much more helpful, if you could
point me to a resource - an online article / book or anything that provides
insight into the intricate details of Kafka.



Thanks and Regards,
Janardhanan V S

Reply via email to