Hey everyone, Requesting some help here..
Thanks, Janardhanan V S On Sat, Mar 16, 2019 at 7:36 AM Janardhanan V S <janardha...@exotel.in> wrote: > Hi, > > I'm new to Kafka and I'm trying to design a wrapper library in both Java > and Go (uses Confluent/Kafka-Go) for Kafka to be used internally. For my > use-case, CommitSync is a crucial step and we should do a read only after > properly committing the old one. Repeated processing is not a big issue and > our client service is idempotent enough. But data loss is a major issue and > should not occur. > > I will create X number of consumers initially and will keep on polling > from them. Hence I would like to know more about the negative scenario's > that could happen here, Impact of them and how to properly handle them. > > I would like to know more about: > > 1) Network issue during consumer processing: > What happens when network goes of for a brief period and comes back? > Does Kafka consumer automatically handle this and becomes alive when > network comes back or do we have to reinitialise them? If they come back > alive do they resume work from where they left of? > Eg: Consumer X read 50 records from Partition Y. Now internally the > consumer offset moved to +50. But before committing network issue happens > and the comes back alive. Now will the consumer have the metadata about > what it read for last poll. Can it go on to commit +50 in offset? > > 2) Rebalancing in consumer groups. Impact of them on existing consumer > process - whether the existing working consumer instance will pause and > resume work during a rebalance or do we have to reinitialize them? How long > can rebalance occur? If the consumer comes back alive after rebalance, does > it have metadata about it last read? > > 3) What happens when a consumer joins during a rebalancing. Ideally it is > again a rebalancing scenario. What will happen now? The existing will be > discarded and the new one starts or will wait for the existing rebalance to > complete? > > Kindly help me understanding these scenarios is detail and suggest > solutions if possible. Also it would be much more helpful, if you could > point me to a resource - an online article / book or anything that provides > insight into the intricate details of Kafka. > > > > Thanks and Regards, > Janardhanan V S >
