Hello.
While writing integration tests for our solution which uses new Kafka
consumer group (the one without zookeeper), I faced an issue with
reconnection. Briefly our test uses an external Kafka server and starts
a consumer with the predefined group id for each test case, then checks
that message is received and afterwards shuts it down. The problem is
that the consumer not always receives the message despite it works
correctly.
I figured out that when a consumer disconnects, it could take some time
for Kafka to remove the consumer from GroupCoordinator and if the
consumer tries to connect right after it has been closed, the whole
group restabilization process will be quite slow. From what I see, the
time of consumer removal from GroupCoordinator directly depends on
session.timeout and the process cannot be forced.
Is there any way to ensure that consumer is fully stopped?
I wrote a small test project to highlight the problem. There consumer is
started up and tore down in a loop and starts failing after some iterations.
https://github.com/ivelichko/kafka-consumer-reconnect-test
--
Best regards,
Igor Velichko
