Viswa Ramamoorthy created CAMEL-13338:
-----------------------------------------
Summary: ConsumerRebalanceListener is not registred when
topicIsPattern if turned off cauing message loss or too many duplicates
Key: CAMEL-13338
URL: https://issues.apache.org/jira/browse/CAMEL-13338
Project: Camel
Issue Type: Bug
Components: camel-kafka
Affects Versions: 2.23.0
Reporter: Viswa Ramamoorthy
Not registering ConsumerRebalanceListener when topicIsPattern turned off,
causes either too many duplicate messages getting delivered to message
processing layer or causes message loss depending on autoOffsetReset flag value.
Issue is noticed with below test scenario
Topic: e.g. TestMessage
Number of partitions: 25 (this is not completely relevant to issue)
ConsumerUri:
kafka:<kafka-broker-uris>?topic=TestMessage&groupId=TestMessage_GROUP&consumersCount=5&autoCommitEnable=false&offsetRepository=#<DB-state-repository>
Number processes running consumers: 2 or more; e.g. two instances of Docker
containers listening from same topic each with 5 consumer threads
Please note that offset management is done in DB with an implementation of
Camel's StateRepository interface to eliminate too duplicate messages arriving
at the message processor (at most one message processing needed for my use case)
When a second process instance (say second container) brought up, since
ConsumerRebalanceListener is not registered, partitions start processing at
earliest offsets if autoOffsetReset is set to earliest as expected. In this
case, too many duplicate messages arrive at the message processing layer. If
autoOffsetReset is set to latest, message loss occurs for certain partitions.
If autoOffsetReset is set to none, no valid offset exception thrown by broker
Issue would be noticed consistently when simulating slow message processing for
the above test scenario
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
