[
https://issues.apache.org/jira/browse/CAMEL-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viswa Ramamoorthy updated CAMEL-13338:
--------------------------------------
Summary: ConsumerRebalanceListener is not registered when topicIsPattern is
turned off. Cauing message loss or too many duplicates (was:
ConsumerRebalanceListener is not registred when topicIsPattern if turned off
cauing message loss or too many duplicates)
> ConsumerRebalanceListener is not registered when topicIsPattern is turned
> off. Cauing message loss or too many duplicates
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-13338
> URL: https://issues.apache.org/jira/browse/CAMEL-13338
> Project: Camel
> Issue Type: Bug
> Components: camel-kafka
> Affects Versions: 2.23.0
> Reporter: Viswa Ramamoorthy
> Priority: Major
>
> Not registering ConsumerRebalanceListener when topicIsPattern turned off,
> causes either too many duplicate messages getting delivered to message
> processing layer or causes message loss depending on autoOffsetReset flag
> value.
>
> Issue is noticed with below test scenario
> Topic: e.g. TestMessage
> Number of partitions: 25 (this is not completely relevant to issue)
> ConsumerUri:
> kafka:<kafka-broker-uris>?topic=TestMessage&groupId=TestMessage_GROUP&consumersCount=5&autoCommitEnable=false&offsetRepository=#<DB-state-repository>
> Number processes running consumers: 2 or more; e.g. two instances of Docker
> containers listening from same topic each with 5 consumer threads
>
> Please note that offset management is done in DB with an implementation of
> Camel's StateRepository interface to eliminate too duplicate messages
> arriving at the message processor (at most one message processing needed for
> my use case)
>
> When a second process instance (say second container) brought up, since
> ConsumerRebalanceListener is not registered, partitions start processing at
> earliest offsets if autoOffsetReset is set to earliest as expected. In this
> case, too many duplicate messages arrive at the message processing layer. If
> autoOffsetReset is set to latest, message loss occurs for certain partitions.
> If autoOffsetReset is set to none, no valid offset exception thrown by broker
>
> Issue would be noticed consistently when simulating slow message processing
> for the above test scenario
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
