[
https://issues.apache.org/jira/browse/CAMEL-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800652#comment-16800652
]
Viswa Ramamoorthy commented on CAMEL-13338:
-------------------------------------------
[~davsclaus] Thanks for merging PR. I see that you have merged fix into 2.23.x
branch as well. Do you know when 2.23.2 would be released?
> ConsumerRebalanceListener is not registered when topicIsPattern is turned
> off. Causing message loss or too many duplicates
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-13338
> URL: https://issues.apache.org/jira/browse/CAMEL-13338
> Project: Camel
> Issue Type: Bug
> Components: camel-kafka
> Affects Versions: 2.23.0
> Reporter: Viswa Ramamoorthy
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 3.0.0, 2.23.2, 2.24.0, 2.22.4, 3.0.0-M2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Not registering ConsumerRebalanceListener when topicIsPattern turned off,
> causes either too many duplicate messages getting delivered to message
> processing layer or causes message loss depending on autoOffsetReset flag
> value.
>
> Issue is noticed with below test scenario
> Topic: e.g. TestMessage
> Number of partitions: 25 (this is not completely relevant to issue)
> ConsumerUri:
> kafka:<kafka-broker-uris>?topic=TestMessage&groupId=TestMessage_GROUP&consumersCount=5&autoCommitEnable=false&offsetRepository=#<DB-state-repository>
> Number processes running consumers: 2 or more; e.g. two instances of Docker
> containers listening from same topic each with 5 consumer threads
>
> Please note that offset management is done in DB with an implementation of
> Camel's StateRepository interface to eliminate too duplicate messages
> arriving at the message processor (at most one message processing needed for
> my use case)
>
> When a second process instance (say second container) brought up, since
> ConsumerRebalanceListener is not registered, partitions start processing at
> earliest offsets if autoOffsetReset is set to earliest as expected. In this
> case, too many duplicate messages arrive at the message processing layer. If
> autoOffsetReset is set to latest, message loss occurs for certain partitions.
> If autoOffsetReset is set to none, no valid offset exception thrown by broker
>
> Issue would be noticed consistently when simulating slow message processing
> for the above test scenario
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
