[
https://issues.apache.org/jira/browse/CAMEL-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-16105:
--------------------------------
Summary: camel-kafka - AllowManualCommit does not ensure full control of
the commit flows (was: AllowManualCommit does not ensure full control of the
commit flows)
> camel-kafka - AllowManualCommit does not ensure full control of the commit
> flows
> --------------------------------------------------------------------------------
>
> Key: CAMEL-16105
> URL: https://issues.apache.org/jira/browse/CAMEL-16105
> Project: Camel
> Issue Type: Improvement
> Components: camel-kafka
> Affects Versions: 3.7.1
> Reporter: Dmitrijs Jesilevskis
> Priority: Major
> Fix For: 3.10.0
>
>
> *Scenario*
> My scenario implies handling business-critical data through Kafka, before
> writing it to the sink system. To ensure maximum possible delivery guarantee
> and control, I choose to use Kafka component with _allowManualCommit_ option,
> as per
> [documentation|https://camel.apache.org/components/latest/kafka-component.html#_using_manual_commit_with_kafka_consumer].
> This is required to be absolutely sure that message is delivered to the sink
> system, before committing the state and move to the next item or batch of
> items.
> *Problem*
> When I use _allowManualCommit_ option but do not (intentionally) include any
> steps to actually execute the commit, the Consumer Group offset is still
> being committed, when I restart integrations.
> The commit is executed due to triggering of _onPartitionsRevoked_ event
> handler in [KafkaConsumer.java,
> L473|https://github.com/apache/camel/blob/0dcbe44d35545a4fc29e5e7b0e9e687d564cbf28/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java#L473].
> This happens due to graceful termination of the running container.
> * Line 485, initiates _commitOffset()_ method with _forceCommit_ parameter
> set to true;
> * Line 453 host a condition that executes offset persistence to custom
> offsetRepository when _allowManualCommit_ is disabled, and offsetRepository
> is specified - this condition is skipped, as in my scenario
> _allowManualCommit_ is enabled
> * Line 458 host an alternative condition to execute a commit when
> _forceCommit_ parameter is _true_, and it is indeed true, as the method is
> invoked by _onPartitionsRevoked_.
> The hidden side-effect of the scenario is that offset is committed, despite
> being ordered. This might potentially lead to the loss of messages, as there
> is no guarantee that messages were processed successfully.
> *Possible solution*
> * Execute _forceCommit_ section only when _allowManualCommit_ is disabled.
> In case of a manual commit, the developer should be in charge in all cases.
> *Extra*
> It must be also ensured that when custom offsetRepository is used, state is
> never persisted to both custom repository and standard consumer group.
> Currently that what happens when offsetRepository is paired with enabled
> _allowManualCommit_, through the above described scenario as well.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
