Dmitrijs Jesilevskis created CAMEL-16105:
--------------------------------------------
Summary: AllowManualCommit does not ensure full control of the
commit flows, possible loss of events
Key: CAMEL-16105
URL: https://issues.apache.org/jira/browse/CAMEL-16105
Project: Camel
Issue Type: Bug
Components: camel-kafka
Affects Versions: 3.7.1
Reporter: Dmitrijs Jesilevskis
*Scenario*
My scenario implies handling business-critical data through Kafka, before
writing it to the sink system. To ensure maximum possible delivery guarantee
and control, I choose to use Kafka component with _allowManualCommit_ option,
as per
[documentation|https://camel.apache.org/components/latest/kafka-component.html#_using_manual_commit_with_kafka_consumer].
This is required to be absolutely sure that message is delivered to the sink
system, before committing the state and move to the next item or batch of items.
*Problem*
When I use _allowManualCommit_ option but do not (intentionally) include any
steps to actually execute the commit, the Consumer Group offset is still being
committed, when I restart integrations.
The commit is executed due to triggering of _onPartitionsRevoked_ event
handler in [KafkaConsumer.java,
L473|https://github.com/apache/camel/blob/0dcbe44d35545a4fc29e5e7b0e9e687d564cbf28/components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java#L473].
This happens due to graceful termination of the running container.
* Line 485, initiates _commitOffset()_ method with _forceCommit_ parameter set
to true;
* Line 453 host a condition that executes offset persistence to custom
offsetRepository when _allowManualCommit_ is disabled, and offsetRepository is
specified - this condition is skipped, as in my scenario _allowManualCommit_ is
enabled
* Line 458 host an alternative condition to execute a commit when
_forceCommit_ parameter is _true_, and it is indeed true, as the method is
invoked by _onPartitionsRevoked_.
The hidden side-effect of the scenario is that offset is committed, despite
being ordered. This might potentially lead to the loss of messages, as there is
no guarantee that messages were processed successfully.
*Possible solution*
* Execute _forceCommit_ section only when _allowManualCommit_ is disabled. In
case of a manual commit, the developer should be in charge in all cases.
*Extra*
It must be also ensured that when custom offsetRepository is used, state is
never persisted to both custom repository and standard consumer group.
Currently that what happens when offsetRepository is paired with enabled
_allowManualCommit_, through the above described scenario as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
