[
https://issues.apache.org/jira/browse/BEAM-6902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819121#comment-16819121
]
Mark Norkin commented on BEAM-6902:
-----------------------------------
Also I've attached sample
[project|[https://github.com/marknorkin/beam-kafka-io-commit-model-examples]]
to this ticket with how currently autocommit model is used in our project and
how we are seeing usage of manual commit functionality in Apache Beam Kafka IO
as we haven't found guidance on how to do it in Apache Beam documentation.
Would be great to add some guidance (i.e. "do this to use manual commit model")
in documentation as well to see if this configuration is correct.
> Beam model contract for finalization of CheckpointMark's
> ------------------------------------------------------------
>
> Key: BEAM-6902
> URL: https://issues.apache.org/jira/browse/BEAM-6902
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, io-java-kafka, runner-core, runner-dataflow,
> sdk-java-core
> Reporter: Mark Norkin
> Priority: Major
> Labels: documentation
> Attachments: beam-kafka-io-commit-model-examples-master.zip
>
>
> Question: What is the contract in Beam model for when checkpoint marks shall
> be finalized, is there any ?
> I'm working on pipeline that reads messages from Kafka using KafkaIO, and I'm
> looking at _commitOffsetsInFinalize()_ option, and KafkaCheckpointMark class.
> I want to achieve at-least-once message delivery semantics and want to be
> sure that offsets committed to Kafka after they are written to some sink.
> Looking at interface of
> [CheckpointMark|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/UnboundedSource.CheckpointMark.html]
> it's not clear when finalization shall be expected to happen.
> Is it runner dependent, what to expect when executing on _DataflowRunner_ ?
> And reading KafkaIO.Read javadoc on _commitOffsetsInFinalize_
> _[https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#commitOffsetsInFinalize--]_
>
> also doesn't bring clarity to my understanding, particularly the phrase
> {quote}But it does not provide *_hard processing guarantees_*
> {quote}
> What exactly are hard processing guarantees ?
> Can I ask, please for documentation improvement in respect of
> _CheckpointMark_ and _commitOffsetsInFinalize_.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
