[
https://issues.apache.org/jira/browse/BEAM-6902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik reassigned BEAM-6902:
-------------------------------
Assignee: Mark Norkin
> Beam model contract for finalization of CheckpointMark's
> ------------------------------------------------------------
>
> Key: BEAM-6902
> URL: https://issues.apache.org/jira/browse/BEAM-6902
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, io-java-kafka, runner-core, runner-dataflow,
> sdk-java-core
> Reporter: Mark Norkin
> Assignee: Mark Norkin
> Priority: Major
> Labels: documentation
> Attachments: BEAM-6902 diagrams.pdf,
> beam-kafka-io-commit-model-examples-master.zip
>
>
> Question: What is the contract in Beam model for when checkpoint marks shall
> be finalized, is there any ?
> I'm working on pipeline that reads messages from Kafka using KafkaIO, and I'm
> looking at _commitOffsetsInFinalize()_ option, and KafkaCheckpointMark class.
> I want to achieve at-least-once message delivery semantics and want to be
> sure that offsets committed to Kafka after they are written to some sink.
> Looking at interface of
> [CheckpointMark|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/UnboundedSource.CheckpointMark.html]
> it's not clear when finalization shall be expected to happen.
> Is it runner dependent, what to expect when executing on _DataflowRunner_ ?
> And reading KafkaIO.Read javadoc on _commitOffsetsInFinalize_
> _[https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#commitOffsetsInFinalize--]_
>
> also doesn't bring clarity to my understanding, particularly the phrase
> {quote}But it does not provide *_hard processing guarantees_*
> {quote}
> What exactly are hard processing guarantees ?
> Can I ask, please for documentation improvement in respect of
> _CheckpointMark_ and _commitOffsetsInFinalize_.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
