Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1341#discussion_r44532032
--- Diff:
flink-streaming-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java
---
@@ -374,12 +376,32 @@ public void open(Configuration parameters) throws
Exception {
// no restore request. Let the offset handler take care
of the initial offset seeking
offsetHandler.seekFetcherToInitialOffsets(subscribedPartitions, fetcher);
}
+
+ // check whether we need to start the periodic checkpoint
committer
+ StreamingRuntimeContext streamingRuntimeContext =
(StreamingRuntimeContext) getRuntimeContext();
+ if(!streamingRuntimeContext.isCheckpointingEnabled()) {
+ // we use Kafka's own configuration parameter key for
this.
+ // Note that the default configuration value in Kafka
is 60 * 1000, so we use the
+ // same here.
+ long commitInterval =
Long.valueOf(props.getProperty("auto.commit.interval.ms", "60000"));
+ offsetCommitter = new
PeriodicOffsetCommitter(commitInterval, this);
+ offsetCommitter.start();
+ LOG.info("Starting periodic offset committer, with
commit interval of {}ms", commitInterval);
+ }
}
@Override
public void run(SourceContext<T> sourceContext) throws Exception {
if (fetcher != null) {
- fetcher.run(sourceContext, valueDeserializer,
lastOffsets);
+ // by default, we use the checkpoint lock for updating
the state
+ Object stateUpdateLock =
sourceContext.getCheckpointLock();
+
+ // if checkpointing is disabled, we use the checkpoint
committer's lock object
+ StreamingRuntimeContext streamingRuntimeContext =
(StreamingRuntimeContext) getRuntimeContext();
+ if(!streamingRuntimeContext.isCheckpointingEnabled()) {
+ stateUpdateLock =
offsetCommitter.getStateUpdateLock();
--- End diff --
This seems funky. Swapping the lock. I think it breaks the exactly-once
guarantees.
Also, why checking the runtime context, why not simply `if (offsetCommitter
!= null)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
