Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1341#discussion_r45461644
--- Diff:
flink-streaming-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java
---
@@ -374,12 +372,33 @@ public void open(Configuration parameters) throws
Exception {
// no restore request. Let the offset handler take care
of the initial offset seeking
offsetHandler.seekFetcherToInitialOffsets(subscribedPartitions, fetcher);
}
+
+
}
@Override
public void run(SourceContext<T> sourceContext) throws Exception {
if (fetcher != null) {
+ // For non-checkpointed sources, a thread which
periodically commits the current offset into ZK.
+ PeriodicOffsetCommitter offsetCommitter = null;
+
+ // check whether we need to start the periodic
checkpoint committer
+ StreamingRuntimeContext streamingRuntimeContext =
(StreamingRuntimeContext) getRuntimeContext();
+ if (!streamingRuntimeContext.isCheckpointingEnabled()) {
+ // we use Kafka's own configuration parameter
key for this.
+ // Note that the default configuration value in
Kafka is 60 * 1000, so we use the
+ // same here.
+ long commitInterval =
Long.valueOf(props.getProperty("auto.commit.interval.ms", "60000"));
+ offsetCommitter = new
PeriodicOffsetCommitter(commitInterval, this);
+ offsetCommitter.start();
+ LOG.info("Starting periodic offset committer,
with commit interval of {}ms", commitInterval);
+ }
+
fetcher.run(sourceContext, valueDeserializer,
lastOffsets);
+
+ if(offsetCommitter != null) {
--- End diff --
Style space ;-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
