[
https://issues.apache.org/jira/browse/FLINK-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15045131#comment-15045131
]
ASF GitHub Bot commented on FLINK-3102:
---------------------------------------
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1437#discussion_r46837638
--- Diff:
flink-streaming-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer.java
---
@@ -563,37 +559,44 @@ public void notifyCheckpointComplete(long
checkpointId) throws Exception {
LOG.warn("Received confirmation for
unknown checkpoint id {}", checkpointId);
return;
}
-
- checkpointOffsets = (long[])
pendingCheckpoints.remove(posInMap);
+
+ //noinspection unchecked
+ checkpointOffsets =
(HashMap<KafkaTopicPartition, Long>) pendingCheckpoints.remove(posInMap);
+
// remove older checkpoints in map
for (int i = 0; i < posInMap; i++) {
pendingCheckpoints.remove(0);
}
}
-
- if (LOG.isInfoEnabled()) {
- LOG.info("Committing offsets {} to offset
store: {}", Arrays.toString(checkpointOffsets), offsetStore);
+ if(checkpointOffsets.size() == 0) {
+ LOG.info("Checkpoint state was empty.");
+ return;
}
// build the map of (topic,partition) -> committed
offset
- Map<TopicPartition, Long> offsetsToCommit = new
HashMap<>();
- for (TopicPartition tp : subscribedPartitions) {
-
- int partition = tp.partition();
- long offset = checkpointOffsets[partition];
- long lastCommitted = commitedOffsets[partition];
-
+ Map<KafkaTopicPartition, Long> offsetsToCommit = new
HashMap<>();
+ for (KafkaTopicPartition tp : subscribedPartitions) {
+ Long offset = checkpointOffsets.get(tp);
+ Long lastCommitted = commitedOffsets.get(tp);
+ if(lastCommitted == null) {
+ lastCommitted = OFFSET_NOT_SET;
+ }
if (offset != OFFSET_NOT_SET) {
if (offset > lastCommitted) {
offsetsToCommit.put(tp, offset);
- LOG.debug("Committing offset {}
for partition {}", offset, partition);
+ //noinspection unchecked
+ commitedOffsets.put(tp, offset);
+ LOG.debug("Committing offset {}
for {}", offset, tp.toString());
}
else {
- LOG.debug("Ignoring offset {}
for partition {} because it is already committed", offset, partition);
+ LOG.debug("Ignoring offset {}
for partition {} because it is already committed", offset, tp);
}
}
}
+ if(LOG.isInfoEnabled() && offsetsToCommit.size() > 0) {
--- End diff --
Is this a good thing on INFO level? Will be a lot of contents in the logs,
especially when running frequent checkpoints. I would vote to log this on
DEBUG...
> Allow reading from multiple topics with one FlinkKafkaConsumer
> --------------------------------------------------------------
>
> Key: FLINK-3102
> URL: https://issues.apache.org/jira/browse/FLINK-3102
> Project: Flink
> Issue Type: New Feature
> Components: Kafka Connector
> Reporter: Robert Metzger
> Assignee: Robert Metzger
>
> Currently, a Kafka consumer allows to read from only one topic.
> For cases where multiple topics contain messages with the same schema, it is
> useful to allow to subscribe to many topics using one FlinkKafkaConsumer
> instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
