jsun98 commented on a change in pull request #6496: Support kafka transactional
topics (#5404)
URL: https://github.com/apache/incubator-druid/pull/6496#discussion_r253278888
##########
File path:
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/IncrementalPublishingKafkaIndexTaskRunner.java
##########
@@ -576,15 +561,19 @@ public void onFailure(Throwable t)
nextOffsets.put(record.partition(), record.offset() + 1);
}
-
- if
(nextOffsets.get(record.partition()).equals(endOffsets.get(record.partition()))
+ if (nextOffsets.get(record.partition()) >=
endOffsets.get(record.partition())
&& assignment.remove(record.partition())) {
log.info("Finished reading topic[%s], partition[%,d].",
record.topic(), record.partition());
KafkaIndexTask.assignPartitions(consumer, topic, assignment);
stillReading = !assignment.isEmpty();
}
}
-
+ if (nextOffsets.get(currentPartition) != null
Review comment:
Sorry for the late reply. If I understand correctly, the line
`nextOffsets.put(currentPartition, endOffsets.get(currentPartition));` is
supposed to update the `nextOffsets` of the partitions with polled offsets that
have exceeded the partition's `endOffsets`. But what if, in the list of polled
`records`, 2 or more partitions have offsets that exceed their respective
endOffsets. Then since `nextOffsets.put(currentPartition,
endOffsets.get(currentPartition));` is outside the loop, it would only update
the `nextOffsets` of 1 such partition. I remember talking to @jihoonson about
this, he may know more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
