Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/3746#discussion_r115996970
--- Diff:
flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/AbstractFetcher.java
---
@@ -106,31 +139,69 @@ protected AbstractFetcher(
}
}
- // create our partition state according to the
timestamp/watermark mode
- this.subscribedPartitionStates =
initializeSubscribedPartitionStates(
- assignedPartitionsWithInitialOffsets,
+ this.unassignedPartitionsQueue = new ClosableBlockingQueue<>();
+
+ // initialize subscribed partition states with seed partitions
+ this.subscribedPartitionStates = createPartitionStateHolders(
+ seedPartitionsWithInitialOffsets,
timestampWatermarkMode,
- watermarksPeriodic, watermarksPunctuated,
+ watermarksPeriodic,
+ watermarksPunctuated,
userCodeClassLoader);
- // check that all partition states have a defined offset
+ // check that all seed partition states have a defined offset
for (KafkaTopicPartitionState partitionState :
subscribedPartitionStates) {
if (!partitionState.isOffsetDefined()) {
- throw new IllegalArgumentException("The fetcher
was assigned partitions with undefined initial offsets.");
+ throw new IllegalArgumentException("The fetcher
was assigned seed partitions with undefined initial offsets.");
}
}
-
+
+ // all seed partitions are not assigned yet, so should be added
to the unassigned partitions queue
+ for (KafkaTopicPartitionState<KPH> partition :
subscribedPartitionStates) {
+ unassignedPartitionsQueue.add(partition);
+ }
+
// if we have periodic watermarks, kick off the interval
scheduler
if (timestampWatermarkMode == PERIODIC_WATERMARKS) {
- KafkaTopicPartitionStateWithPeriodicWatermarks<?, ?>[]
parts =
-
(KafkaTopicPartitionStateWithPeriodicWatermarks<?, ?>[])
subscribedPartitionStates;
-
- PeriodicWatermarkEmitter periodicEmitter =
- new PeriodicWatermarkEmitter(parts,
sourceContext, processingTimeProvider, autoWatermarkInterval);
+ @SuppressWarnings("unchecked")
+ PeriodicWatermarkEmitter periodicEmitter = new
PeriodicWatermarkEmitter(
--- End diff --
Could add the generic parameter `KPH` here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
