[
https://issues.apache.org/jira/browse/FLINK-37864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18051945#comment-18051945
]
Bowen Li commented on FLINK-37864:
----------------------------------
just saw this ticket. Close as dup of
https://issues.apache.org/jira/browse/FLINK-38876
> Extend DynamicKafkaSource to allow for cluster-specific OffsetsInitializer
> ---------------------------------------------------------------------------
>
> Key: FLINK-37864
> URL: https://issues.apache.org/jira/browse/FLINK-37864
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 2.0.0
> Reporter: Joey Pereira
> Priority: Minor
>
> At the moment, the DynamicKafkaSource cannot have OffsetsInitializers that
> vary per-cluster but only per-topic and partition provided the
> parameterization of {{org.apache.kafka.common.TopicPartition}}.
> For more complex use-cases with consumption from multiple Kafka sources, it
> would be ideal to allow for offset customization per-cluster.
> One way this can be added while keeping things simple would be to:
> - Add {{setStartingOffsets}} and {{setBounded}} with the arguments
> {{Map<String, OffsetsInitializer>}}
> - Plumb that map as a seperate property through to the
> {{DynamicKafkaSourceEnumerator}} and select the appropriate one during
> {{createEnumeratorWithAssignedTopicPartitions}}
> Additionally, we'll want to:
> - Include adjustments to a Python interface FLINK-37863
> - Adjust the documentation
> (cc [~shuyichen])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
