[
https://issues.apache.org/jira/browse/FLINK-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972243#comment-16972243
]
chaiyongqiang edited comment on FLINK-13052 at 11/12/19 10:08 AM:
------------------------------------------------------------------
There're some limits when we support multi source topics in
kafkaTableSourceSinkFactoryBase. It's like :
When we assign the startup mode as specific-offsets, it's difficult to assign
the start-offset for each topic partition because of the way to set the
connector configs. As the following codes show, when we assign several topics
like MyTable, MyTable1,MyTable2 , it's difficult to distinguish the partition
and offset belong to wich topic.
{code:java}
final Map<String, String> props2 = new HashMap<>();
props2.put("connector.property-version", "1");
props2.put("connector.type", "kafka");
props2.put("connector.version", "0.11");
props2.put("connector.topic", "MyTable");
props2.put("connector.startup-mode", "specific-offsets");
props2.put("connector.specific-offsets.0.partition", "0");
props2.put("connector.specific-offsets.0.offset", "42");
props2.put("connector.specific-offsets.1.partition", "1");
{code}
But fortunately， setting a fixed topic-partition start offset when we start an
app with table API is rare.
My solution is supporting multi-topic for other kind of start-up mode like
GROUP_OFFSETS, EARLIEST,LATEST, for SPECIFIC_OFFSETS mode when using Table
API like kafkaTableSourceSinkFactoryBase with several topics we use the
GROUP_OFFSETS instead.
Does someone have a better idea ? Please let me know. Many thanks
was (Author: chaiyq):
There're some limits when we support multi source topics in
kafkaTableSourceSinkFactoryBase. It's like :
When we assign the startup mode as specific-offsets, it's difficult to assign
the start-offset for each topic partition because of the way to set the
connector configs. As the following codes show, when we assign several topics
like MyTable, MyTable1,MyTable2 , it's difficult to distinguish the partition
and offset belong to wich topic.
{code:java}
final Map<String, String> props2 = new HashMap<>();
props2.put("connector.property-version", "1");
props2.put("connector.type", "kafka");
props2.put("connector.version", "0.11");
props2.put("connector.topic", "MyTable");
props2.put("connector.startup-mode", "specific-offsets");
props2.put("connector.specific-offsets.0.partition", "0");
props2.put("connector.specific-offsets.0.offset", "42");
props2.put("connector.specific-offsets.1.partition", "1");
{code}
But fortunately， setting a fixed topic-partition start offset when we start an
app with table API is rare. For now, i suggest ignoring this situation. My
solution is supporting multi-topic for other kind of start-up mode like
GROUP_OFFSETS, EARLIEST,LATEST, for SPECIFIC_OFFSETS mode when using Table
API like kafkaTableSourceSinkFactoryBase with several topics we use the
GROUP_OFFSETS instead.
> Supporting multi-topic when using
> kafkaTableSourceSinkFactoryBase.createStreamTableSource
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-13052
> URL: https://issues.apache.org/jira/browse/FLINK-13052
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Table SQL / API
> Affects Versions: 1.8.0
> Reporter: chaiyongqiang
> Priority: Critical
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
