[
https://issues.apache.org/jira/browse/FLINK-10806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889932#comment-16889932
]
Jiayi Liao commented on FLINK-10806:
------------------------------------
[~becket_qin] Thanks for pointing this out. What you described is more like a
migration of state(migrate T1 state into Pipeline B), which reminds me that the
savepoint connector that community is building now, can't support the state
migration of kafka(maybe you can help build it ^_^).
I've already support the simple strategy(earliest/latest) in our own flink
branch. But your concerns make sense so I can't give a conclusion now. Maybe we
can wait until we get more user cases and scenarios?
> Support multiple consuming offsets when discovering a new topic
> ---------------------------------------------------------------
>
> Key: FLINK-10806
> URL: https://issues.apache.org/jira/browse/FLINK-10806
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 1.6.2, 1.8.1
> Reporter: Jiayi Liao
> Assignee: Jiayi Liao
> Priority: Major
>
> In KafkaConsumerBase, we discover the TopicPartitions and compare them with
> the restoredState. It's reasonable when a topic's partitions scaled. However,
> if we add a new topic which has too much data and restore the Flink program,
> the data of the new topic will be consumed from the start, which may not be
> what we want. I think this should be an option for developers.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
