[
https://issues.apache.org/jira/browse/FLINK-21817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306603#comment-17306603
]
Stephan Ewen commented on FLINK-21817:
--------------------------------------
I agree that this is a bug in the Kafka Enumerator.
I think we need to make sure users understand that the Enumerator should not in
any case make any assumption about what is going on at the readers after a
failure or restore. That is an anti-pattern in my experience, doing bookkeeping
twice and trying to keep it in sync. It will very often break at some points.
A similar bug was FLINK-21452, and the solution was to drop the bookkeeping on
the enumerator side, and rely on reader registrations to learn about the
current state.
Following a similar argument, I would suggest to go with variant (1) here.
> FLIP-27 Source might break subtask and split assignment upon rescale
> --------------------------------------------------------------------
>
> Key: FLINK-21817
> URL: https://issues.apache.org/jira/browse/FLINK-21817
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
> Affects Versions: 1.13.0
> Reporter: Kezhu Wang
> Priority: Major
>
> On restoring, splits are add back directly to {{SourceReader}} in
> {{SourceOperator}}. In no rescaled restoring, bindings between splits and
> subtasks are preserved due to no repartition in
> {{RoundRobinOperatorStateRepartitioner}}. But in rescaled restoring, these
> operator states will be redistributed cross existing subtasks. This might
> break possible assignment from {{SourceEnumerator}}.
> Given {{KafkaSource}} as an example, the partition to subtask assignment is
> decided by {{KafkaSourceEnumerator.getSplitOwner}}. The mappings will break
> after rescaling.
> I pushed [a test
> case|https://github.com/kezhuw/flink/commit/9dc13cd9d7355d468a6ac8aae1b14b3a267581b6#diff-ad6e86c3757199ac3247687a71f9c34ee67b9ac743ae88a9f660950f27bec6eeR238]
> using {{KafkaSource}} for evaluation.
> I think it requires api addition to solve in generic and configurable way.
> Is it a valid issue ? I am not that sure.
> cc [~jqin] [~sewen]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
