[
https://issues.apache.org/jira/browse/FLINK-25034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu reassigned FLINK-25034:
-------------------------------
Assignee: Lijie Wang
> Support flexible number of subpartitions in IntermediateResultPartition
> -----------------------------------------------------------------------
>
> Key: FLINK-25034
> URL: https://issues.apache.org/jira/browse/FLINK-25034
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Lijie Wang
> Assignee: Lijie Wang
> Priority: Major
>
> Currently, when a task is deployed, it needs to know the parallelism of its
> consumer job vertex. This is because the consumer vertex parallelism is
> needed to decide the _numberOfSubpartitions_ of _PartitionDescriptor_ which
> is part of the {_}ResultPartitionDeploymentDescriptor{_}. The reason behind
> that is, at the moment, for one result partition, different subpartitions
> serve different consumer execution vertices. More specifically, one consumer
> execution vertex only consumes data from subpartition with the same index.
> Considering a dynamic graph, the parallelism of a job vertex may not have
> been decided when its upstream vertices are deployed. To enable Flink to work
> in this case, we need a way to allow an execution vertex to run without
> knowing the parallelism of its consumer job vertices. One basic idea is to
> enable multiple subpartitions in one result partition to serve the same
> consumer execution vertex.
> To achieve this goal, we can set the number of subpartitions to be the *max
> parallelism* of the consumer job vertex. When the consumer vertex is
> deployed, it should be assigned with a subpartition range to consume.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
