[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-18295:
----------------------------
Description:
Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
That's why the consumers of an {{IntermediateResultPartition}} is in the form
of {{List<List<ExecutionEdge>>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
{{IntermediateResultPartition}} can be consumed by one only
{{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
partition consumers to contain a single list.
Given that there is no plan yet to support multiple consumer {{JobEdge}} of one
{{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to have
one only consumer {{JobEdge}}. Thus the scheduler can get rid of these hack
logics.
was:
Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
That's why the consumers of an {{IntermediateResultPartition}} is in the form
of {{List<List<ExecutionEdge>>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
{{IntermediateResultPartition}} can be consumed by one only
{{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
partition consumers to contain a single list.
Given that there is no plan yet to support multiple consumer {{JobEdge}}s of
one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
hack logics.
> Remove the hack logics of result consumers
> ------------------------------------------
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
> Reporter: Zhu Zhu
> Priority: Major
>
> Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
> That's why the consumers of an {{IntermediateResultPartition}} is in the form
> of {{List<List<ExecutionEdge>>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> {{IntermediateResultPartition}} can be consumed by one only
> {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> Given that there is no plan yet to support multiple consumer {{JobEdge}} of
> one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
> have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
> hack logics.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
