Zhilong Hong created FLINK-21326:
------------------------------------
Summary: Optimize building topology when initializing
ExecutionGraph
Key: FLINK-21326
URL: https://issues.apache.org/jira/browse/FLINK-21326
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Reporter: Zhilong Hong
Fix For: 1.13.0
The main idea of optimizing the procedure of building topology is to put all
the vertices that consumed the same result partitions into one group, and put
all the result partitions that have the same consumer vertices into one
consumer group. The corresponding data structure is {{ConsumedPartitionGroup}}
and {{ConsumerVertexGroup}}. {{EdgeManager}} is used to store the relationship
between the groups. The procedure of creating {{ExecutionEdge}} is replaced
with building {{EdgeManager}}.
With these improvements, the complexity of building topology in ExecutionGraph
decreases from O(N^2) to O(N).
Furthermore, {{ExecutionEdge}} and all its related calls are replaced with
{{ConsumedPartitionGroup}} and {{ConsumerVertexGroup}}.
The detailed design doc is located at:
https://docs.google.com/document/d/1OjGAyJ9Z6KsxcMtBHr6vbbrwP9xye7CdCtrLvf8dFYw/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
