[
https://issues.apache.org/jira/browse/FLINK-18690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168535#comment-17168535
]
Zhu Zhu edited comment on FLINK-18690 at 7/31/20, 9:11 AM:
-----------------------------------------------------------
Suboptimal case 1:
The grouping of execution vertices from brother job vertices are possible
suboptimal because it does not take connections to consumers vertices into
account.
Example - case1.png:
Topology:
A(parallelism=4) -- forward --> C(parallelism=4), B(parallelism=2) -- rescale
--> C
Execution edges:
A1->C1, A2->C2, A3->C3, A4->C4; B1->C1, B1->C2, B2->C3,B2->C4
Optimal grouping:
{A1,B1,C1}{A2,C2}{A3,B2,C3}{A4,C4} -> there would be 2 remote edges
Current grouping result:
{A1,B1,C1}{A2,B2,C2}{A3,C3}{A4,C4} -> there would be 3 remote edges
Possible solution:
Change the rescale edge connection pattern
e.g. execution edge from B to C in this case would be: B1->C1, B1->C3,
B2->C2,B2->C4
And put execution vertices with same index into the same slot sharing group.
was (Author: zhuzh):
Suboptimal case 1:
The grouping of execution vertices from brother job vertices are possible
suboptimal because it does not take connections to consumers vertices into
account.
Example - case1.png:
Topology:
A(parallelism=4) -- forward --> C(parallelism=4), B(parallelism=2) -- rescale
--> C
Execution edges:
A1->C1, A2->C2, A3->C3, A4->C4; B1->C1, B1->C2, B2->C3,B2->C4
Optimal grouping:
{A1,B1,C1}{A2,C2}{A3,B2,C3}{A4,C4} -> there would be 2 remote edges
Current grouping result:
{A1,B1,C1}{A2,B2,C2}{A3,C3}{A4,C4} -> there would be 3 remote edges
> Implement LocalInputPreferredSlotSharingStrategy
> ------------------------------------------------
>
> Key: FLINK-18690
> URL: https://issues.apache.org/jira/browse/FLINK-18690
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Andrey Zagrebin
> Assignee: Zhu Zhu
> Priority: Major
> Labels: pull-request-available
> Attachments: case1.png
>
>
> Implement ExecutionSlotSharingGroup, SlotSharingStrategy and default
> LocalInputPreferredSlotSharingStrategy.
> The default strategy would be LocalInputPreferredSlotSharingStrategy. It will
> try to reduce remote data exchanges. Subtasks, which are connected and belong
> to the same SlotSharingGroup, tend to be put in the same
> ExecutionSlotSharingGroup.
> See [design
> doc|https://docs.google.com/document/d/10CbCVDBJWafaFOovIXrR8nAr2BnZ_RAGFtUeTHiJplw/edit#heading=h.t4vfmm4atqoy]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
