[
https://issues.apache.org/jira/browse/FLINK-31757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711848#comment-17711848
]
Weihua Hu commented on FLINK-31757:
-----------------------------------
Hi [~RocMarshal] , thanks for reporting this.
For now, you could set different slot sharing group for tasks to prevent too
may tasks in same slots
But, we, in bytedance, also face this unbalanced issue.
And we did some work on our internal custom flink.
We did two things to spread the tasks evenly.
# enrich LocalInputPreferredSlotSharingStrategySplit to assign tasks to
SlotSharingGroup evenly.
I think that could solve the problem you described. In this case, 200 tasks
will be assigned to 100 slots, and each slot should have 2 tasks.
# Schedule tasks (slots) to TaskManager evenly. This is for slots with
different task numbers, such as we have 10 source and 20 sink operators, so we
need 20 slots, and there are 10 slots with 2 sub-tasks and 10 slots with 1
sub-task. If we have 10 task managers with 20 slots in total, we should
schedule a slot with 2 sub-tasks, and a slot with 1 sub-task, for one task
manager.
Do you think this would be valuable to the flink community? I would be glad to
contribute it. [~Weijie Guo] [~martijnvisser]
> Optimize Flink un-balanced tasks scheduling
> -------------------------------------------
>
> Key: FLINK-31757
> URL: https://issues.apache.org/jira/browse/FLINK-31757
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: RocMarshal
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
