[
https://issues.apache.org/jira/browse/FLINK-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237111#comment-16237111
]
Sihua Zhou commented on FLINK-7866:
-----------------------------------
[~till.rohrmann] thanks and so happy for thinking of me when you plan to
revision scheduler, I do have some thought about a properly scheduler, works
incrementally is a nice requirement as you have explained. IMO, it also need
the follow features.
1, Strong Extendibility.
This means that scheduler should be easy to extends for other
`schedule-factor` not only just for state & inputs, E.g: TM's status and
cluster's status.
2, Consider the Runtime Information of the job.
This means that when do scheduling we need to consider the previous
runtime information of the execution, the runtime information should contains
but only `task manager location`, `state size`, `input flow rate`,
`thoughtput`, I think these will be helpful for scheduler. For example, imagine
that vertex `A`,`B` both connect to vertex 'C' with `forward` and if A's
`thoughtput` was `1M \ s` and B's `thoughput` was `100M\s`, than B's slot will
be picked for 'C'. Currently, runtime information can only be filled when we do
recover, is there a chance that the scheduler can Self-Regulating, dynamic
change the schdule result. Ah, what I want to express is runtime information of
execution is helpful.
I am looking forward to your plan and interest in it :)
> Weigh list of preferred locations for scheduling
> ------------------------------------------------
>
> Key: FLINK-7866
> URL: https://issues.apache.org/jira/browse/FLINK-7866
> Project: Flink
> Issue Type: Improvement
> Components: Scheduler
> Affects Versions: 1.4.0, 1.3.2
> Reporter: Till Rohrmann
> Assignee: Sihua Zhou
> Priority: Major
> Fix For: 1.5.0
>
>
> [~sihuazhou] proposed to not only use the list of preferred locations to
> decide where to schedule a task, but to also weigh the list according to how
> often a location appeared and then select the location based on the weight.
> That way, we would obtain better locality in some cases.
> Example:
> Preferred locations list: {{[location1, location2, location2]}}
> Weighted preferred locations list {{[(location2 , 2), (location1, 1)]}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
