[
https://issues.apache.org/jira/browse/FLINK-20864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song reassigned FLINK-20864:
------------------------------------
Assignee: Yangze Guo
> Apply exact matching rules in fulfilling resource requirement with slot
> resource
> --------------------------------------------------------------------------------
>
> Key: FLINK-20864
> URL: https://issues.apache.org/jira/browse/FLINK-20864
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Reporter: Yangze Guo
> Assignee: Yangze Guo
> Priority: Major
> Fix For: 1.13.0
>
> Attachments: 屏幕快照 2021-01-06 下午5.34.17.png
>
>
> Currently, ResourceProfile::isMatching uses the following rules (hereinafter,
> *loose matching*) to decide whether a slot resource can be used to fulfill
> the given resource requirement, in both SlotManager and SlotPool：
> * An unspecified requirement (ResourceProfile::UNKNOWN) can be fulfilled by
> any resource.
> * A specified requirement can be fulfilled by any resource that is greater
> than or equal to itself. Note that this rule is not taking effect since
> there’s no specified requirement atm.
> The loose matching rules were designed before the dynamic slot allocation.
> Under the assumption that resources of slots are decided when the TM is
> started and cannot be changed, the loose matching rules have the following
> advantages.
> * For standalone deployments, it allows slot requests to be fulfilled when
> the slots of pre-launched TMs can hardly have the exact required resources.
> * For active resource manager deployments, it increases the chance of slots
> being reused, thus reducing the cost of starting new TMs for various resource
> requirements.
> With dynamic slot allocation introduced in FLIP-56, the benefits of the loose
> matching rules have been significantly reduced. As slots can be dynamically
> created after the TMs being started, with any desired resources as long as
> available, the only benefit the loose matching rules retain is to avoid
> allocating new slots when the slots can be reused on the JM side, which is
> insignificant since there’s no need to start new TMs.
>
> On the other hand, the loose matching rules also introduce some problems.
> * Reusing larger slots for fulfilling smaller requirements can harm resource
> utilization.
> * It’s not straightforward to always find a feasible matching solution
> (assuming there is one) when matching a set of requirements and slots, in
> cases of job failovers or declarative slot allocation protocol.
> !屏幕快照 2021-01-06 下午5.34.17.png|width=863,height=278!
> The above figure demonstrates how it could fail to find the feasible matching
> solution with the loose matching rules. Assuming there are two resource
> requirements A and B, and there are two slots X and Y. The number below each
> Requirement/Slot represents the amount of resource. Then A can be fulfilled
> with X and Y, while B can only be fulfilled with Y. A feasible matching is
> shown on the left, where both requirements can be fulfilled. However, the
> loose matching rules can also result in another matching, shown on the right,
> where A is fulfilled by Y, leaving B and X unmatched.
> Given the reduction of its benefits and the problems it introduced, we
> proposed to replace the loose matching rules with the following *exact
> matching* rules.
> * An unspecified requirement (ResourceProfile::UNKNOWN) can only be
> fulfilled by a TM's default slot resource.
> * A specified requirement can only be fulfilled by a resource that is equal
> to itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
