Chesnay Schepler created FLINK-19715:
----------------------------------------
Summary: Optimize re-assignment of excess resources
Key: FLINK-19715
URL: https://issues.apache.org/jira/browse/FLINK-19715
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Reporter: Chesnay Schepler
Fix For: 1.13.0
The {{JobScopedResourceTracker}} tracks acquired resources that exceed the jobs
requirements as {{excess}} resources.
Whenever the requirements increase, or a (non-excess) resource was lost, we try
to assign any excess resources we have to fill either fulfill the requirements
of fill in the lost resource.
This re-assignment is currently implemented by doing a full copy of map
containing the excess resources, and going through the usual code path for
acquired resources.
This is fine in terms of correctness (although it can cause misleading log
messages), but in the worst case, where we cannot re-assign any excess
resources, we not only rebuild the original map in the process, but also
potentially iterate over every outstanding requirement for every excess slot.
We should optimize this step by iterating over the excess resource map once,
removing items on-demand and aborting early for a given excess resource profile
if no matching requirement could be found.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
