[
https://issues.apache.org/jira/browse/FLINK-24005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17406816#comment-17406816
]
Chesnay Schepler edited comment on FLINK-24005 at 8/30/21, 4:20 PM:
--------------------------------------------------------------------
??On a related note, isn't DefaultDeclarativeSlotPool#adjustRequirements only
required by the default scheduler???
In practice this is currently the case, but if the AdaptiveScheduler were to
actually support resource profiles then it would also need it I believe.
??The problem I see is that it moves responsibilities of the
DeclarativeSlotPoolBridge into the DefaultDeclarativeSlotPool.??
Generally yes but this was already the case ever since we added the
DeclarativeSlotPool. This problem isn't new, and we made the conscious decision
to limit any resource concerns to the pool. It certainly wasn't ideal, the so
is that we still need the bridge in the first place.
??fix the problem properly with 1.14.1 and 1.15.0.??
I'm more inclined to leave things as is until we have a clear plan on how we
resolve the duality of schedulers and slot-allocation-protocols. When we
originally worked on the declarative resource management the intent was to
write a new scheduler that supports all jobs; the DeclarativeSlotPoolBridge was
only meant as a temporary measure.
We then limited the adaptive scheduler work to streaming jobs, but never really
concluded what that meant for the bridge and DefaultScheduler.
Will we extend the adaptive scheduler to cover batch jobs?
Will we refactor the DefaultScheduler to directly work with declarative
resource management (at the very least declaring requirements explicitly (which
would solve _a lot_ of issues))?
Will we continue to have this compatibility layer?
Before we start any larger refactorings we should have a clear idea where we
are even headed.
was (Author: zentol):
> On a related note, isn't DefaultDeclarativeSlotPool#adjustRequirements only
> required by the default scheduler?
In practice this is currently the case, but if the AdaptiveScheduler were to
actually support resource profiles then it would also need it I believe.
> The problem I see is that it moves responsibilities of the
> DeclarativeSlotPoolBridge into the DefaultDeclarativeSlotPool.
Generally yes but this was already the case ever since we added the
DeclarativeSlotPool. This problem isn't new, and we made the conscious decision
to limit any resource concerns to the pool. It certainly wasn't ideal, the so
is that we still need the bridge in the first place.
> fix the problem properly with 1.14.1 and 1.15.0.
I'm more inclined to leave things as is until we have a clear plan on how we
resolve the duality of schedulers and slot-allocation-protocols. When we
originally worked on the declarative resource management the intent was to
write a new scheduler that supports all jobs; the DeclarativeSlotPoolBridge was
only meant as a temporary measure.
We then limited the adaptive scheduler work to streaming jobs, but never really
concluded what that meant for the bridge and DefaultScheduler.
Will we extend the adaptive scheduler to cover batch jobs?
Will we refactor the DefaultScheduler to directly work with declarative
resource management (at the very least declaring requirements explicitly (which
would solve _a lot_ of issues))?
Will we continue to have this compatibility layer?
Before we start any larger refactorings we should have a clear idea where we
are even headed.
> Resource requirements declaration may be incorrect if JobMaster disconnects
> with a TaskManager with available slots in the SlotPool
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-24005
> URL: https://issues.apache.org/jira/browse/FLINK-24005
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.14.0, 1.12.5, 1.13.2
> Reporter: Zhu Zhu
> Assignee: Chesnay Schepler
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0, 1.12.6, 1.13.3
>
> Attachments: decrease_resource_requirements.log
>
>
> When a TaskManager disconnects with JobMaster, it will trigger the
> `DeclarativeSlotPoolService#decreaseResourceRequirementsBy()` for all the
> slots that are registered to the JobMaster from the TaskManager. If the slots
> are still available, i.e. not assigned to any task, the
> `decreaseResourceRequirementsBy` may lead to incorrect resource requirements
> declaration.
> For example, there is one job with 3 source tasks only. It requires 3 slots
> and declares for 3 slots. Initially all the tasks are running. Suddenly one
> task failed and waits for some delay before restarting. The previous slot is
> returned to the SlotPool. Now the job requires 2 slots and declares for 2
> slots. At this moment, the TaskManager of that returned slot get lost. After
> the triggered `decreaseResourceRequirementsBy`, the job only declares for 1
> slot. Finally, when the failed task starts to re-schedule, the job will
> declare for 2 slots while it actually needs 3 slots.
> The attached log of a real job and logs of the added test in
> https://github.com/zhuzhurk/flink/commit/59ca0ac5fa9c77b97c6e8a43dcc53ca8a0ad6c37
> can demonstrate this case.
> Note that the real job is configured with a large
> "restart-strategy.fixed-delay.delay" and and large "slot.idle.timeout". So
> possibly in production it is a rare case.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
