[
https://issues.apache.org/jira/browse/FLINK-37813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Baozhu Zhao updated FLINK-37813:
--------------------------------
Attachment: slot-report.png
new-jm.log
old-jm.log
re-allocate-slot.png
Description:
environment ：
* version : 1.17
* resource provider:
* job desc: The job parallelism=27 ， slotPerWorker=10，need 3 worker
* job config：cluster.fine-grained-resource-management.enabled=true
issue Desc：
* When jobmanager failover, the SlotReport of the registered taskmanager did
not meet expectations, resulting in ResourceManager unable to release the free
taskmanager.
Reproduce steps：
1、Killing a taskManager causes the job to fail, and the slot manager will
reallocate the slot to the existing taskManagers. Before the slot allocation is
completed, killing the jobmanager and put the job in a SUSPEND state.There is a
probability that `
FineGrainedSlotManager` will call method `declareNeededResources()` again to
allocate slots after releasing them.[^old-jm.log]
^!re-allocate-slot.png|width=1152,height=501!^
2、After the new JobManager is launched, the existing taskmanager will register,
and the slotNum in the slotReport reported by the existing task manager will be
larger than slotPerWorker. Causing the `ActiveResourceManager` to fail to
correctly calculate the 'releaseOrRequestWorkerNumber' when checking and
releasing idle task managers on a scheduled basis. [^new-jm.log]
!slot-report.png|width=991,height=554!
!注册的tm.png|width=372,height=190!
was:
environment ：
* version : 1.17
* resource provider:
* job desc: The job parallelism=27 ， slotPerWorker=10，need 3 worker
* job config：cluster.fine-grained-resource-management.enabled=true
issue Desc：
* When jobmanager failover, the SlotReport of the registered taskmanager did
not meet expectations, resulting in ResourceManager unable to release the free
taskmanager.
Reproduce steps：
1、Killing a taskManager causes the job to fail, and the slot manager will
reallocate the slot to the existing taskManagers. Before the slot allocation is
completed, killing the jobmanager and put the job in a SUSPEND state.
[^old-tm.log]
2、After the new JobManager is launched, the existing taskmanager will register,
and the slotNum in the slotReport reported by the existing task manager will be
larger than slotPerWorker. Causing the `ActiveResourceManager` to fail to
correctly calculate the 'releaseOrRequestWorkerNumber' when checking and
releasing idle task managers on a scheduled basis. [^new-tm.log]
!注册的tm.png|width=372,height=190!
> JobManager failover during allocation slots causes ResourceManager to release
> unwanted TaskManager failure
> ----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-37813
> URL: https://issues.apache.org/jira/browse/FLINK-37813
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.17.2, 1.19.2
> Reporter: Baozhu Zhao
> Priority: Major
> Attachments: new-jm.log, new-tm.log, old-jm.log, old-tm.log,
> re-allocate-slot.png, slot-report.png, 注册的tm.png
>
>
> environment ：
> * version : 1.17
> * resource provider:
> * job desc: The job parallelism=27 ， slotPerWorker=10，need 3 worker
> * job config：cluster.fine-grained-resource-management.enabled=true
>
> issue Desc：
> * When jobmanager failover, the SlotReport of the registered taskmanager did
> not meet expectations, resulting in ResourceManager unable to release the
> free taskmanager.
>
> Reproduce steps：
> 1、Killing a taskManager causes the job to fail, and the slot manager will
> reallocate the slot to the existing taskManagers. Before the slot allocation
> is completed, killing the jobmanager and put the job in a SUSPEND state.There
> is a probability that `
> FineGrainedSlotManager` will call method `declareNeededResources()` again to
> allocate slots after releasing them.[^old-jm.log]
> ^!re-allocate-slot.png|width=1152,height=501!^
> 2、After the new JobManager is launched, the existing taskmanager will
> register, and the slotNum in the slotReport reported by the existing task
> manager will be larger than slotPerWorker. Causing the
> `ActiveResourceManager` to fail to correctly calculate the
> 'releaseOrRequestWorkerNumber' when checking and releasing idle task managers
> on a scheduled basis. [^new-jm.log]
> !slot-report.png|width=991,height=554!
> !注册的tm.png|width=372,height=190!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
