[
https://issues.apache.org/jira/browse/YUNIKORN-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Craig Condit resolved YUNIKORN-1712.
------------------------------------
Fix Version/s: 1.3.0
Resolution: Fixed
Merged to master. Thanks [~zhuqi] for the contribution.
> Fix allocatedResource and availableResource should be updated at the sametime
> for ReplaceAllocation
> ---------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-1712
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1712
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Craig Condit
> Assignee: Qi Zhu
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.3.0
>
> Attachments: fullstatedump.json, logs.txt, logs_debug_level.txt
>
>
> Under some circumstances, it seems that placeholder allocations are being
> removed multiple times:
> {code:java}
> 2023-04-25T06:25:46.279Z INFO scheduler/partition.go:1233
> replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar",
> "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> ...
> 2023-04-25T06:25:46.299Z INFO scheduler/partition.go:1233
> replacing placeholder allocation {"appID": "spark-000000031tn2lgv2gar",
> "allocationId": "20a4cf77-7095-4635-b9e9-43a7564385c4"}
> {code}
> This message only appears once in the codebase, in
> {{{}PartitionContext.removeAllocation(){}}}. Furthermore, it is guarded by a
> test for {{{}release.TerminationType ==
> si.TerminationType_PLACEHOLDER_REPLACED{}}}. This would seem to indicate that
> {{removeAllocation()}} is somehow being called {*}twice{*}. I believe this
> would cause the used resources on the node to be subtracted twice for the
> same allocation. This quickly results in health checks failing:
> {code:java}
> 2023-04-25T06:26:10.632Z WARN scheduler/health_checker.go:176
> Scheduler is not healthy {"health check values": [...,
> {"Name":"Consistency of data","Succeeded":false,"Description":"Check if node
> total resource = allocated resource + occupied resource + available
> resource","DiagnosisMessage":"Nodes with inconsistent data:
> [\"ip-10-0-112-148.eu-central-1.compute.internal\"]"}, ...]}
> {code}
> This was originally thought to be YUNIKORN-1615, but that seems related to
> occupied (rather than used) resources.
> Updated by [~zhuqi] :
> The release allocation called twice, but it not cause the node and other
> resource update twice.
> The title will be changed to:
> Fix allocatedResource and availableResource should be updated at the sametime
> for ReplaceAllocation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
