[
https://issues.apache.org/jira/browse/YUNIKORN-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Craig Condit closed YUNIKORN-2327.
----------------------------------
> Race condition during update Occupied Resource from Shim to Core
> ----------------------------------------------------------------
>
> Key: YUNIKORN-2327
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2327
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - common, shim - kubernetes
> Reporter: Yu-Lin Chen
> Assignee: Yu-Lin Chen
> Priority: Major
> Attachments: Verify_basic_preemption_k8sClusterInfo.txt,
> Verify_basic_preemption_ykContainerLog.txt,
> Verify_basic_preemption_ykFullStateDump.json
>
>
> When initializing YuniKorn, existing Non-YuniKorn pods (ForeignPods) are
> counted as node's occupied resources. An SchedulerAPI.UpdateNode(request) is
> triggered asynchronously to update the occupied resources for the node in the
> core. However, a race condition occurs on the core side during this
> asynchronous update process.
> {*}How to reproduce{*}:
> - Add 2 seconds delay for the first pod, the final occupied resource will
> equal to the first pod's resource size after restart YuniKorn.
> ([example|https://github.com/apache/yunikorn-core/compare/master...chenyulin0719:yunikorn-core:YUNIKORN-2313-ADD-2-SECOND-DELAY#diff-3bd07740ee12121844b14ddafec10a36332fe2cd80421174110edf042f780e23R397-R399])
> - The issue is the root cause of YUNIKORN-2313
> {*}Error Logs{*}: ([E2E test
> link-v1.29.0|https://github.com/chenyulin0719/yunikorn-k8shim/actions/runs/7530262471])
> Shim logs: (yk8s-worker)
> {code:json}
> 2024-01-15T14:39:20.135Z Shim trigger SchedulerAPI() request: occupied:
> resources:{key:"pods" value:{value:1}}
> 2024-01-15T14:39:20.135Z Shim trigger SchedulerAPI() request: occupied:
> resources:{key:"memory" value:{value:52428800}} resources:{key:"pods"
> value:{value:2}} resources:{key:"vcore" value:{value:100}}
> 2024-01-15T14:39:20.136Z Shim trigger SchedulerAPI() request: occupied:
> resources:{key:"memory" value:{value:576716800}} resources:{key:"pods"
> value:{value:3}} resources:{key:"vcore" value:{value:200}}
> {code}
> Core logs: (yk8s-worker)
> {code:json}
> 2024-01-15T14:39:20.137Z set occupiedResource: map[memory:52428800 pods:2
> vcore:100]
> 2024-01-15T14:39:20.137Z set occupiedResource: map[memory:576716800 pods:3
> vcore:200]
> 2024-01-15T14:39:22.136Z set occupiedResource: map[pods:1]
> {code}
> {*}Final occupied resource in state dump{*}:
> {code:json}
> ...
> {
> "nodeID": "yk8s-worker",
> "attributes": {
> "ready": "true",
> "si.io/hostname": "yk8s-worker",
> "si.io/rackname": "/rack-default",
> "si/node-partition": "[mycluster]default"
> },
> ...
> "occupied": {
> "pods": 1
> }...
> }
> {code}
> Key code of this issue:
> * (Go routine)
> https://github.com/apache/yunikorn-core/blob/master/pkg/rmproxy/rmproxy.go#L378-L389
>
> * (Set function)
> https://github.com/apache/yunikorn-core/blob/master/pkg/scheduler/objects/node.go#L195-L197
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
