[
https://issues.apache.org/jira/browse/YUNIKORN-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rainie Li updated YUNIKORN-1988:
--------------------------------
Description:
*Background:*
We set tier based priorityClass and using YuniKorn 1.3 with Admission
controller in production (our prod cluster has hundreds of EKS nodes).
Many production tier2 jobs got preempted unexpectedly. From application log, we
saw driver pods all got shutdown around same time.
Most failed jobs were from the same queue, we set 300G as guaranteed memory for
queue that got preempted, all driver pods required 24G memory.
Right now we disabled preemption feature in production to mitigate the issue.
*Investigation:*
Reproduced the issue on dev env, preemption can happen when a queue is lower
than its guaranteed capacity.
I am investigating how to fix the issue.
was:
*Background:*
We set tier based priorityClass and using YuniKorn 1.3 with Admission
controller in production (our prod cluster has hundreds of EKS nodes).
Many production tier2 jobs got preempted unexpectedly. From application log, we
saw driver pods all got shutdown around same time.
Most failed jobs were from the same queue, we set 300G as guaranteed memory for
queue that got preempted, all driver pods required 24G memory.
Right now we disabled preemption feature in production to mitigate the issue.
*Investigation:*
Reproduced the issue on dev env, preemption can happen when a queue lower than
its guaranteed capacity
I am investigating how to fix the issue.
> Preemption happens when a queue lower than its guaranteed capacity
> -------------------------------------------------------------------
>
> Key: YUNIKORN-1988
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1988
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Rainie Li
> Assignee: Rainie Li
> Priority: Critical
>
> *Background:*
> We set tier based priorityClass and using YuniKorn 1.3 with Admission
> controller in production (our prod cluster has hundreds of EKS nodes).
> Many production tier2 jobs got preempted unexpectedly. From application log,
> we saw driver pods all got shutdown around same time.
> Most failed jobs were from the same queue, we set 300G as guaranteed memory
> for queue that got preempted, all driver pods required 24G memory.
> Right now we disabled preemption feature in production to mitigate the issue.
> *Investigation:*
> Reproduced the issue on dev env, preemption can happen when a queue is lower
> than its guaranteed capacity.
> I am investigating how to fix the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
