[
https://issues.apache.org/jira/browse/YUNIKORN-3230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wilfred Spiegelenburg reassigned YUNIKORN-3230:
-----------------------------------------------
Assignee: Siddharth Krishna
> fail application when placeholder creation is rejected by LimitRange
> --------------------------------------------------------------------
>
> Key: YUNIKORN-3230
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3230
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Environment: YuniKorn version: 1.8.0
> Spark Operator version: 2.3.0
> Gang scheduling: enabled
> imitRange configuration
> Example:
> apiVersion: v1
> kind: LimitRange
> metadata:
> name: spark-limit
> spec:
> limits:
> - type: Pod
> max:
> cpu: "7"
> memory: 7Gi
> Reporter: Alex Ovchenkov
> Assignee: Siddharth Krishna
> Priority: Major
> Labels: newbie
>
> We are using YuniKorn with gang scheduling (placeholders enabled) together
> with Kubernetes {{{}LimitRange{}}}.
> {{LimitRange}} is configured intentionally to reject Pods that exceed the
> maximum resources allowed per node (fail-fast behavior).
> When a Spark application is submitted:
> * Driver pod is created successfully.
> * YuniKorn tries to create placeholder pods for executors.
> * Placeholder creation is rejected by Kubernetes admission controller due to
> {{{}LimitRange{}}}.
> Expected behavior:
> * The Spark application (YuniKorn application) should transition to a FAILED
> state.
> * Driver pod should not continue running indefinitely.
> Actual behavior:
> * Placeholder creation fails with a {{Forbidden}} error.
> * Driver pod continues running.
> * Application remains in inconsistent state (neither fully running nor
> properly failed).
> ERROR shim.cache.placeholder cache/placeholder_manager.go:92
> failed to create placeholder pod
> \{"error": "pods \"tg-spark-...\" is forbidden:
> [maximum cpu usage per Pod is 7, but limit is 8,
> maximum memory usage per Pod is 7Gi, but limit is 12025069568]"}
>
> h3. Expected behavior
> If placeholder creation fails with a permanent Kubernetes error (e.g.
> {{{}Forbidden{}}}):
> * YuniKorn application should transition to FAILED.
> * Driver pod should be cleaned up (or at least the application should be
> marked as failed).
> * No partial execution should continue.
> ----
> h3. Proposed behavior
> When {{create placeholder pod}} returns a non-retryable error (e.g. HTTP 403
> Forbidden):
> * Treat this as terminal application failure.
> * Move application state to FAILED.
> * Emit clear event explaining the reason.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
