Craig Condit created YUNIKORN-2092:
--------------------------------------
Summary: Remove Spark Operator AppManager
Key: YUNIKORN-2092
URL: https://issues.apache.org/jira/browse/YUNIKORN-2092
Project: Apache YuniKorn
Issue Type: Improvement
Components: shim - kubernetes
Reporter: Craig Condit
Assignee: Craig Condit
For some time, YuniKorn has supported a configurable AppManager interface.
However, there are only two implementations:
* general: used for general application management (maps via applicationID)
* spark-k8s-operator: mapping via spark operator
This interface has grown in complexity over time, but is not used consistently
within the code base. Disabling the general AppManager results in a broken
scheduler, as pods do not map correctly. Additionally, the spark-k8s-operator
AppManager is only used in non-default configurations and only maps Spark
Application completion state onto YuniKorn application completion state via
calling NotifyApplicationFail() or NotifyApplicationComplete(). This can
conflict with the normal processing of these events and lead to failures.
To simplify and eventually eliminate the AppManager abstraction, we should
remove the spark-k8s-operator AppManager entirely. No functionality will be
lost (except for a questionable mapping between spark app success/fail and
yunikorn app success/fail). This mapping is questionable anyway as there is no
real concept of Application success in Kubernetes, so it is not used in the
vast majority of cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
