This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 1b2b1d11820 Make KubernetesStartKueueJobOperator wait through Kueue
preemption (#75658)
1b2b1d11820 is described below
commit 1b2b1d11820a3be0ee2e3817d4f8708cadba8732
Author: raphaelauv <[email protected]>
AuthorDate: Wed Nov 18 20:22:04 2026 +0100
Make KubernetesStartKueueJobOperator wait through Kueue preemption (#75658)
* Wait for Kueue Jobs through admission and preemption
KubernetesStartKueueJobOperator inherited the KubernetesJobOperator wait,
which looks for the Job's pods right after creating it. Kueue keeps the
Job suspended without pods until it admits it, and may preempt it (suspend
it again and delete its pods) then admit it again, so the task failed to
find pods before admission, and after a preemption waited on, or read logs
and XCom from, pods that had been deleted.
Least-privilege Roles that grant get on jobs but not on jobs/status also
got a 403 when job operators and triggers waited for a Job: Kubernetes
RBAC checks subresources separately, although the Job object carries the
same status.
closes: #74373
* Address Kueue review feedback
* review 2
* Ignore pods of other Jobs when waiting for a Kueue Job
With manualSelector, the selector of a Job can also match the pods of
other Jobs, such as the same task in another Dag run. The operator then
read their XCom, stopping their sidecar, returned it as its own result,
and deleted those pods once its Job finished.
* Simplify Kueue operator docs and comments
The docs and docstrings explained Kueue preemption several times, and
some sentences needed several reads; reviewers and users only need the
behaviour of the operator once, in the guide.
---
docs/spelling_wordlist.txt | 2 +
providers/cncf/kubernetes/docs/changelog.rst | 6 +
providers/cncf/kubernetes/docs/kubernetes_rbac.rst | 17 +-
providers/cncf/kubernetes/docs/operators.rst | 82 ++++++
providers/cncf/kubernetes/provider.yaml | 1 +
.../providers/cncf/kubernetes/get_provider_info.py | 1 +
.../providers/cncf/kubernetes/hooks/kubernetes.py | 11 +-
.../providers/cncf/kubernetes/operators/kueue.py | 156 +++++++++++
.../providers/cncf/kubernetes/triggers/kueue.py | 293 ++++++++++++++++++++
.../unit/cncf/kubernetes/hooks/test_kubernetes.py | 8 +-
.../unit/cncf/kubernetes/operators/test_kueue.py | 274 +++++++++++++++++-
.../unit/cncf/kubernetes/triggers/test_kueue.py | 308 +++++++++++++++++++++
12 files changed, 1141 insertions(+), 18 deletions(-)
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index 02ff683960a..f050126d416 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1774,6 +1774,8 @@ subprocess
subprocesses
subqueries
subquery
+subresource
+subresources
SubscriberClient
subscriptionId
substring
diff --git a/providers/cncf/kubernetes/docs/changelog.rst
b/providers/cncf/kubernetes/docs/changelog.rst
index 7ec63ca253a..b4833431af4 100644
--- a/providers/cncf/kubernetes/docs/changelog.rst
+++ b/providers/cncf/kubernetes/docs/changelog.rst
@@ -38,6 +38,12 @@ Changelog
with ``in_cluster``, ``kube_config`` or ``config_dict`` raises instead of
being ignored — drop
whichever option you did not mean to use.
+.. note::
+ Job operators and triggers now read the Job object instead of the
``jobs/status`` subresource, so they
+ need ``get`` on ``jobs`` (``batch`` API group) rather than on
``jobs/status``. If the ``ServiceAccount``
+ of your triggerer or workers was granted ``get`` on ``jobs/status`` but not
on ``jobs``, grant it
+ ``get`` on ``jobs``.
+
11.0.0
......
diff --git a/providers/cncf/kubernetes/docs/kubernetes_rbac.rst
b/providers/cncf/kubernetes/docs/kubernetes_rbac.rst
index 971c7c06770..5569a5baafe 100644
--- a/providers/cncf/kubernetes/docs/kubernetes_rbac.rst
+++ b/providers/cncf/kubernetes/docs/kubernetes_rbac.rst
@@ -128,23 +128,24 @@ permissions:
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "get", "list", "watch", "patch", "delete"]
- - apiGroups: ["batch"]
- resources: ["jobs/status"]
- verbs: ["get", "watch"]
+
+The Job status is read from the Job object itself, so no permission on the
+``jobs/status`` subresource is needed.
When a job operator waits for completion, streams pod logs, reads XCom, or
cleans up discovered pods, it also needs the relevant pod permissions from
the pod launch section.
For deferrable job operators, the worker creates the Job before deferring and
-the triggerer polls Job status. If the triggerer uses in-cluster credentials,
-bind the triggerer's ``ServiceAccount`` to the Job status permissions. When
+the triggerer polls the Job. If the triggerer uses in-cluster credentials,
+grant the triggerer's ``ServiceAccount`` permission to get jobs. When
XCom is enabled, the triggerer also needs permission to get pods and exec into
-the XCom sidecar container. The worker reads pod logs after the task resumes.
+the XCom sidecar container, and, for ``KubernetesStartKueueJobOperator``, to
+list pods. The worker reads pod logs after the task resumes.
``KubernetesPatchJobOperator`` only needs permission to patch jobs.
-``KubernetesDeleteJobOperator`` reads Job status before deleting a Job, so it
-needs permission to get ``jobs/status`` and delete jobs.
+``KubernetesDeleteJobOperator`` reads the Job before deleting it, so it
+needs permission to get and delete jobs.
Cleanup permissions
-------------------
diff --git a/providers/cncf/kubernetes/docs/operators.rst
b/providers/cncf/kubernetes/docs/operators.rst
index 70a8996379e..4d894f90510 100644
--- a/providers/cncf/kubernetes/docs/operators.rst
+++ b/providers/cncf/kubernetes/docs/operators.rst
@@ -866,6 +866,23 @@ When the task is killed, ``on_kill`` deletes the Job (with
foreground cascade).
For discovered pods, deletion is controlled by ``on_kill_action``:
``delete_pod`` attempts direct pod deletion and ``keep_pod`` skips it.
+XComs of a Job
+^^^^^^^^^^^^^^
+
+With ``wait_until_job_complete=True`` and ``do_xcom_push=True``, each pod
writes its XCom to
+``/airflow/xcom/return.json``, as with ``KubernetesPodOperator``.
``return_value`` is the list of these
+values, one per pod (``None`` for a pod that writes nothing). A list with a
single value is unwrapped,
+unless ``unwrap_single=False``.
+
+Whatever ``do_xcom_push`` is, the operator also pushes:
+
+* ``job_name`` and ``job_namespace``, when the Job is created;
+* ``job``: the Job as a dictionary, with its ``status`` (the finished Job when
+ ``wait_until_job_complete=True``).
+
+``KubernetesJobOperator`` reads the XComs of the pods it found when the Job
started; for
+``KubernetesStartKueueJobOperator``, see
:ref:`howto/operator:KubernetesStartKueueJobOperator`.
+
.. _howto/operator:KubernetesDeleteJobOperator:
@@ -935,6 +952,71 @@ you to start a Kueue job in a Kubernetes cluster
:start-after: [START howto_operator_k8s_install_kueue]
:end-before: [END howto_operator_k8s_install_kueue]
+Waiting for the Job
+^^^^^^^^^^^^^^^^^^^
+
+With ``wait_until_job_complete=True``, the operator waits on the Job itself,
not on its pods, because
+Kueue controls when the pods exist:
+
+* the Job stays suspended, without pods, until Kueue admits it;
+* Kueue may preempt the Job, suspending it again and deleting its running
pods, then admit it again
+ later, any number of times;
+* each admission creates new pods, with new names. Pods that already succeeded
are kept and still count
+ for the Job.
+
+Once the Job is finished, logs are read from the pods it has at that time.
+
+The wait has no timeout of its own. If Kueue never admits the Job (for
example, ``queue_name`` doesn't
+match a ``LocalQueue`` in the namespace), the task waits forever. Set
``execution_timeout`` to limit the
+wait; it also applies while the task is deferred. On timeout, the task fails
and the Job is deleted, so
+Kueue doesn't run it later. ``active_deadline_seconds`` doesn't help here:
Kubernetes resets its timer
+each time the Job is suspended.
+
+XComs
+^^^^^
+
+With ``do_xcom_push=True``:
+
+* the XCom of each pod is read while the Job runs, at the first poll after the
pod's base container
+ ends. Until then, the XCom sidecar keeps the pod, and so the Job, from
finishing;
+* once the Job is finished, only the XComs of pods that are ``Succeeded`` are
kept. XComs of failed
+ pods, and of pods deleted by a preemption, are dropped;
+* ``return_value`` is the list of the kept XComs, one per succeeded pod,
sorted by pod name. See
+ :ref:`howto/operator:kubernetesjoboperator` for ``unwrap_single`` and the
other XComs pushed.
+
+The task fails, rather than return an incomplete list, when the XCom of a
succeeded pod could not be
+read: for example, the pod was deleted before the task finished, or its
``return.json`` is not valid
+JSON. A pod that writes no XCom doesn't fail the task: its value is ``None``.
+
+With ``deferrable=True``, the XComs read so far are kept only in the trigger's
memory. If the trigger
+restarts while the Job runs (for example, when the triggerer is redeployed),
those XComs are lost: their
+sidecars are already stopped, so they can't be read again. If any of those
pods succeeded, the task
+fails once the Job is finished, and a retry runs the whole Job again.
+
+Number of values in ``return_value``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``return_value`` has one value per pod that succeeded:
+
+* by default, one value;
+* with ``completions=N``, ``N`` values;
+* with ``parallelism=M`` and no ``completions`` (work queue), between 1 and
``M`` values.
+
+Values are sorted by pod name, not by the order in which pods ran. If the
order matters, set
+``completion_mode="Indexed"``, write ``$JOB_COMPLETION_INDEX`` into the XCom,
and sort on it in the
+downstream task:
+
+.. code-block:: bash
+
+ echo "{\"index\": $JOB_COMPLETION_INDEX, \"rows\": 42}" >
/airflow/xcom/return.json
+
+Kueue reserves quota for all the pods the Job runs at once, so a high
``parallelism`` can delay admission.
+See the `Kubernetes Job documentation
<https://kubernetes.io/docs/concepts/workloads/controllers/job/>`__
+for how ``completions``, ``parallelism`` and ``backoff_limit`` work.
+
+Reference
+^^^^^^^^^
+
For further information, look at:
* `Kubernetes Documentation <https://kubernetes.io/docs/home/>`__
diff --git a/providers/cncf/kubernetes/provider.yaml
b/providers/cncf/kubernetes/provider.yaml
index 05b588275d3..84c0b29ace8 100644
--- a/providers/cncf/kubernetes/provider.yaml
+++ b/providers/cncf/kubernetes/provider.yaml
@@ -177,6 +177,7 @@ triggers:
python-modules:
- airflow.providers.cncf.kubernetes.triggers.pod
- airflow.providers.cncf.kubernetes.triggers.job
+ - airflow.providers.cncf.kubernetes.triggers.kueue
secrets-backends:
diff --git
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py
index 47e5e5cf536..7d92864d62c 100644
---
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py
+++
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py
@@ -73,6 +73,7 @@ def get_provider_info():
"python-modules": [
"airflow.providers.cncf.kubernetes.triggers.pod",
"airflow.providers.cncf.kubernetes.triggers.job",
+ "airflow.providers.cncf.kubernetes.triggers.kueue",
],
}
],
diff --git
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
index 085ec397ffb..5d2c8b3c99e 100644
---
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
+++
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
@@ -632,13 +632,14 @@ class KubernetesHook(BaseHook, PodOperatorHookProtocol):
"""
Get job with status of specified name and namespace.
+ Reads the Job object rather than the ``jobs/status`` subresource: both
return the same ``status``,
+ but RBAC checks subresources separately, so this only needs ``get`` on
``jobs``.
+
:param job_name: Name of Job to fetch.
:param namespace: Namespace of the Job.
:return: Job object
"""
- return self.batch_v1_client.read_namespaced_job_status(
- name=job_name, namespace=namespace, pretty=True
- )
+ return self.batch_v1_client.read_namespaced_job(name=job_name,
namespace=namespace, pretty=True)
def wait_until_job_complete(self, job_name: str, namespace: str,
job_poll_interval: float = 10) -> V1Job:
"""
@@ -1344,12 +1345,14 @@ class AsyncKubernetesHook(KubernetesHook):
"""
Get job's status object.
+ Reads the Job object, see :meth:`KubernetesHook.get_job_status`.
+
:param name: Name of the pod.
:param namespace: Name of the pod's namespace.
"""
async with self.get_conn() as connection:
v1_api = async_client.BatchV1Api(connection)
- job: V1Job = await v1_api.read_namespaced_job_status(
+ job: V1Job = await v1_api.read_namespaced_job(
name=name,
namespace=namespace,
)
diff --git
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/kueue.py
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/kueue.py
index 4b126163757..d9226c665f7 100644
---
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/kueue.py
+++
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/kueue.py
@@ -19,13 +19,29 @@
from __future__ import annotations
import json
+import time
from collections.abc import Sequence
from functools import cached_property
+from typing import TYPE_CHECKING, Any
+from kubernetes.client import models as k8s
+from kubernetes.client.rest import ApiException
from kubernetes.utils import FailToCreateError
+from urllib3.exceptions import HTTPError
from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
from airflow.providers.cncf.kubernetes.operators.job import
KubernetesJobOperator
+from airflow.providers.cncf.kubernetes.triggers.kueue import (
+ JOB_STATE_MESSAGES,
+ KubernetesKueueJobTrigger,
+ KueueJobState,
+ build_job_event,
+ collect_xcom_results,
+ get_job_state,
+ get_succeeded_pod_xcoms,
+ list_job_pods,
+)
+from airflow.providers.cncf.kubernetes.utils.pod_manager import
XComRetrievalError
from airflow.providers.cncf.kubernetes.version_compat import AIRFLOW_V_3_1_PLUS
from airflow.providers.common.compat.sdk import AirflowException
@@ -34,6 +50,16 @@ if AIRFLOW_V_3_1_PLUS:
else:
from airflow.models import BaseOperator
+# The ``error`` of a deferral timeout in Airflow 3
(``TriggerFailureReason.TRIGGER_TIMEOUT``) and in
+# Airflow 2. Matched as text: Airflow 2 raises the generic
``TaskDeferralError`` for any trigger
+# failure, and ``TaskDeferralTimeout`` moved to another module in Airflow 3.2.
+_DEFERRAL_TIMEOUT_ERRORS = ("Trigger timeout", "Trigger/execution timeout")
+
+if TYPE_CHECKING:
+ from pendulum import DateTime
+
+ from airflow.sdk import Context
+
class KubernetesInstallKueueOperator(BaseOperator):
"""
@@ -105,6 +131,9 @@ class
KubernetesStartKueueJobOperator(KubernetesJobOperator):
"""
Executes a Kubernetes Job in Kueue.
+ With ``wait_until_job_complete=True``, the operator polls the Job itself
rather than its pods: the Job
+ has no pods until Kueue admits it, and loses its running pods when Kueue
preempts it.
+
.. seealso::
For more information on how to use this operator, take a look at the
guide:
:ref:`howto/operator:KubernetesStartKueueJobOperator`
@@ -133,3 +162,130 @@ class
KubernetesStartKueueJobOperator(KubernetesJobOperator):
self.suspend = True
self.labels.update({"kueue.x-k8s.io/queue-name": self.queue_name})
self.annotations.update({"kueue.x-k8s.io/queue-name": self.queue_name})
+
+ def execute(self, context: Context):
+ if not self.wait_until_job_complete:
+ return super().execute(context)
+
+ if self.parallelism is not None and self.parallelism < 1:
+ raise ValueError("parallelism cannot be less than 1 with
`wait_until_job_complete=True`.")
+ self.name = self._set_name(self.name)
+ self.job_request_obj = self.build_job_request_obj(context)
+ self.job = self.create_job(job_request_obj=self.job_request_obj) #
must set `self.job` for `on_kill`
+
+ ti = context["ti"]
+ ti.xcom_push(key="job_name", value=self.job.metadata.name)
+ ti.xcom_push(key="job_namespace", value=self.job.metadata.namespace)
+
+ if self.deferrable:
+ self.defer(
+ trigger=KubernetesKueueJobTrigger(
+ job_name=self.job.metadata.name,
+ job_namespace=self.job.metadata.namespace,
+ base_container_name=self.base_container_name,
+ kubernetes_conn_id=self.kubernetes_conn_id,
+ cluster_context=self.cluster_context,
+ config_file=self.config_file,
+ in_cluster=self.in_cluster,
+ poll_interval=self.job_poll_interval,
+ do_xcom_push=self.do_xcom_push,
+ ),
+ method_name="execute_complete",
+ # Airflow 3 does not apply execution_timeout to the time spent
deferred
+ timeout=self.execution_timeout,
+ )
+
+ return self.execute_complete(
+ context, self._wait_until_job_finished(self.job.metadata.name,
self.job.metadata.namespace)
+ )
+
+ def _wait_until_job_finished(self, job_name: str, namespace: str) ->
dict[str, Any]:
+ """Poll the Job until it is finished, and return the event
``KubernetesKueueJobTrigger`` yields."""
+ state: KueueJobState | None = None
+ xcom_results: dict[str, str] = {}
+ while True:
+ job = self.hook.get_job_status(job_name=job_name,
namespace=namespace)
+ if (new_state := get_job_state(job)) != state:
+ state = new_state
+ self.log.info("The job '%s' %s", job_name,
JOB_STATE_MESSAGES[state])
+ if state == KueueJobState.FINISHED:
+ return build_job_event(job, xcom_results)
+ if state == KueueJobState.RUNNING and self.do_xcom_push:
+ collect_xcom_results(
+ self.client, self.pod_manager, job,
self.base_container_name, xcom_results
+ )
+ time.sleep(self.job_poll_interval)
+
+ def execute_complete(self, context: Context, event: dict, **kwargs):
+ if not isinstance(event.get("xcom_result"), dict):
+ # Only KubernetesKueueJobTrigger gives "xcom_result" as a dict (by
pod name): this event is from
+ # KubernetesJobTrigger, for a task deferred before the provider
upgrade
+ return super().execute_complete(context, event, **kwargs)
+ pods = sorted(
+ list_job_pods(
+ self.client,
+ event["namespace"],
+ event["pod_label_selector"],
+ event["job"]["metadata"]["uid"],
+ ),
+ key=lambda pod: pod.metadata.name,
+ )
+ for pod in pods:
+ self.log_matching_pod(pod=pod, context=context)
+ if self.get_logs and event["status"] == "error":
+ # super() raises for a failed Job before it writes the logs of the
pods
+ for pod in pods:
+ self._write_logs(pod)
+ xcom_result = get_succeeded_pod_xcoms(pods, event["xcom_result"])
+ result = super().execute_complete(
+ context,
+ {
+ **event,
+ "pod_names": [pod.metadata.name for pod in pods],
+ "pod_namespace": event["namespace"],
+ "xcom_result": xcom_result,
+ },
+ **kwargs,
+ )
+ # Only reached for a successful Job: super() raises for a failed one
+ succeeded = (event["job"].get("status") or {}).get("succeeded") or 0
+ if self.do_xcom_push and len(xcom_result) < succeeded:
+ raise XComRetrievalError(
+ f"Read the XComs of only {len(xcom_result)} of the {succeeded}
pods that succeeded in "
+ f"the job '{event['name']}': the other pods were deleted, or
their XComs could not be read"
+ )
+ return result
+
+ def resume_execution(self, next_method: str, next_kwargs: dict[str, Any] |
None, context: Context):
+ if next_method == "__fail__" and (next_kwargs or {}).get("error") in
_DEFERRAL_TIMEOUT_ERRORS:
+ self._delete_job_after_deferral_timeout(context)
+ return super().resume_execution(next_method, next_kwargs, context)
+
+ def _delete_job_after_deferral_timeout(self, context: Context) -> None:
+ """
+ Delete the Job when the deferral timed out, as ``on_kill`` does when
the sync wait times out.
+
+ ``on_kill`` is not called for a deferred task, and a Job that Kueue
never admits would otherwise
+ stay in its queue, and be run if Kueue admits it later.
+ """
+ ti = context["ti"]
+ # Without map_indexes, xcom_pull returns the values of every map index
of a mapped task
+ name = ti.xcom_pull(task_ids=ti.task_id, key="job_name",
map_indexes=ti.map_index)
+ namespace = ti.xcom_pull(task_ids=ti.task_id, key="job_namespace",
map_indexes=ti.map_index)
+ if not name:
+ return
+ self.log.info("Deleting the job '%s' as the task timed out while
waiting for it", name)
+ # on_kill deletes self.job, with on_kill_propagation_policy and
termination_grace_period
+ self.job = k8s.V1Job(metadata=k8s.V1ObjectMeta(name=name,
namespace=namespace))
+ try:
+ self.on_kill()
+ except ApiException as e:
+ self.log.warning("Could not delete the job '%s': %s", name, e)
+
+ def _write_logs(self, pod: k8s.V1Pod, follow: bool = False, since_time:
DateTime | None = None) -> None:
+ """Write the logs of the ``container_logs`` of a pod of the finished
Job."""
+ # Failed pods of earlier attempts may have no readable logs left, e.g.
once evicted
+ try:
+ self.pod_manager.fetch_requested_container_logs(pod=pod,
containers=self.container_logs)
+ except (HTTPError, ApiException) as e:
+ self.log.warning("Could not read the logs of pod %s: %s",
pod.metadata.name, e)
diff --git
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/kueue.py
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/kueue.py
new file mode 100644
index 00000000000..9c4f0a43802
--- /dev/null
+++
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/kueue.py
@@ -0,0 +1,293 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Wait for a Kubernetes Job run through Kueue."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from collections.abc import AsyncIterator, Iterable
+from enum import Enum
+from functools import cached_property
+from typing import TYPE_CHECKING, Any
+
+from kubernetes.client.rest import ApiException
+
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import (
+ JOB_FINAL_STATUS_CONDITION_TYPES,
+ AsyncKubernetesHook,
+ KubernetesHook,
+)
+from airflow.providers.cncf.kubernetes.kubernetes_helper_functions import
generic_api_retry
+from airflow.providers.cncf.kubernetes.utils.container import
container_is_running, container_is_terminated
+from airflow.providers.cncf.kubernetes.utils.pod_manager import PodManager,
XComRetrievalError
+from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
+from airflow.triggers.base import BaseTrigger, TriggerEvent
+
+if TYPE_CHECKING:
+ from kubernetes.client import CoreV1Api, models as k8s
+
+
+class KueueJobState(str, Enum):
+ """State of a Job run through Kueue."""
+
+ SUSPENDED = "suspended"
+ RUNNING = "running"
+ FINISHED = "finished"
+
+
+JOB_STATE_MESSAGES = {
+ KueueJobState.SUSPENDED: "is suspended: waiting for admission by Kueue, or
preempted by Kueue",
+ KueueJobState.RUNNING: "is admitted by Kueue: its pods are starting or
running",
+ KueueJobState.FINISHED: "is finished",
+}
+
+
+def get_job_state(job: k8s.V1Job) -> KueueJobState:
+ """Return the state of a Job run through Kueue; finished means Complete or
Failed."""
+ conditions = (job.status.conditions if job.status else None) or []
+ if any(c.type in JOB_FINAL_STATUS_CONDITION_TYPES and c.status == "True"
for c in conditions):
+ return KueueJobState.FINISHED
+ # The Job is created suspended; Kueue resumes it on admission and suspends
it again on preemption
+ if job.spec and job.spec.suspend:
+ return KueueJobState.SUSPENDED
+ return KueueJobState.RUNNING
+
+
+def build_pod_label_selector(job: k8s.V1Job) -> str:
+ """
+ Return the label selector of the pods of the Job, built from its
``spec.selector``.
+
+ Kubernetes adds the ``batch.kubernetes.io/job-name`` label to the pods
only without ``manualSelector``,
+ while ``spec.selector`` matches them in both modes.
+ """
+ selector = job.spec.selector
+ requirements = [f"{key}={value}" for key, value in
sorted((selector.match_labels or {}).items())]
+ for expression in selector.match_expressions or []:
+ values = ",".join(expression.values or [])
+ requirements.append(
+ {
+ "In": f"{expression.key} in ({values})",
+ "NotIn": f"{expression.key} notin ({values})",
+ "Exists": expression.key,
+ "DoesNotExist": f"!{expression.key}",
+ }[expression.operator]
+ )
+ return ",".join(requirements)
+
+
+def build_job_event(job: k8s.V1Job, xcom_results: dict[str, str]) -> dict[str,
Any]:
+ """
+ Build the event of a finished Job, as passed to
``KubernetesStartKueueJobOperator.execute_complete``.
+
+ :param job: the finished Job.
+ :param xcom_results: the raw XComs read from the pods, by pod name.
+ """
+ error_message = KubernetesHook.is_job_failed(job=job)
+ return {
+ "name": job.metadata.name,
+ "namespace": job.metadata.namespace,
+ "status": "error" if error_message else "success",
+ "message": f"Job failed with error: {error_message}"
+ if error_message
+ else "Job completed successfully",
+ "job": job.to_dict(),
+ "pod_label_selector": build_pod_label_selector(job),
+ "xcom_result": xcom_results,
+ }
+
+
+def get_succeeded_pod_xcoms(pods: Iterable[k8s.V1Pod], xcom_results: dict[str,
str]) -> list[str]:
+ """Return the XComs read from the pods that succeeded, skipping failed and
preempted pods."""
+ return [
+ xcom_results[pod.metadata.name]
+ for pod in pods
+ if pod.metadata.name in xcom_results and pod.status and
pod.status.phase == "Succeeded"
+ ]
+
+
+@generic_api_retry
+def list_job_pods(client: CoreV1Api, namespace: str, label_selector: str,
job_uid: str) -> list[k8s.V1Pod]:
+ """
+ Return the pods the Job has now: those matched by
``build_pod_label_selector`` that the Job owns.
+
+ With ``manualSelector``, the selector may also match the pods of other
Jobs, such as the same task
+ in another Dag run; the Job controller tells them apart by owner too.
+ """
+ pods = client.list_namespaced_pod(namespace=namespace,
label_selector=label_selector).items
+ return [pod for pod in pods if any(owner.uid == job_uid for owner in
pod.metadata.owner_references or [])]
+
+
+@generic_api_retry
+def is_pod_deleted(client: CoreV1Api, pod: k8s.V1Pod) -> bool:
+ """Return whether the pod no longer exists."""
+ try:
+ client.read_namespaced_pod(name=pod.metadata.name,
namespace=pod.metadata.namespace)
+ except ApiException as e:
+ if e.status == 404:
+ return True
+ raise
+ return False
+
+
+def collect_xcom_results(
+ client: CoreV1Api,
+ pod_manager: PodManager,
+ job: k8s.V1Job,
+ base_container_name: str,
+ xcom_results: dict[str, str],
+) -> None:
+ """
+ Read into ``xcom_results`` the XCom of each pod of the Job whose base
container has ended.
+
+ Reading the XCom also stops the XCom sidecar: until then the pod, and so
the Job, cannot finish.
+ Pods already read are skipped, so this is called at each poll.
+ """
+ for pod in list_job_pods(client, job.metadata.namespace,
build_pod_label_selector(job), job.metadata.uid):
+ if (
+ pod.metadata.name in xcom_results
+ or not container_is_terminated(pod, base_container_name)
+ or not container_is_running(pod,
PodDefaults.SIDECAR_CONTAINER_NAME)
+ ):
+ continue
+ try:
+ xcom_results[pod.metadata.name] = pod_manager.extract_xcom(pod,
ignore_kill_failure=False)
+ except XComRetrievalError as e:
+ pod_manager.log.warning("Cannot read the XCom of the pod %s: %s",
pod.metadata.name, e)
+ except ApiException as e:
+ # Skip a pod deleted meanwhile, e.g. by a preemption:
kubernetes.stream reports any failed
+ # exec with status 0, so check that the pod is gone. Raise other
failures, such as a
+ # forbidden exec: skipped, the sidecar would keep the Job running
forever.
+ if e.status != 404 and not (e.status == 0 and
is_pod_deleted(client, pod)):
+ raise
+ pod_manager.log.warning("The pod %s was deleted before its XCom
was read", pod.metadata.name)
+ except json.JSONDecodeError as e:
+ # The sidecar is stopped anyway, so the pod is not read again. If
the pod succeeded, the task
+ # fails once the Job is finished, as its XCom is missing.
+ pod_manager.log.warning("The XCom of the pod %s is not valid JSON:
%s", pod.metadata.name, e)
+
+
+class KubernetesKueueJobTrigger(BaseTrigger):
+ """
+ Wait until a Job run through Kueue is finished, even if Kueue preempts it
several times.
+
+ Unlike
:class:`~airflow.providers.cncf.kubernetes.triggers.job.KubernetesJobTrigger`,
it polls the Job
+ rather than the pods found when the Job started, and reads the XComs from
the pods the Job has at
+ each poll.
+
+ :param job_name: The name of the job.
+ :param job_namespace: The namespace of the job.
+ :param base_container_name: The name of the base container in the pod.
+ :param kubernetes_conn_id: The :ref:`kubernetes connection id
<howto/connection:kubernetes>`
+ for the Kubernetes cluster.
+ :param cluster_context: Context that points to kubernetes cluster.
+ :param config_file: Path to kubeconfig file.
+ :param poll_interval: Polling period in seconds to check for the status.
+ :param in_cluster: run kubernetes client with in_cluster configuration.
+ :param do_xcom_push: If True, the content of the file
+ /airflow/xcom/return.json in the container will also be pushed to an
+ XCom when the container completes.
+ """
+
+ def __init__(
+ self,
+ job_name: str,
+ job_namespace: str,
+ base_container_name: str,
+ kubernetes_conn_id: str | None = None,
+ poll_interval: float = 10.0,
+ cluster_context: str | None = None,
+ config_file: str | None = None,
+ in_cluster: bool | None = None,
+ do_xcom_push: bool = False,
+ ):
+ super().__init__()
+ self.job_name = job_name
+ self.job_namespace = job_namespace
+ self.base_container_name = base_container_name
+ self.kubernetes_conn_id = kubernetes_conn_id
+ self.poll_interval = poll_interval
+ self.cluster_context = cluster_context
+ self.config_file = config_file
+ self.in_cluster = in_cluster
+ self.do_xcom_push = do_xcom_push
+
+ def serialize(self) -> tuple[str, dict[str, Any]]:
+ """Serialize KubernetesKueueJobTrigger arguments and classpath."""
+ return (
+
"airflow.providers.cncf.kubernetes.triggers.kueue.KubernetesKueueJobTrigger",
+ {
+ "job_name": self.job_name,
+ "job_namespace": self.job_namespace,
+ "base_container_name": self.base_container_name,
+ "kubernetes_conn_id": self.kubernetes_conn_id,
+ "poll_interval": self.poll_interval,
+ "cluster_context": self.cluster_context,
+ "config_file": self.config_file,
+ "in_cluster": self.in_cluster,
+ "do_xcom_push": self.do_xcom_push,
+ },
+ )
+
+ async def run(self) -> AsyncIterator[TriggerEvent]:
+ """Poll the Job until it is finished and yield a TriggerEvent."""
+ state: KueueJobState | None = None
+ xcom_results: dict[str, str] = {}
+ while True:
+ job = await self.hook.get_job_status(name=self.job_name,
namespace=self.job_namespace)
+ if (new_state := get_job_state(job)) != state:
+ state = new_state
+ self.log.info("The job '%s' %s", self.job_name,
JOB_STATE_MESSAGES[state])
+ if state == KueueJobState.FINISHED:
+ yield TriggerEvent(build_job_event(job, xcom_results))
+ return
+ if state == KueueJobState.RUNNING and self.do_xcom_push:
+ await asyncio.to_thread(self._collect_xcom_results, job,
xcom_results)
+ await asyncio.sleep(self.poll_interval)
+
+ def _collect_xcom_results(self, job: k8s.V1Job, xcom_results: dict[str,
str]) -> None:
+ # Run in a thread: the sync client loads its connection and kube
config on first use
+ collect_xcom_results(
+ self.sync_hook.core_v1_client, self.pod_manager, job,
self.base_container_name, xcom_results
+ )
+
+ @cached_property
+ def hook(self) -> AsyncKubernetesHook:
+ return AsyncKubernetesHook(
+ conn_id=self.kubernetes_conn_id,
+ in_cluster=self.in_cluster,
+ config_file=self.config_file,
+ cluster_context=self.cluster_context,
+ )
+
+ @cached_property
+ def sync_hook(self) -> KubernetesHook:
+ return KubernetesHook(
+ conn_id=self.kubernetes_conn_id,
+ in_cluster=self.in_cluster,
+ config_file=self.config_file,
+ cluster_context=self.cluster_context,
+ )
+
+ @cached_property
+ def pod_manager(self) -> PodManager:
+ return PodManager(kube_client=self.sync_hook.core_v1_client)
+
+ async def cleanup(self) -> None:
+ """Release the hook's cached API client when the trigger exits."""
+ await super().cleanup()
+ await self.hook.close()
diff --git
a/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/hooks/test_kubernetes.py
b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/hooks/test_kubernetes.py
index bfb9b769fd5..0e3b1f02f43 100644
---
a/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/hooks/test_kubernetes.py
+++
b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/hooks/test_kubernetes.py
@@ -704,12 +704,12 @@ class TestKubernetesHook:
@patch("kubernetes.config.kube_config.KubeConfigMerger")
@patch(f"{HOOK_MODULE}.KubernetesHook.batch_v1_client")
def test_get_job_status(self, mock_client, mock_kube_config_merger,
mock_kube_config_loader):
- job_expected = mock_client.read_namespaced_job_status.return_value
+ job_expected = mock_client.read_namespaced_job.return_value
hook = KubernetesHook()
job_actual = hook.get_job_status(job_name=JOB_NAME,
namespace=NAMESPACE)
- mock_client.read_namespaced_job_status.assert_called_once_with(
+ mock_client.read_namespaced_job.assert_called_once_with(
name=JOB_NAME, namespace=NAMESPACE, pretty=True
)
assert job_actual == job_expected
@@ -2062,7 +2062,7 @@ class TestAsyncKubernetesHook:
assert exc_info.value.status == 404
@pytest.mark.asyncio
- @mock.patch(KUBE_BATCH_API.format("read_namespaced_job_status"))
+ @mock.patch(KUBE_BATCH_API.format("read_namespaced_job"))
async def test_get_job_status(self, lib_method, kube_config_loader, hook):
lib_method.return_value = self.mock_await_result(None)
@@ -2071,7 +2071,7 @@ class TestAsyncKubernetesHook:
namespace=NAMESPACE,
)
- lib_method.assert_called_once()
+ lib_method.assert_called_once_with(name=JOB_NAME, namespace=NAMESPACE)
@pytest.mark.asyncio
@mock.patch(HOOK_MODULE + ".asyncio.sleep")
diff --git
a/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_kueue.py
b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_kueue.py
index 907d0e44023..e7640c821b5 100644
---
a/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_kueue.py
+++
b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_kueue.py
@@ -17,19 +17,33 @@
from __future__ import annotations
import json
-from datetime import datetime
+from datetime import datetime, timedelta
from unittest import mock
import pytest
+from kubernetes.client import BatchV1Api, CoreV1Api, models as k8s
+from kubernetes.client.rest import ApiException
from kubernetes.utils import FailToCreateError
from airflow.models.dag import DAG
+from airflow.models.taskinstance import TaskInstance
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook
from airflow.providers.cncf.kubernetes.operators.job import
KubernetesJobOperator
from airflow.providers.cncf.kubernetes.operators.kueue import (
KubernetesInstallKueueOperator,
KubernetesStartKueueJobOperator,
)
-from airflow.providers.common.compat.sdk import AirflowException
+from airflow.providers.cncf.kubernetes.triggers.kueue import
KubernetesKueueJobTrigger, build_job_event
+from airflow.providers.cncf.kubernetes.utils.pod_manager import PodManager,
XComRetrievalError
+from airflow.providers.common.compat.sdk import AirflowException, TaskDeferred
+
+from unit.cncf.kubernetes.triggers.test_kueue import (
+ JOB_NAME,
+ NAMESPACE,
+ POD_LABEL_SELECTOR,
+ build_job,
+ build_pod,
+)
TEST_TASK_ID = "test_task"
TEST_K8S_CONN_ID = "test_kubernetes_conn_id"
@@ -193,3 +207,259 @@ class TestKubernetesStartKueueJobOperator:
assert operator.labels == {"kueue.x-k8s.io/queue-name":
TEST_QUEUE_NAME}
assert operator.annotations == {"kueue.x-k8s.io/queue-name":
TEST_QUEUE_NAME}
mock_log.info.assert_called_once_with(expected_info_message,
"KubernetesStartKueueJobOperator")
+
+
+class TestKubernetesStartKueueJobOperatorWait:
+ @pytest.fixture(autouse=True)
+ def setup_mocks(self):
+ operator_class = KubernetesStartKueueJobOperator
+ with (
+ mock.patch.object(
+ operator_class, "hook", mock.create_autospec(KubernetesHook,
instance=True)
+ ) as self.mock_hook,
+ mock.patch.object(
+ operator_class, "client", mock.create_autospec(CoreV1Api,
instance=True)
+ ) as self.mock_client,
+ mock.patch.object(
+ operator_class, "pod_manager",
mock.create_autospec(PodManager, instance=True)
+ ) as self.mock_pod_manager,
+ mock.patch.object(operator_class, "post_complete_action",
autospec=True),
+ mock.patch.object(
+ operator_class, "build_job_request_obj", autospec=True,
return_value=build_job(True)
+ ),
+ mock.patch.object(
+ operator_class,
+ "create_job",
+ autospec=True,
+ side_effect=lambda operator, job_request_obj: job_request_obj,
+ ) as self.mock_create_job,
+ ):
+ self.mock_hook.get_pod.side_effect = lambda name, namespace:
self.pods[name]
+ yield
+
+ @staticmethod
+ def build_context() -> dict:
+ return {"ti": mock.MagicMock(spec=TaskInstance, try_number=1)}
+
+ def set_pods(self, *pod_lists: list[k8s.V1Pod]) -> None:
+ """Make each list_namespaced_pod call return the next list of pods."""
+ self.pods = {pod.metadata.name: pod for pods in pod_lists for pod in
pods}
+ self.mock_client.list_namespaced_pod.side_effect =
[k8s.V1PodList(items=pods) for pods in pod_lists]
+
+ def build_operator(self, **kwargs) -> KubernetesStartKueueJobOperator:
+ return KubernetesStartKueueJobOperator(
+ task_id=TEST_TASK_ID,
+ queue_name=TEST_QUEUE_NAME,
+ namespace=NAMESPACE,
+ wait_until_job_complete=True,
+ job_poll_interval=0,
+ **kwargs,
+ )
+
+ @mock.patch.object(KubernetesJobOperator, "execute", autospec=True)
+ def test_execute_without_wait_runs_job_operator(self, mock_execute):
+ operator = self.build_operator()
+ operator.wait_until_job_complete = False
+
+ assert operator.execute({}) == mock_execute.return_value
+ self.mock_hook.get_job_status.assert_not_called()
+
+ def test_execute_raises_when_parallelism_is_less_than_1(self):
+ with pytest.raises(ValueError, match="parallelism cannot be less than
1"):
+ self.build_operator(parallelism=0).execute(self.build_context())
+ self.mock_create_job.assert_not_called()
+
+ def test_execute_waits_through_preemption_and_reads_pods_of_last_run(self):
+ """Kueue admits the Job, preempts it (its first pod is deleted),
admits it again, then it completes."""
+ last_run = build_pod("last-run", base="terminated", sidecar="running",
phase="Succeeded")
+ self.set_pods([build_pod("first-run", base="running",
sidecar="running")], [last_run], [last_run])
+ self.mock_hook.get_job_status.side_effect = [
+ build_job(True),
+ build_job(False),
+ build_job(True),
+ build_job(False),
+ build_job(False, "Complete"),
+ ]
+ self.mock_pod_manager.extract_xcom.return_value = '{"a": 1}'
+
+ result = self.build_operator(do_xcom_push=True,
get_logs=True).execute(self.build_context())
+
+ assert result == {"a": 1}
+ self.mock_hook.get_job_status.assert_called_with(job_name=JOB_NAME,
namespace=NAMESPACE)
+ self.mock_client.list_namespaced_pod.assert_called_with(
+ namespace=NAMESPACE, label_selector=POD_LABEL_SELECTOR
+ )
+ self.mock_pod_manager.extract_xcom.assert_called_once_with(last_run,
ignore_kill_failure=False)
+
self.mock_pod_manager.fetch_requested_container_logs.assert_called_once_with(
+ pod=last_run, containers="base"
+ )
+
+ def test_execute_writes_logs_of_pods_when_job_failed(self):
+ failed = build_pod("failed", base="terminated", phase="Failed")
+ self.set_pods([failed])
+ self.mock_hook.get_job_status.side_effect = [build_job(True),
build_job(False, "Failed")]
+
+ with pytest.raises(AirflowException, match="Job failed with error:
BackoffLimitExceeded"):
+ self.build_operator(get_logs=True, container_logs=["base",
"side"]).execute(self.build_context())
+
self.mock_pod_manager.fetch_requested_container_logs.assert_called_once_with(
+ pod=failed, containers=["base", "side"]
+ )
+
+ def
test_execute_complete_writes_logs_of_other_pods_when_logs_of_a_pod_cannot_be_read(self):
+ evicted = build_pod("evicted", phase="Failed")
+ succeeded = build_pod("succeeded", base="terminated",
phase="Succeeded")
+ self.set_pods([evicted, succeeded])
+ self.mock_pod_manager.fetch_requested_container_logs.side_effect =
[ApiException(status=400), []]
+ event = build_job_event(build_job(False, "Complete", succeeded=1), {})
+
+
self.build_operator(get_logs=True).execute_complete(self.build_context(), event)
+
+ assert
self.mock_pod_manager.fetch_requested_container_logs.call_args_list == [
+ mock.call(pod=evicted, containers="base"),
+ mock.call(pod=succeeded, containers="base"),
+ ]
+
+ def
test_execute_deferrable_defers_to_kueue_job_trigger_until_execution_timeout(self):
+ with pytest.raises(TaskDeferred) as deferred:
+ self.build_operator(deferrable=True,
execution_timeout=timedelta(minutes=5)).execute(
+ self.build_context()
+ )
+
+ assert isinstance(deferred.value.trigger, KubernetesKueueJobTrigger)
+ assert deferred.value.method_name == "execute_complete"
+ assert deferred.value.timeout == timedelta(minutes=5)
+ self.mock_client.list_namespaced_pod.assert_not_called()
+
+ @pytest.mark.parametrize(
+ "delete_error", [None, ApiException(status=404)], ids=["job-deleted",
"job-already-deleted"]
+ )
+ @pytest.mark.parametrize(
+ "error", ["Trigger timeout", "Trigger/execution timeout"],
ids=["airflow-3", "airflow-2"]
+ )
+ @mock.patch.object(KubernetesJobOperator, "resume_execution",
autospec=True)
+ @mock.patch.object(
+ KubernetesStartKueueJobOperator,
+ "job_client",
+ new_callable=lambda: mock.create_autospec(BatchV1Api, instance=True),
+ )
+ def test_resume_execution_deletes_job_when_deferral_timed_out(
+ self, mock_job_client, mock_resume_execution, error, delete_error
+ ):
+ mock_resume_execution.side_effect = AirflowException(error)
+ mock_job_client.delete_namespaced_job.side_effect = delete_error
+ context = self.build_context()
+ context["ti"].task_id = TEST_TASK_ID
+ context["ti"].map_index = 1
+ context["ti"].xcom_pull.side_effect = lambda task_ids, key,
map_indexes: {
+ ("job_name", 1): JOB_NAME,
+ ("job_namespace", 1): NAMESPACE,
+ }[(key, map_indexes)]
+
+ with pytest.raises(AirflowException, match=error):
+ self.build_operator(deferrable=True).resume_execution("__fail__",
{"error": error}, context)
+ mock_job_client.delete_namespaced_job.assert_called_once_with(
+ name=JOB_NAME, namespace=NAMESPACE, propagation_policy="Foreground"
+ )
+
+ @mock.patch.object(KubernetesJobOperator, "resume_execution",
autospec=True)
+ @mock.patch.object(
+ KubernetesStartKueueJobOperator,
+ "job_client",
+ new_callable=lambda: mock.create_autospec(BatchV1Api, instance=True),
+ )
+ def test_resume_execution_deletes_no_job_when_job_name_was_not_pushed(
+ self, mock_job_client, mock_resume_execution
+ ):
+ mock_resume_execution.side_effect = AirflowException("Trigger timeout")
+ context = self.build_context()
+ context["ti"].xcom_pull.return_value = None
+
+ with pytest.raises(AirflowException, match="Trigger timeout"):
+ self.build_operator(deferrable=True).resume_execution(
+ "__fail__", {"error": "Trigger timeout"}, context
+ )
+ mock_job_client.delete_namespaced_job.assert_not_called()
+
+ @mock.patch.object(KubernetesJobOperator, "resume_execution",
autospec=True)
+ @mock.patch.object(
+ KubernetesStartKueueJobOperator,
+ "job_client",
+ new_callable=lambda: mock.create_autospec(BatchV1Api, instance=True),
+ )
+ def test_resume_execution_keeps_job_when_trigger_failed(self,
mock_job_client, mock_resume_execution):
+ mock_resume_execution.side_effect = AirflowException("Trigger failure")
+
+ with pytest.raises(AirflowException, match="Trigger failure"):
+ self.build_operator(deferrable=True).resume_execution(
+ "__fail__", {"error": "Trigger failure"}, self.build_context()
+ )
+ mock_job_client.delete_namespaced_job.assert_not_called()
+
+ @pytest.mark.parametrize("xcom_result", [None, ['{"a": 1}']],
ids=["without-xcom", "with-xcom"])
+ @mock.patch.object(KubernetesJobOperator, "execute_complete",
autospec=True)
+ def test_execute_complete_hands_event_of_job_trigger_to_job_operator(
+ self, mock_execute_complete, xcom_result
+ ):
+ """A task deferred to KubernetesJobTrigger by an earlier provider
version resumes after the upgrade."""
+ event = {
+ "name": JOB_NAME,
+ "namespace": NAMESPACE,
+ "pod_names": ["pod"],
+ "pod_namespace": NAMESPACE,
+ "status": "success",
+ "message": "Job completed successfully",
+ "job": build_job(False, "Complete", succeeded=1).to_dict(),
+ "xcom_result": xcom_result,
+ }
+ operator = self.build_operator(do_xcom_push=True)
+ context = self.build_context()
+
+ assert operator.execute_complete(context, event) ==
mock_execute_complete.return_value
+ mock_execute_complete.assert_called_once_with(operator, context, event)
+ self.mock_client.list_namespaced_pod.assert_not_called()
+
+ def test_execute_complete_returns_xcom_of_succeeded_pods_of_the_job(self):
+ self.set_pods(
+ [
+ build_pod("succeeded", base="terminated", phase="Succeeded"),
+ build_pod("failed", base="terminated", phase="Failed"),
+ build_pod("other-job", base="terminated", phase="Succeeded",
owner_uid="other-job-uid"),
+ ]
+ )
+ event = build_job_event(
+ build_job(False, "Complete", succeeded=1),
+ {"succeeded": '{"a": 1}', "failed": '{"b": 2}', "preempted": "3",
"other-job": '{"c": 3}'},
+ )
+
+ result = self.build_operator(do_xcom_push=True,
unwrap_single=False).execute_complete(
+ self.build_context(), event
+ )
+
+ assert result == [{"a": 1}]
+
+ def test_execute_complete_orders_xcom_by_pod_name(self):
+ self.set_pods(
+ [build_pod(name, base="terminated", phase="Succeeded") for name in
("job-1-b", "job-0-a")]
+ )
+ event = build_job_event(
+ build_job(False, "Complete", succeeded=2), {"job-1-b": '{"index":
1}', "job-0-a": '{"index": 0}'}
+ )
+
+ result =
self.build_operator(do_xcom_push=True).execute_complete(self.build_context(),
event)
+
+ assert result == [{"index": 0}, {"index": 1}]
+
+ @pytest.mark.parametrize(
+ ("pod_names", "xcom_results"),
+ [
+ pytest.param(["read"], {"read": '{"a": 1}', "deleted": '{"b":
2}'}, id="succeeded-pod-deleted"),
+ pytest.param(["read", "not-read"], {"read": '{"a": 1}'},
id="xcom-of-succeeded-pod-not-read"),
+ ],
+ )
+ def
test_execute_complete_raises_when_xcom_of_a_succeeded_pod_is_missing(self,
pod_names, xcom_results):
+ self.set_pods([build_pod(name, base="terminated", phase="Succeeded")
for name in pod_names])
+ event = build_job_event(build_job(False, "Complete", succeeded=2),
xcom_results)
+
+ with pytest.raises(XComRetrievalError, match="Read the XComs of only 1
of the 2 pods that succeeded"):
+
self.build_operator(do_xcom_push=True).execute_complete(self.build_context(),
event)
+ self.mock_pod_manager.fetch_requested_container_logs.assert_called()
diff --git
a/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_kueue.py
b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_kueue.py
new file mode 100644
index 00000000000..10bc38ba47f
--- /dev/null
+++
b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_kueue.py
@@ -0,0 +1,308 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import json
+from contextlib import nullcontext
+from unittest import mock
+
+import pytest
+import tenacity
+from kubernetes.client import CoreV1Api, models as k8s
+from kubernetes.client.rest import ApiException
+
+from airflow.providers.cncf.kubernetes.hooks.kubernetes import
AsyncKubernetesHook, KubernetesHook
+from airflow.providers.cncf.kubernetes.triggers.kueue import (
+ KubernetesKueueJobTrigger,
+ KueueJobState,
+ build_job_event,
+ build_pod_label_selector,
+ collect_xcom_results,
+ get_job_state,
+ list_job_pods,
+)
+from airflow.providers.cncf.kubernetes.utils.pod_manager import PodManager,
XComRetrievalError
+from airflow.triggers.base import TriggerEvent
+
+TRIGGER_PATH = "airflow.providers.cncf.kubernetes.triggers.kueue.{}"
+JOB_NAME = "job-test"
+NAMESPACE = "team-a"
+JOB_UID = "job-uid"
+POD_LABEL_SELECTOR = "batch.kubernetes.io/controller-uid=job-uid"
+TRIGGER_KWARGS = {
+ "job_name": JOB_NAME,
+ "job_namespace": NAMESPACE,
+ "base_container_name": "base",
+ "kubernetes_conn_id": "kubernetes_default",
+ "poll_interval": 0,
+ "cluster_context": "cluster",
+ "config_file": "/path/to/config",
+ "in_cluster": False,
+ "do_xcom_push": True,
+}
+
+
+def build_job(suspend: bool, condition: str | None = None, succeeded: int |
None = None) -> k8s.V1Job:
+ conditions = (
+ [k8s.V1JobCondition(type=condition, status="True",
reason="BackoffLimitExceeded")]
+ if condition
+ else []
+ )
+ return k8s.V1Job(
+ metadata=k8s.V1ObjectMeta(name=JOB_NAME, namespace=NAMESPACE,
uid=JOB_UID),
+ spec=k8s.V1JobSpec(
+ suspend=suspend,
+ template=k8s.V1PodTemplateSpec(),
+
selector=k8s.V1LabelSelector(match_labels={"batch.kubernetes.io/controller-uid":
"job-uid"}),
+ ),
+ status=k8s.V1JobStatus(conditions=conditions, succeeded=succeeded),
+ )
+
+
+def build_pod(
+ name: str,
+ base: str | None = None,
+ sidecar: str | None = None,
+ phase: str = "Running",
+ owner_uid: str | None = JOB_UID,
+) -> k8s.V1Pod:
+ """
+ Build a pod of the Job, with its base and XCom sidecar containers
"running", "terminated" or None.
+
+ ``owner_uid`` is the uid of the Job that owns the pod, None for a pod
without owner.
+ """
+ owner_references = (
+ [k8s.V1OwnerReference(api_version="batch/v1", kind="Job",
name=JOB_NAME, uid=owner_uid)]
+ if owner_uid
+ else None
+ )
+ statuses = [
+ k8s.V1ContainerStatus(
+ name=container_name,
+ image="",
+ image_id="",
+ ready=state == "running",
+ restart_count=0,
+ state=k8s.V1ContainerState(
+ running=k8s.V1ContainerStateRunning() if state == "running"
else None,
+ terminated=k8s.V1ContainerStateTerminated(exit_code=0) if
state == "terminated" else None,
+ ),
+ )
+ for container_name, state in (("base", base), ("airflow-xcom-sidecar",
sidecar))
+ if state
+ ]
+ return k8s.V1Pod(
+ metadata=k8s.V1ObjectMeta(
+ name=name,
+ namespace=NAMESPACE,
+ labels={"try_number": "1", "batch.kubernetes.io/controller-uid":
"job-uid"},
+ owner_references=owner_references,
+ ),
+ status=k8s.V1PodStatus(phase=phase, container_statuses=statuses),
+ )
+
+
[email protected](
+ ("job", "expected_state"),
+ [
+ (build_job(True), KueueJobState.SUSPENDED),
+ (build_job(False), KueueJobState.RUNNING),
+ (build_job(False, "Complete"), KueueJobState.FINISHED),
+ (build_job(False, "Failed"), KueueJobState.FINISHED),
+ ],
+ ids=["suspended", "running", "complete", "failed"],
+)
+def test_get_job_state(job, expected_state):
+ assert get_job_state(job) == expected_state
+
+
[email protected](
+ ("selector", "expected_label_selector"),
+ [
+ pytest.param(
+
k8s.V1LabelSelector(match_labels={"batch.kubernetes.io/controller-uid":
"job-uid"}),
+ POD_LABEL_SELECTOR,
+ id="generated-selector",
+ ),
+ pytest.param(
+ k8s.V1LabelSelector(
+ match_labels={"team": "a", "app": "train"},
+ match_expressions=[
+ k8s.V1LabelSelectorRequirement(key="tier", operator="In",
values=["gpu", "cpu"]),
+ k8s.V1LabelSelectorRequirement(key="env",
operator="NotIn", values=["dev"]),
+ k8s.V1LabelSelectorRequirement(key="run",
operator="Exists"),
+ k8s.V1LabelSelectorRequirement(key="skip",
operator="DoesNotExist"),
+ ],
+ ),
+ "app=train,team=a,tier in (gpu,cpu),env notin (dev),run,!skip",
+ id="manual-selector",
+ ),
+ ],
+)
+def test_build_pod_label_selector(selector, expected_label_selector):
+ job = k8s.V1Job(spec=k8s.V1JobSpec(template=k8s.V1PodTemplateSpec(),
selector=selector))
+
+ assert build_pod_label_selector(job) == expected_label_selector
+
+
+class TestCollectXcomResults:
+ def setup_method(self):
+ self.client = mock.create_autospec(CoreV1Api, instance=True)
+ self.pod_manager = mock.create_autospec(PodManager, instance=True)
+
+ def
test_reads_only_new_pods_with_terminated_base_and_running_sidecar(self):
+ ready = build_pod("ready", base="terminated", sidecar="running")
+ self.client.list_namespaced_pod.return_value = k8s.V1PodList(
+ items=[
+ build_pod("pending"),
+ build_pod("running", base="running", sidecar="running"),
+ ready,
+ build_pod("already-read", base="terminated",
sidecar="running"),
+ ]
+ )
+ self.pod_manager.extract_xcom.return_value = '{"a": 1}'
+ xcom_results = {"already-read": "{}"}
+
+ collect_xcom_results(self.client, self.pod_manager, build_job(False),
"base", xcom_results)
+
+ self.client.list_namespaced_pod.assert_called_once_with(
+ namespace=NAMESPACE, label_selector=POD_LABEL_SELECTOR
+ )
+ self.pod_manager.extract_xcom.assert_called_once_with(ready,
ignore_kill_failure=False)
+ assert xcom_results == {"already-read": "{}", "ready": '{"a": 1}'}
+
+ @pytest.mark.parametrize(
+ ("error", "read_pod_error", "expectation"),
+ [
+ (XComRetrievalError("sidecar stopped"), None, nullcontext()),
+ (ApiException(status=404), None, nullcontext()),
+ (ApiException(status=0), ApiException(status=404), nullcontext()),
+ (ApiException(status=0), None, pytest.raises(ApiException)),
+ (ApiException(status=403), None, pytest.raises(ApiException)),
+ (json.JSONDecodeError("Unterminated string", '{"a', 1), None,
nullcontext()),
+ ],
+ ids=[
+ "sidecar-stopped",
+ "pod-deleted",
+ "exec-failed-on-deleted-pod",
+ "exec-failed-on-existing-pod",
+ "other-api-error",
+ "partial-xcom-json",
+ ],
+ )
+ def test_skips_pods_stopped_or_deleted_by_preemption(self, error,
read_pod_error, expectation):
+ self.client.list_namespaced_pod.return_value = k8s.V1PodList(
+ items=[build_pod("preempted", base="terminated",
sidecar="running")]
+ )
+ self.client.read_namespaced_pod.side_effect = read_pod_error
+ self.pod_manager.extract_xcom.side_effect = error
+ xcom_results: dict[str, str] = {}
+
+ with expectation:
+ collect_xcom_results(self.client, self.pod_manager,
build_job(False), "base", xcom_results)
+ assert xcom_results == {}
+
+
[email protected](list_job_pods.retry, "wait", tenacity.wait_none())
+def test_list_job_pods_retries_transient_api_errors():
+ client = mock.create_autospec(CoreV1Api, instance=True)
+ pods = [build_pod("pod")]
+ client.list_namespaced_pod.side_effect = [ApiException(status=503),
k8s.V1PodList(items=pods)]
+
+ assert list_job_pods(client, NAMESPACE, POD_LABEL_SELECTOR, JOB_UID) ==
pods
+ assert client.list_namespaced_pod.call_count == 2
+
+
+def test_list_job_pods_keeps_only_pods_owned_by_the_job():
+ """
+ With manualSelector, a selector that is not unique may also match the pods
of other Jobs, or
+ orphan pods the Job controller has not adopted yet.
+ """
+ client = mock.create_autospec(CoreV1Api, instance=True)
+ owned = build_pod("owned")
+ client.list_namespaced_pod.return_value = k8s.V1PodList(
+ items=[owned, build_pod("other-job", owner_uid="other-job-uid"),
build_pod("orphan", owner_uid=None)]
+ )
+
+ assert list_job_pods(client, NAMESPACE, POD_LABEL_SELECTOR, JOB_UID) ==
[owned]
+ client.list_namespaced_pod.assert_called_once_with(namespace=NAMESPACE,
label_selector=POD_LABEL_SELECTOR)
+
+
+class TestKubernetesKueueJobTrigger:
+ def test_serialize(self):
+ assert KubernetesKueueJobTrigger(**TRIGGER_KWARGS).serialize() == (
+
"airflow.providers.cncf.kubernetes.triggers.kueue.KubernetesKueueJobTrigger",
+ TRIGGER_KWARGS,
+ )
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(
+ ("do_xcom_push", "expected_collect_calls", "expected_xcom_result"),
+ [
+ pytest.param(True, 2, {"pod": "{}"}, id="xcom-read-while-running"),
+ pytest.param(False, 0, {}, id="without-xcom"),
+ ],
+ )
+ @mock.patch(TRIGGER_PATH.format("collect_xcom_results"), autospec=True)
+ @mock.patch(
+ TRIGGER_PATH.format("KubernetesKueueJobTrigger.sync_hook"),
+ new_callable=lambda: mock.create_autospec(KubernetesHook,
instance=True),
+ )
+ @mock.patch(
+ TRIGGER_PATH.format("KubernetesKueueJobTrigger.hook"),
+ new_callable=lambda: mock.create_autospec(AsyncKubernetesHook,
instance=True),
+ )
+ async def test_run_waits_through_preemption(
+ self,
+ mock_hook,
+ mock_sync_hook,
+ mock_collect,
+ do_xcom_push,
+ expected_collect_calls,
+ expected_xcom_result,
+ ):
+ """Kueue admits the Job, preempts it, admits it again, then the Job
completes."""
+ jobs = [
+ build_job(True),
+ build_job(False),
+ build_job(True),
+ build_job(False),
+ build_job(False, "Complete"),
+ ]
+ mock_hook.get_job_status.side_effect = jobs
+
+ def read_xcom(client, pod_manager, job, base_container_name,
xcom_results):
+ xcom_results["pod"] = "{}"
+
+ mock_collect.side_effect = read_xcom
+ trigger = KubernetesKueueJobTrigger(**TRIGGER_KWARGS)
+ trigger.do_xcom_push = do_xcom_push
+
+ events = [event async for event in trigger.run()]
+
+ assert events == [TriggerEvent(build_job_event(jobs[-1],
expected_xcom_result))]
+ mock_hook.get_job_status.assert_awaited_with(name=JOB_NAME,
namespace=NAMESPACE)
+ assert mock_collect.call_count == expected_collect_calls
+
+ @pytest.mark.asyncio
+ @mock.patch(TRIGGER_PATH.format("AsyncKubernetesHook.close"),
autospec=True)
+ async def test_cleanup_closes_hook(self, mock_close):
+ trigger = KubernetesKueueJobTrigger(**TRIGGER_KWARGS)
+
+ await trigger.cleanup()
+
+ mock_close.assert_awaited_once_with(trigger.hook)