GitHub user imishchuk-tsgs edited a discussion: Can't launch cross-namespace
DAG with Kubernetes Pod Operator
Hey, folks.
We have an Airflow `2.10.2` deployed in EKS.
EKS has two namespaces of interest `data` and `ai` and Airflow webserver,
scheduler and workers by default run in `data` namespace.
I want to execute some specific DAGs in `ai` namespace. I've added following
configuration to the Airflow
```
[kubernetes_executor]
...
multi_namespace_mode = True
multi_namespace_mode_namespace_list = data,ai
...
```
DAG is a placeholder for now
```
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import
KubernetesPodOperator
from include.constants import EKSNamespaces, EKSServiceAccounts
from include.utils import get_default_dag_args
DAG_NAME = "ai_test"
with DAG(
dag_id=DAG_NAME,
default_args=get_default_dag_args(namespace=EKSNamespaces.AI.value), # "ai"
schedule=None,
start_date=datetime(2024, 12, 3),
tags=["test"],
) as dag:
KubernetesPodOperator(
name=DAG_NAME,
image="busybox:latest",
arguments=["sh", "-c", "echo hello && sleep 300"],
task_id=DAG_NAME,
service_account_name=EKSServiceAccounts.DATA_AIRFLOW_WORKER.value,
"data-airflow-worker"
)
```
Service account `data-airflow-worker` has following permissions in `ai`
namespace
```
airflow_serviceaccounts = [
"data-airflow-migrate-database-job",
"data-airflow-reconciler",
"data-airflow-scheduler",
"data-airflow-webserver",
"data-airflow-worker",
]
resource "kubernetes_role" "ai_airflow" {
metadata {
name = "ai-airflow"
namespace = kubernetes_namespace.ai.id
}
rule {
api_groups = [""]
resources = [
"pods",
"jobs",
]
verbs = [
"get",
"list",
"create",
"update",
"delete",
"watch",
"patch",
]
}
}
resource "kubernetes_role_binding" "ai_airflow" {
for_each = toset(local.airflow_serviceaccounts)
metadata {
name = "ai-${each.key}"
namespace = kubernetes_namespace.ai.id
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "Role"
name = kubernetes_role.ai_airflow.metadata[0].name
}
subject {
kind = "ServiceAccount"
name = each.key
namespace = kubernetes_namespace.data.id
}
}
```
When I trigger the DAG, KPO pod is started in `data` namespace and then fails
with following error
```
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "pods \"ai-test-kxotpx8z\" is forbidden: error looking up service
account ai/data-airflow-worker: serviceaccount \"data-airflow-worker\" not
found",
"reason": "Forbidden",
"details": {
"name": "ai-test-kxotpx8z",
"kind": "pods"
},
"code": 403
}
```
`data-airflow-worker` service account exists only in `data` namespace but has
necessary permissions to work with `ai` namespace.
It is unclear for me, why KPO pod is looking for `data-airflow-worker` in `ai`
namespace and how to make sure that it uses `data-airflow-worker` in `data`
namespace instead.
Obvious thing to do would be to create a service account in `ai` namespace, but
that feels wrong, as cross-namespace operations should be supported.
Can you, please, suggest, what am I missing?
Thank you.
GitHub link: https://github.com/apache/airflow/discussions/44730
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
