GitHub user DYNOSuprovo added a comment to the discussion: Custom worker images to lower the number of kubernetespodoperator tasks
Moving high-frequency, short-lived tasks (running only a few seconds) to `KubernetesPodOperator` on OpenShift is a well-known anti-pattern that quickly degrades cluster health. Each `KubernetesPodOperator` execution incurs: 1. An `etcd` transactional write and `kube-apiserver` event churn. 2. Scheduler cycle latency. 3. CNI (`OVN-Kubernetes`) IPAM allocation and virtual interface creation. 4. Container runtime (`CRI-O`) sandbox creation and teardown. When you have bursts of 3-to-5-second tasks, the cluster spends **10x more time provisioning and tearing down pods than executing actual business logic**. Beyond the multiple Celery queue approach mentioned by @itzmitto, here are three battle-tested architectural patterns on OpenShift that eliminate pod churn while maintaining isolation: --- ### Pattern 1: Shift Short-Lived Jobs to OpenShift Serverless (Knative Services) via `HttpOperator` (Recommended) Instead of having Airflow launch pods directly through the Kubernetes API, package the short-lived applications as **OpenShift Serverless (Knative) services**: ``` [ Airflow Worker ] --(HTTP POST /fast-path)--> [ OpenShift Serverless / Knative Service ] ``` - **How it works**: The app is containerized as a lightweight microservice (e.g. FastAPI / Go / Quarkus). The DAG calls it using `HttpOperator` or `SimpleHttpOperator`. - **Why it solves the problem**: - **Zero Kubernetes API Churn**: The call is a standard L7 HTTP/gRPC request. `kube-apiserver` and `etcd` do not see a single pod creation. - **Warm Concurrency**: Knative keeps pods warm (`minScale: 1` or concurrency-based autoscaling). Response time drops from 10–15 seconds (pod lifecycle overhead) down to **sub-100ms**. - **Maintains Container Isolation**: Application dependencies remain completely isolated in their own container images without polluting Airflow worker images. --- ### Pattern 2: `CeleryKubernetesExecutor` (Hybrid Routing) If you already have Celery in place, use `CeleryKubernetesExecutor`: - Route high-churn, short-duration tasks to persistent, warm **Celery Workers** (`queue="default"`). - Route heavy, long-running, or resource-isolated batch processing to **KubernetesExecutor** (`queue="kubernetes"`). This gives you the best of both worlds without forcing every 3-second task through the Kubernetes scheduler. --- ### Pattern 3: Optimizing `KubernetesPodOperator` (If pods are mandatory) If security boundaries or compliance mandate that every execution must be a separate pod: 1. **Pre-warm Images with an OpenShift DaemonSet**: Ensure worker nodes never pull images during runtime. Deploy a lightweight `DaemonSet` on the worker nodes that pulls the target application image so `image_pull_policy="IfNotPresent"` is a cache hit. 2. **Dedicated Task Node Pool & Namespace**: Taint and dedicate a specific worker node pool for short-lived task pods to isolate `CRI-O` sandbox churn from the OpenShift control plane and core platform nodes. 3. **Tune `terminationGracePeriodSeconds` & Clean-up**: Set `termination_grace_period=0` and `is_delete_operator_pod=True` so CRI-O cleans up dead containers immediately rather than maintaining defunct state. GitHub link: https://github.com/apache/airflow/discussions/76535#discussioncomment-19495983 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
