GitHub user itzmitto added a comment to the discussion: Custom worker images to lower the number of kubernetespodoperator tasks
Hallo @karjaljo, Yes, this is possible with CeleryExecutor. You can keep separate worker images for different applications rather than install everything into one image. For example: - Build an Airflow worker image containing application A and its dependencies. - Run a persistent worker group listening to an app_a queue with `airflow celery worker -q app_a`. - Run that application's commands through ordinary BashOperator tasks assigned to `queue="app_a"`. You can repeat that setup for another application and queue 😀 Those tasks execute on the existing Celery workers, avoiding a separate KubernetesPodOperator pod for each application invocation. Each image still needs Airflow and the appropriate configuration and DAG access; an application-only image isn't enough. The documented approach is here: https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#using-multiple-docker-images-and-celery-queues Queue routing: https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html#queues The trade-off is maintaining multiple worker deployments. The docs warn about the extra configuration and resource-management complexity. Also, KubernetesExecutor still creates a pod per task, so changing operators alone wouldn't remove that overhead if you're using it. Which Airflow version and executor are you deploying? Hope this helps! GitHub link: https://github.com/apache/airflow/discussions/76535#discussioncomment-19416369 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
