GitHub user raphaelauv edited a comment on the discussion: Custom worker images to lower the number of kubernetespodoperator tasks
using airflow celery workers will limit what you can run with pythonoperator or you will need to use dockeroperator ... so you should go for a new operator , ask an LLM this : ``` In Airflow 3.3, I want something like KubernetesPodOperator, but that reuses existing containers instead of making Kubernetes create a new pod for every task. Please write a custom operator that: - runs each task's command inside an already-running pod (like kubectl exec), with logs streamed live and the task failing if the command fails - starts a new pod if none exists, or if all existing pods are busy with another task (one task per pod at a time, with a maximum number of pods) - makes sure two tasks can never pick the same pod, and that a pod held by a crashed worker becomes reusable after a while - lets Kubernetes clean up pods automatically after a few minutes without use - creates the pods through Kubernetes Jobs with ttlSecondsAfterFinished, so Kubernetes deletes them by itself once they exit after being idle Use the Airflow 3 SDK (airflow.sdk) and the cncf.kubernetes provider. Give me the operator, an example DAG, the RBAC the service account needs, and the limitations of the design. ``` GitHub link: https://github.com/apache/airflow/discussions/76535#discussioncomment-19417454 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
