hterik commented on issue #19135: URL: https://github.com/apache/airflow/issues/19135#issuecomment-1025500528
I would advice you to look into the Airlflow Kubernetes Scheduler. If you already use Kubernetes as your main airflow scheduler you can simply use taskflow and have python functions running in kubernetes as default. If you don't use the scheduler but want to have _some_ tasks running in kubernetes anyway, there is still a lot of useful functions to reuse or take inspiration from in the scheduler code. For example washing dag ids to be compatible to kubernetes and adding them as labels and annotations on the Pod. See `construct_pod` function in `airflow/kubernetes/pod_generator.py` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
