teaglebuilt opened a new issue #16135: URL: https://github.com/apache/airflow/issues/16135
**Description** Run an entire dag on the kubernetes pod operator. There is a label called `reattach on restart` that designates the task back to the pod for execution in the event of a schedule failure, but it would be nice to have this feature in general for all tasks. **Use case / motivation** We might want to launch a pod to run an isolated environment for a dagrun but not want a pod spinned up for every task. Its possible that a pipeline might be executing remote processes and not in need of heavy computing resources. So if a pod could be launched as a dagrun, this would help provide multi tenant solutions. Almost like a sub dag operator, but with using the Kubernetes Pod Operator. Kedro's Airflow implementation is closer to this concept by using it to launch a pod per node in the pipeline, but that is still representing it a task / step level. **Are you willing to submit a PR?** I am willing to contribute to this effort, if deemed so. **Related Issues** No, although i am wondering if this would cause bottle necks with the scheduler, but it can scale now... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
