kevinmingtarja opened a new issue, #53895: URL: https://github.com/apache/airflow/issues/53895
### Description Hi! We are the team behind [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running AI and batch workloads across any infra. We have an existing guide on how to orchestrate SkyPilot tasks using Airflow: https://docs.skypilot.co/en/latest/examples/orchestrators/airflow.html But it involves the user writing a lot of boilerplate code just to run a simple SkyPilot task. Recently, we've been getting some requests to simplify this process, so we thought it would be beneficial to package the logic there as an Airflow operator. ### Use case/motivation Some of our users use Airflow for orchestrating SkyPilot tasks. So they would define their jobs with SkyPilot (i.e. running a python script for data preprocessing) and SkyPilot would handle the provisioning of nodes, environment setup, and the running of the jobs. And airflow would handle the scheduling (i.e. make it recurring every 12h), data flow between different tasks, logging, monitoring, etc. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
