GitHub user kevinmingtarja created a discussion: Support for SkyPilot provider in Airflow
### Description Hi! We are the team behind [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running AI and batch workloads across any infra. We have an existing guide on how to orchestrate SkyPilot tasks using Airflow: https://docs.skypilot.co/en/latest/examples/orchestrators/airflow.html But it involves the user writing a lot of boilerplate code just to run a simple SkyPilot task. Recently, we've been getting some requests to simplify this process, so we thought it would be beneficial to package the logic there as an Airflow operator. ### Use case/motivation Some of our users use Airflow for orchestrating SkyPilot tasks. So they would define their jobs with SkyPilot (i.e. running a python script for data preprocessing) and SkyPilot would handle the provisioning of nodes, environment setup, and the running of the jobs. And airflow would handle the scheduling (i.e. make it recurring every 12h), data flow between different tasks, logging, monitoring, etc. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) GitHub link: https://github.com/apache/airflow/discussions/53925 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
