jason810496 opened a new pull request, #71154: URL: https://github.com/apache/airflow/pull/71154
## Why Ray is one of the most important AI Infra nowadays but deprecated in https://github.com/astronomer/astro-provider-ray. The existing Ray integration lives outside the Apache Airflow repository and uses Airflow 2 APIs. Bringing it into the community provider ecosystem makes Ray cluster and job orchestration available through supported Airflow 3 interfaces. ## How - Add the `apache-airflow-providers-ray` distribution with provider metadata, packaging, documentation, and workspace registration. - Port Ray connection handling, Kubernetes cluster lifecycle operations, job submission operators, deferred triggers, and the `@task.ray` decorator. - Migrate Dags and implementation code to public Airflow 3 SDK and provider APIs. - Add unit coverage for hooks, operators, triggers, decorators, timeout rendering, failure cleanup, and cluster cleanup. - Add system Dags and KubeRay manifests for existing clusters, dynamically created clusters, and GPU workloads. ## Verification - `uv run --project providers/ray --no-sync pytest providers/ray/tests/unit -q` (`113 passed`) - `prek run --files $(git diff --name-only upstream/main...HEAD)` - `breeze run mypy $(find providers/ray/src -type f -name '*.py' ! -name '__init__.py' | sort)` - Loaded all `providers/ray/tests/system/ray` Dags with Airflow 3 `DagBag` and confirmed no import errors. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes, with help of Codex GPT-5 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
