Hi Airflow community, We'd like to propose adding a new community provider for Apache Polaris: apache-airflow-providers-apache-polaris.
Integration overview --------------------------- Apache Polaris (https://polaris.apache.org/) is an open-source catalog for Apache Iceberg. It implements the Iceberg REST catalog API and adds centralized access control, credential vending and catalog-level policies. Many teams running Iceberg already orchestrate their data platform with Airflow. Today, managing Polaris from a Dag means hand-written REST calls or shelling out to the Polaris CLI. The proposed provider would add: - A "polaris" connection type. It supports Polaris's internal OAuth tokens, and it masks every token it fetches. - Hooks for the Polaris Management API and the catalog APIs. They can also hand ready-made catalog properties to pyiceberg or to Spark and other engines. - Operators for catalogs, namespaces, tables, principals, principal and catalog roles, grants, and policies. Typical uses include provisioning a team's catalog with its roles and grants, onboarding a pipeline principal, and managing table lifecycle. - A helper that turns Polaris table-maintenance policies (compaction, snapshot expiry, orphan-file removal) into Iceberg Spark procedure calls, so a Dag can execute the maintenance a policy defines. It is built on the official Apache Polaris Python SDK (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and pyiceberg is an optional extra. It has been tested against Polaris server 1.8 and 1.9, and installs cleanly against Airflow's constraints for every 3.1.1–3.4.0 release. System tests ----------------- Polaris is open source and runs as a single container, so no external paid service is involved. The PR adds: - a Breeze integration (breeze --integration polaris) that starts Polaris 1.9.0 in CI, with integration tests against it; - system-test example Dags for provisioning, tables and policies, which run against the same server. All tests can run in Airflow's own CI, so no external dashboard is needed. Proposed stewards ------------------------- - Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer) - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris PMC/Committer) - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow PMC/Committer) Sponsoring committer ----------------------------- - Xiaodong Deng, "XD-DENG", [email protected] Working implementation -------------------------------- Draft PR: https://github.com/apache/airflow/pull/77188. It includes unit, integration and system tests, and how-to docs for every operator. Incubation commitment ------------------------------- We commit to: - maintaining the provider and responding to issues within a reasonable time; - meeting the incubation health metrics within 6 months; - participating in quarterly governance updates. Regards, XD
