+1 Would love to have polaris support.

On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev <[email protected]>
wrote:

> Hi Airflow community,
>
> We'd like to propose adding a new community provider for Apache Polaris:
> apache-airflow-providers-apache-polaris.
>
> Integration overview
> ---------------------------
> Apache Polaris (https://polaris.apache.org/) is an open-source catalog for
> Apache Iceberg. It implements the Iceberg REST catalog API and adds
> centralized access control, credential vending and catalog-level policies.
> Many teams running Iceberg already orchestrate their data platform with
> Airflow. Today, managing Polaris from a Dag means hand-written REST calls
> or shelling out to the Polaris CLI.
>
> The proposed provider would add:
>
> - A "polaris" connection type. It supports Polaris's internal OAuth tokens,
> and it masks every token it fetches.
> - Hooks for the Polaris Management API and the catalog APIs. They can also
> hand ready-made catalog properties to pyiceberg or to Spark and other
> engines.
> - Operators for catalogs, namespaces, tables, principals, principal and
> catalog roles, grants, and policies. Typical uses include provisioning a
> team's catalog with its roles and grants, onboarding a pipeline principal,
> and managing table lifecycle.
> - A helper that turns Polaris table-maintenance policies (compaction,
> snapshot expiry, orphan-file removal) into Iceberg Spark procedure calls,
> so a Dag can execute the maintenance a policy defines.
>
> It is built on the official Apache Polaris Python SDK
> (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and
> pyiceberg is an optional extra. It has been tested against Polaris server
> 1.8 and 1.9, and installs cleanly against Airflow's constraints for every
> 3.1.1–3.4.0 release.
>
> System tests
> -----------------
> Polaris is open source and runs as a single container, so no external paid
> service is involved. The PR adds:
>
> - a Breeze integration (breeze --integration polaris) that starts Polaris
> 1.9.0 in CI, with integration tests against it;
> - system-test example Dags for provisioning, tables and policies, which run
> against the same server.
>
> All tests can run in Airflow's own CI, so no external dashboard is needed.
>
> Proposed stewards
> -------------------------
> - Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer)
> - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris
> PMC/Committer)
> - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow
> PMC/Committer)
>
> Sponsoring committer
> -----------------------------
> - Xiaodong Deng, "XD-DENG", [email protected]
>
> Working implementation
> --------------------------------
> Draft PR: https://github.com/apache/airflow/pull/77188. It includes unit,
> integration and system tests, and how-to docs for every operator.
>
> Incubation commitment
> -------------------------------
> We commit to:
> - maintaining the provider and responding to issues within a reasonable
> time;
> - meeting the incubation health metrics within 6 months;
> - participating in quarterly governance updates.
>
>
> Regards,
>
> XD
>

Reply via email to