+1

On Wed, Nov 18, 2026 at 9:01 AM constance.astronomer.io via dev <
[email protected]> wrote:

> +1 love it
>
> > On Nov 17, 2026, at 12:03 PM, Vikram Koka via dev <
> [email protected]> wrote:
> >
> > +1
> > Seems like a great addition
> >
> > Vikram
> >
> >
> >> On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]> wrote:
> >>
> >> +1 Would love to have polaris support.
> >>
> >>
> >> On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev <
> >> [email protected]>
> >> wrote:
> >>
> >>> Hi Airflow community,
> >>>
> >>> We'd like to propose adding a new community provider for Apache
> Polaris:
> >>> apache-airflow-providers-apache-polaris.
> >>>
> >>> Integration overview
> >>> ---------------------------
> >>> Apache Polaris (https://polaris.apache.org/) is an open-source catalog
> >> for
> >>> Apache Iceberg. It implements the Iceberg REST catalog API and adds
> >>> centralized access control, credential vending and catalog-level
> >> policies.
> >>> Many teams running Iceberg already orchestrate their data platform with
> >>> Airflow. Today, managing Polaris from a Dag means hand-written REST
> calls
> >>> or shelling out to the Polaris CLI.
> >>>
> >>> The proposed provider would add:
> >>>
> >>> - A "polaris" connection type. It supports Polaris's internal OAuth
> >> tokens,
> >>> and it masks every token it fetches.
> >>> - Hooks for the Polaris Management API and the catalog APIs. They can
> >> also
> >>> hand ready-made catalog properties to pyiceberg or to Spark and other
> >>> engines.
> >>> - Operators for catalogs, namespaces, tables, principals, principal and
> >>> catalog roles, grants, and policies. Typical uses include provisioning
> a
> >>> team's catalog with its roles and grants, onboarding a pipeline
> >> principal,
> >>> and managing table lifecycle.
> >>> - A helper that turns Polaris table-maintenance policies (compaction,
> >>> snapshot expiry, orphan-file removal) into Iceberg Spark procedure
> calls,
> >>> so a Dag can execute the maintenance a policy defines.
> >>>
> >>> It is built on the official Apache Polaris Python SDK
> >>> (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and
> >>> pyiceberg is an optional extra. It has been tested against Polaris
> server
> >>> 1.8 and 1.9, and installs cleanly against Airflow's constraints for
> every
> >>> 3.1.1–3.4.0 release.
> >>>
> >>> System tests
> >>> -----------------
> >>> Polaris is open source and runs as a single container, so no external
> >> paid
> >>> service is involved. The PR adds:
> >>>
> >>> - a Breeze integration (breeze --integration polaris) that starts
> Polaris
> >>> 1.9.0 in CI, with integration tests against it;
> >>> - system-test example Dags for provisioning, tables and policies, which
> >> run
> >>> against the same server.
> >>>
> >>> All tests can run in Airflow's own CI, so no external dashboard is
> >> needed.
> >>>
> >>> Proposed stewards
> >>> -------------------------
> >>> - Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer)
> >>> - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris
> >>> PMC/Committer)
> >>> - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow
> >>> PMC/Committer)
> >>>
> >>> Sponsoring committer
> >>> -----------------------------
> >>> - Xiaodong Deng, "XD-DENG", [email protected]
> >>>
> >>> Working implementation
> >>> --------------------------------
> >>> Draft PR: https://github.com/apache/airflow/pull/77188. It includes
> >> unit,
> >>> integration and system tests, and how-to docs for every operator.
> >>>
> >>> Incubation commitment
> >>> -------------------------------
> >>> We commit to:
> >>> - maintaining the provider and responding to issues within a reasonable
> >>> time;
> >>> - meeting the incubation health metrics within 6 months;
> >>> - participating in quarterly governance updates.
> >>>
> >>>
> >>> Regards,
> >>>
> >>> XD
> >>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>

Reply via email to