+1

On Tue, Nov 17, 2026 at 7:41 PM Yufei Gu <[email protected]> wrote:

> +1, this is very cool. Thanks for adding this, XD!
>
> One use case that comes to mind is Iceberg table maintenance. If you run
> Iceberg tables in production, you need to do things like compact data files
> and expire old snapshots. Native support in Airflow and Polaris would make
> it easier to schedule those jobs as part of your existing Airflow
> workflows.
>
> A bit about me for those who don’t know me. I’m a PMC member of both Apache
> Polaris and Apache Iceberg. I also wrote the policy store and table
> maintenance features in Polaris.
> Yufei
>
>
> On Tue, Nov 17, 2026 at 2:16 PM Jarek Potiuk <[email protected]> wrote:
>
> > +1
> >
> > On Wed, Nov 18, 2026 at 9:01 AM constance.astronomer.io via dev <
> > [email protected]> wrote:
> >
> >> +1 love it
> >>
> >> > On Nov 17, 2026, at 12:03 PM, Vikram Koka via dev <
> >> [email protected]> wrote:
> >> >
> >> > +1
> >> > Seems like a great addition
> >> >
> >> > Vikram
> >> >
> >> >
> >> >> On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]>
> >> wrote:
> >> >>
> >> >> +1 Would love to have polaris support.
> >> >>
> >> >>
> >> >> On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev <
> >> >> [email protected]>
> >> >> wrote:
> >> >>
> >> >>> Hi Airflow community,
> >> >>>
> >> >>> We'd like to propose adding a new community provider for Apache
> >> Polaris:
> >> >>> apache-airflow-providers-apache-polaris.
> >> >>>
> >> >>> Integration overview
> >> >>> ---------------------------
> >> >>> Apache Polaris (https://polaris.apache.org/) is an open-source
> >> catalog
> >> >> for
> >> >>> Apache Iceberg. It implements the Iceberg REST catalog API and adds
> >> >>> centralized access control, credential vending and catalog-level
> >> >> policies.
> >> >>> Many teams running Iceberg already orchestrate their data platform
> >> with
> >> >>> Airflow. Today, managing Polaris from a Dag means hand-written REST
> >> calls
> >> >>> or shelling out to the Polaris CLI.
> >> >>>
> >> >>> The proposed provider would add:
> >> >>>
> >> >>> - A "polaris" connection type. It supports Polaris's internal OAuth
> >> >> tokens,
> >> >>> and it masks every token it fetches.
> >> >>> - Hooks for the Polaris Management API and the catalog APIs. They
> can
> >> >> also
> >> >>> hand ready-made catalog properties to pyiceberg or to Spark and
> other
> >> >>> engines.
> >> >>> - Operators for catalogs, namespaces, tables, principals, principal
> >> and
> >> >>> catalog roles, grants, and policies. Typical uses include
> >> provisioning a
> >> >>> team's catalog with its roles and grants, onboarding a pipeline
> >> >> principal,
> >> >>> and managing table lifecycle.
> >> >>> - A helper that turns Polaris table-maintenance policies
> (compaction,
> >> >>> snapshot expiry, orphan-file removal) into Iceberg Spark procedure
> >> calls,
> >> >>> so a Dag can execute the maintenance a policy defines.
> >> >>>
> >> >>> It is built on the official Apache Polaris Python SDK
> >> >>> (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+,
> and
> >> >>> pyiceberg is an optional extra. It has been tested against Polaris
> >> server
> >> >>> 1.8 and 1.9, and installs cleanly against Airflow's constraints for
> >> every
> >> >>> 3.1.1–3.4.0 release.
> >> >>>
> >> >>> System tests
> >> >>> -----------------
> >> >>> Polaris is open source and runs as a single container, so no
> external
> >> >> paid
> >> >>> service is involved. The PR adds:
> >> >>>
> >> >>> - a Breeze integration (breeze --integration polaris) that starts
> >> Polaris
> >> >>> 1.9.0 in CI, with integration tests against it;
> >> >>> - system-test example Dags for provisioning, tables and policies,
> >> which
> >> >> run
> >> >>> against the same server.
> >> >>>
> >> >>> All tests can run in Airflow's own CI, so no external dashboard is
> >> >> needed.
> >> >>>
> >> >>> Proposed stewards
> >> >>> -------------------------
> >> >>> - Yufei Gu, "flyrain", [email protected] (Apache Polaris
> >> PMC/Committer)
> >> >>> - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris
> >> >>> PMC/Committer)
> >> >>> - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow
> >> >>> PMC/Committer)
> >> >>>
> >> >>> Sponsoring committer
> >> >>> -----------------------------
> >> >>> - Xiaodong Deng, "XD-DENG", [email protected]
> >> >>>
> >> >>> Working implementation
> >> >>> --------------------------------
> >> >>> Draft PR: https://github.com/apache/airflow/pull/77188. It includes
> >> >> unit,
> >> >>> integration and system tests, and how-to docs for every operator.
> >> >>>
> >> >>> Incubation commitment
> >> >>> -------------------------------
> >> >>> We commit to:
> >> >>> - maintaining the provider and responding to issues within a
> >> reasonable
> >> >>> time;
> >> >>> - meeting the incubation health metrics within 6 months;
> >> >>> - participating in quarterly governance updates.
> >> >>>
> >> >>>
> >> >>> Regards,
> >> >>>
> >> >>> XD
> >> >>>
> >> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [email protected]
> >> For additional commands, e-mail: [email protected]
> >>
> >>
>

Reply via email to