+1 love it

> On Nov 17, 2026, at 12:03 PM, Vikram Koka via dev <[email protected]> 
> wrote:
> 
> +1
> Seems like a great addition
> 
> Vikram
> 
> 
>> On Tue, Nov 17, 2026 at 4:18 AM Kaxil Naik <[email protected]> wrote:
>> 
>> +1 Would love to have polaris support.
>> 
>> 
>> On Tue, 17 Nov 2026 at 04:18, Xiaodong Deng via dev <
>> [email protected]>
>> wrote:
>> 
>>> Hi Airflow community,
>>> 
>>> We'd like to propose adding a new community provider for Apache Polaris:
>>> apache-airflow-providers-apache-polaris.
>>> 
>>> Integration overview
>>> ---------------------------
>>> Apache Polaris (https://polaris.apache.org/) is an open-source catalog
>> for
>>> Apache Iceberg. It implements the Iceberg REST catalog API and adds
>>> centralized access control, credential vending and catalog-level
>> policies.
>>> Many teams running Iceberg already orchestrate their data platform with
>>> Airflow. Today, managing Polaris from a Dag means hand-written REST calls
>>> or shelling out to the Polaris CLI.
>>> 
>>> The proposed provider would add:
>>> 
>>> - A "polaris" connection type. It supports Polaris's internal OAuth
>> tokens,
>>> and it masks every token it fetches.
>>> - Hooks for the Polaris Management API and the catalog APIs. They can
>> also
>>> hand ready-made catalog properties to pyiceberg or to Spark and other
>>> engines.
>>> - Operators for catalogs, namespaces, tables, principals, principal and
>>> catalog roles, grants, and policies. Typical uses include provisioning a
>>> team's catalog with its roles and grants, onboarding a pipeline
>> principal,
>>> and managing table lifecycle.
>>> - A helper that turns Polaris table-maintenance policies (compaction,
>>> snapshot expiry, orphan-file removal) into Iceberg Spark procedure calls,
>>> so a Dag can execute the maintenance a policy defines.
>>> 
>>> It is built on the official Apache Polaris Python SDK
>>> (apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and
>>> pyiceberg is an optional extra. It has been tested against Polaris server
>>> 1.8 and 1.9, and installs cleanly against Airflow's constraints for every
>>> 3.1.1–3.4.0 release.
>>> 
>>> System tests
>>> -----------------
>>> Polaris is open source and runs as a single container, so no external
>> paid
>>> service is involved. The PR adds:
>>> 
>>> - a Breeze integration (breeze --integration polaris) that starts Polaris
>>> 1.9.0 in CI, with integration tests against it;
>>> - system-test example Dags for provisioning, tables and policies, which
>> run
>>> against the same server.
>>> 
>>> All tests can run in Airflow's own CI, so no external dashboard is
>> needed.
>>> 
>>> Proposed stewards
>>> -------------------------
>>> - Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer)
>>> - Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris
>>> PMC/Committer)
>>> - Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow
>>> PMC/Committer)
>>> 
>>> Sponsoring committer
>>> -----------------------------
>>> - Xiaodong Deng, "XD-DENG", [email protected]
>>> 
>>> Working implementation
>>> --------------------------------
>>> Draft PR: https://github.com/apache/airflow/pull/77188. It includes
>> unit,
>>> integration and system tests, and how-to docs for every operator.
>>> 
>>> Incubation commitment
>>> -------------------------------
>>> We commit to:
>>> - maintaining the provider and responding to issues within a reasonable
>>> time;
>>> - meeting the incubation health metrics within 6 months;
>>> - participating in quarterly governance updates.
>>> 
>>> 
>>> Regards,
>>> 
>>> XD
>>> 
>> 

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to