Glad to see the traction, thanks for weighing in everyone! > Is this based on the Astronomer one? https://github.com/astronomer/airflow-provider-duckdb
No code was based on that provider. It hasn't been updated in 3 years and it focused on a minimal set of DuckDB features. There wasn't much to reuse there. On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <[email protected]> wrote: > +10 > > On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <[email protected]> wrote: > > > Sounds good! > > > > On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <[email protected]> wrote: > > > > > Hi folks, > > > > > > I'd like to propose adding a new community provider for DuckDB. > > > > > > Integration overview: > > > --------------------- > > > > > > DuckDB is an in-process analytical (OLAP) database. Unlike most SQL > > > providers there is no server to connect to: queries run inside the task > > > process, which makes it a cheap way to run an analytical transform that > > > doesn't justify standing up a cluster. It reads and writes Parquet, CSV > > and > > > JSON directly from object storage, so it fits naturally between an > > extract > > > and a load step in a > > > Dag. > > > > > > It also fits Airflow's existing shape well: The proposed DuckDB > provider > > > speaks DbAPI, so it plugs into the common-sql ecosystem rather than > > needing > > > new machinery, and it fits in the data-processing/ETL space. DuckDB > Labs > > > was acquired by Amazon but DuckDB remains MIT licensed and controlled > by > > > the DuckDB Foundation. It is usable with no account, no signup and no > > > hosted service. > > > > > > The provider exposes a DuckDBHook extending DbApiHook, so it works with > > > SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus > a > > > DuckDBExecuteQueryOperator and a duckdb connection type. The connection > > is > > > optional: with none configured the hook opens an in-memory database. > > > > > > Provider specific integrations would live in their relevant provider > > > package (e.g. for AWS, an AwsDuckDBHook in > > apache-airflow-providers-amazon > > > for credential management). > > > > > > System tests: > > > ------------- > > > > > > DuckDB runs in-process with no live external service, so system tests > are > > > not > > > required for this provider; the unit tests exercise a real in-memory > > > DuckDB. > > > An AWS-specific system test could be added in the Amazon provider > > package, > > > if folks think that's required. > > > > > > Proposed stewards: > > > ------------------ > > > > > > Myself and the AWS team plan to sponsor and steward this package, you > > know > > > us and our track record well :) > > > > > > - Niko Oliveira (@o-nikolas) > > > - Vincent Beck (@vincbeck) > > > > > > Working implementation > > > ---------------------- > > > > > > https://github.com/apache/airflow/pull/72845 > > > > > > Incubation commitment > > > --------------------- > > > > > > We commit to: > > > > > > - Maintaining the provider and responding to issues within a reasonable > > > time > > > - Meeting the incubation health metrics within 6 months > > > - Participating in quarterly governance updates > > > > > > Cheers, > > > Niko > > > > > >
