I have started a vote on the datafusion dev list to accept the code: https://lists.apache.org/thread/6k5lws31c61y1lx2s34pxo4lmc7334lr
On Wed, Sep 9, 2026 at 12:45 PM Andrew Lamb <[email protected]> wrote: > Thanks Kevin -- that would be great. > > If you start the iceberg vote thread, then someone else can start a vote > thread for DataFusion to accept it. > > Andrew > > On Wed, Sep 9, 2026 at 12:02 PM Kevin Liu <[email protected]> wrote: > >> Thanks Andrew. And thank you Gabriel for porting over the code, I've +1'ed >> the PR (https://github.com/apache/datafusion-iceberg/pull/1) >> >> Happy to help start a vote on the iceberg side. >> >> Best, >> Kevin Liu >> >> On Wed, Sep 9, 2026 at 4:36 AM Andrew Lamb <[email protected]> >> wrote: >> >> > The new repository is available: >> > https://github.com/apache/datafusion-iceberg >> > >> > Gabriel, can you take a look at making a proposed PR with the content to >> > move? >> > >> > On Wed, Sep 9, 2026 at 7:07 AM Andrew Lamb <[email protected]> >> wrote: >> > >> >> I have filed a INFR request to create the new repository: >> >> https://issues.apache.org/jira/browse/INFRA-28379 >> >> >> >> On Tue, Sep 8, 2026 at 8:02 PM Andrew Lamb <[email protected]> >> >> wrote: >> >> >> >> > I will plan to run both voting threads -- I think first we need a >> repo >> >> and >> >> > a PR to vote on. We will work on that starting tomorrow >> >> > >> >> > On Tue, Sep 8, 2026 at 7:52 PM Neelesh Salian < >> [email protected] >> >> > >> >> > wrote: >> >> > >> >> > > Sounds good to me. Feel free to start both threads. Glad to see >> this >> >> > going >> >> > > forward. >> >> > > >> >> > > On Tue, Sep 8, 2026 at 16:46 Andrew Lamb <[email protected]> >> >> wrote: >> >> > > >> >> > > > I think we should do a formal vote in a separate thread -- for >> both >> >> > > > communities >> >> > > > >> >> > > > On Tue, Sep 8, 2026 at 7:11 PM Neelesh Salian < >> >> > [email protected]> >> >> > > > wrote: >> >> > > > >> >> > > >> Yes, to the new repo apache/datafusion-iceberg >> >> > > >> >> >> > > >> Question to the folks here: Do we need a formal community vote >> >> across >> >> > > >> communities or this thread suffices? >> >> > > >> >> >> > > >> On Tue, Sep 8, 2026 at 03:37 Andrew Lamb <[email protected]> >> >> > wrote: >> >> > > >> >> >> > > >> > Thank for all the feedback so far -- unless anyone has >> concerns, >> >> I >> >> > > will >> >> > > >> > file a request a new repository for apache/datafusion-iceberg >> >> > tomorrow >> >> > > >> > >> >> > > >> > Andrew >> >> > > >> > >> >> > > >> > On Tue, Sep 8, 2026 at 5:49 AM Renjie Liu < >> >> [email protected]> >> >> > > >> wrote: >> >> > > >> > >> >> > > >> > > +1 for Andrew's proposal. >> >> > > >> > > >> >> > > >> > > On Sat, Sep 5, 2026 at 1:56 AM Kevin Liu < >> >> [email protected]> >> >> > > >> wrote: >> >> > > >> > > >> >> > > >> > > > + dev@iceberg >> >> > > >> > > > >> >> > > >> > > > (lesson learned, emailing 2 devlist might not be the best >> >> idea >> >> > > >> > > logistically >> >> > > >> > > > haha) >> >> > > >> > > > >> >> > > >> > > > On Fri, Sep 4, 2026 at 10:48 AM Kevin Liu < >> >> > [email protected]> >> >> > > >> > wrote: >> >> > > >> > > > >> >> > > >> > > > > Thanks for all the great discussions so far. I'm glad we >> >> > found a >> >> > > >> > > solution >> >> > > >> > > > > that benefits the broader ecosystem. >> >> > > >> > > > > I'm +1 to moving to an apache-governed location, >> >> > > >> > > > > apache/datafusion-iceberg seems like a great place. I >> like >> >> the >> >> > > >> > process >> >> > > >> > > > > Andrew proposed, happy to help with the logistics. >> >> > > >> > > > > >> >> > > >> > > > > Best, >> >> > > >> > > > > Kevin Liu >> >> > > >> > > > > >> >> > > >> > > > > PS I have also removed the iceberg python binding that >> >> exports >> >> > > >> > > > > DataFusion's TableProvider, so that's 1 less thing we >> have >> >> to >> >> > > >> worry >> >> > > >> > > > about. >> >> > > >> > > > > https://github.com/apache/iceberg-rust/issues/3036 >> >> > > >> > > > > >> >> > > >> > > > > On Fri, Sep 4, 2026 at 6:28 AM Gabriel Musat < >> >> > > >> [email protected]> >> >> > > >> > > > wrote: >> >> > > >> > > > > >> >> > > >> > > > >> Sounds like a good direction, +1 (non binding) >> >> > > >> > > > >> >> >> > > >> > > > >> I can help porting the commit history of the >> >> > DataFusion-Iceberg >> >> > > >> > > > >> integration to the new repo if PMCs agree. >> >> > > >> > > > >> >> >> > > >> > > > >> On 2026/09/04 12:27:30 Andrew Lamb wrote: >> >> > > >> > > > >> > I think this is a great idea as well -- thank you for >> >> > > bringing >> >> > > >> it >> >> > > >> > up >> >> > > >> > > > and >> >> > > >> > > > >> > for the great discussions so far. >> >> > > >> > > > >> > >> >> > > >> > > > >> > While at VLDB this past week, I spoke to at least >> three >> >> > > people >> >> > > >> > from >> >> > > >> > > > >> > companies adding Apache Iceberg support to their >> >> products. >> >> > > All >> >> > > >> of >> >> > > >> > > them >> >> > > >> > > > >> had >> >> > > >> > > > >> > to fork iceberg-rust for one reason or another. >> >> > > >> > > > >> > >> >> > > >> > > > >> > I think we have a huge need to improve our ability to >> >> work >> >> > > >> > together >> >> > > >> > > > and >> >> > > >> > > > >> > accelerate everyone's efforts. Giving the DataFusion >> >> > > >> integration >> >> > > >> > > > access >> >> > > >> > > > >> to >> >> > > >> > > > >> > more expert maintainers with bandwidth I think will >> help >> >> > > >> everyone. >> >> > > >> > > > >> > >> >> > > >> > > > >> > There seems to be consensus that we should move the >> >> > > integration >> >> > > >> > code >> >> > > >> > > > to >> >> > > >> > > > >> > DataFusion governance, and that it should remain in >> the >> >> > ASF, >> >> > > >> > though >> >> > > >> > > > some >> >> > > >> > > > >> > open technical questions remain. >> >> > > >> > > > >> > >> >> > > >> > > > >> > If that is the case, I propose the following specific >> >> > > process: >> >> > > >> > > > >> > 1. Create the new gitub repository in Apache for the >> >> code >> >> > > (e.g. >> >> > > >> > > > >> > apache/datafusion-iceberg) >> >> > > >> > > > >> > 2. Create a PR in the new repo with the proposed code >> >> > > >> > > > >> > 3. Hold a formal vote on the iceberg dev list to move >> >> the >> >> > > >> > > integration >> >> > > >> > > > >> code >> >> > > >> > > > >> > to DataFusion >> >> > > >> > > > >> > 4. Hold a formal vote on the DataFusion dev list to >> >> accept >> >> > > the >> >> > > >> new >> >> > > >> > > > code >> >> > > >> > > > >> > >> >> > > >> > > > >> > I am happy to help with the logistics (e.g. ASF INFRA >> >> > ticket >> >> > > to >> >> > > >> > > create >> >> > > >> > > > >> the >> >> > > >> > > > >> > new repo, votes, etc) but I am not expert enough to >> >> create >> >> > > the >> >> > > >> > > > proposed >> >> > > >> > > > >> PR. >> >> > > >> > > > >> > >> >> > > >> > > > >> > Please let me know your thoughts, >> >> > > >> > > > >> > Andrew >> >> > > >> > > > >> > (PMC Chair of DataFusion) >> >> > > >> > > > >> > >> >> > > >> > > > >> > On Tue, Sep 1, 2026 at 9:48 PM Renjie Liu < >> >> > > >> > [email protected]> >> >> > > >> > > > >> wrote: >> >> > > >> > > > >> > >> >> > > >> > > > >> > > To add more background about the relationship >> between >> >> > comet >> >> > > >> and >> >> > > >> > > > >> datafusion >> >> > > >> > > > >> > > for those who are not familiar with them. >> >> > > >> > > > >> > > >> >> > > >> > > > >> > > Apache datafusion is a popular extensible compute >> >> engine >> >> > > >> written >> >> > > >> > > in >> >> > > >> > > > >> rust. >> >> > > >> > > > >> > > Apache comet is an apache spark accelerator >> builton on >> >> > > apache >> >> > > >> > > > >> datafusion, >> >> > > >> > > > >> > > and also a subproject of apache datafusion. >> >> > > >> > > > >> > > >> >> > > >> > > > >> > > datafusion-iceberg is an apache datafusion >> extension >> >> > built >> >> > > on >> >> > > >> > > > >> iceberg-rust, >> >> > > >> > > > >> > > and comet's iceberg support is built on it. >> >> > > >> > > > >> > > >> >> > > >> > > > >> > > On Tue, Sep 1, 2026 at 9:39 PM Andy Grove < >> >> > > >> > [email protected]> >> >> > > >> > > > >> wrote: >> >> > > >> > > > >> > > >> >> > > >> > > > >> > > > +1 for moving to DataFusion PMC. The >> >> iceberg-datafusion >> >> > > >> > > > integration >> >> > > >> > > > >> is >> >> > > >> > > > >> > > > very important for Comet. >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > > On Mon, Aug 31, 2026 at 11:42 PM Xuanwo < >> >> > > [email protected] >> >> > > >> > >> >> > > >> > > > wrote: >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > TBH, I also support moving to the DataFusion >> PMC. >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > - DataFusion is the largest dependency in >> >> > > >> > iceberg-datafusion. >> >> > > >> > > > >> > > > > - The largest downstream user of >> >> iceberg-datafusion >> >> > is >> >> > > >> > Comet, >> >> > > >> > > > >> which >> >> > > >> > > > >> > > > shares many of the same PMC members as >> DataFusion. >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > It feels natural to be part of the DataFusion >> >> PMC. As >> >> > > >> long >> >> > > >> > as >> >> > > >> > > DF >> >> > > >> > > > >> PMC is >> >> > > >> > > > >> > > > willing to accept this project, it LGTM. >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > On Tue, Sep 1, 2026, at 12:03, Renjie Liu >> wrote: >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > I don't think the testing should be a blocker >> of >> >> > moving >> >> > > >> > > > >> > > > iceberg-datafusion out of iceberg-rust repo. From >> >> what >> >> > I >> >> > > >> > learn, >> >> > > >> > > > >> most of >> >> > > >> > > > >> > > the >> >> > > >> > > > >> > > > sqllogictests are in pr of modifying >> >> iceberg-datafusion >> >> > > >> > > > integration, >> >> > > >> > > > >> > > there >> >> > > >> > > > >> > > > are only few cases where we rely on >> sqllogictests to >> >> > > verify >> >> > > >> > > > >> features. >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > > How would that sound for the `iceberg-rust` >> >> > community >> >> > > >> to >> >> > > >> > > > grant a >> >> > > >> > > > >> > > > couple of Apache DataFusion PMCs committer >> access to >> >> > the >> >> > > >> > > > repository, >> >> > > >> > > > >> > > > limited to the `iceberg-datafusion` crate via a >> >> > > CODEOWNERS >> >> > > >> > file, >> >> > > >> > > > so >> >> > > >> > > > >> that >> >> > > >> > > > >> > > > they can help push reviews and PRs forward >> >> > independently? >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > I'm not sure if this is feasible, but moving >> the >> >> > > >> > > > >> iceberg-datafusion >> >> > > >> > > > >> > > > crate to apache datafusion project sounds a more >> >> > > reasonable >> >> > > >> > > > >> approach to >> >> > > >> > > > >> > > me. >> >> > > >> > > > >> > > > It's still governed by Apache, and most of the >> code >> >> is >> >> > > >> related >> >> > > >> > > to >> >> > > >> > > > >> > > > DataFusion, so I think the DataFusion community >> is >> >> in a >> >> > > >> better >> >> > > >> > > > >> position >> >> > > >> > > > >> > > to >> >> > > >> > > > >> > > > define the vision and design for it. >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > On Mon, Aug 31, 2026 at 10:45 PM Gabriel Musat >> < >> >> > > >> > > > >> [email protected]> >> >> > > >> > > > >> > > > wrote: >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > Hi everyone, >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > Based on these two facts: >> >> > > >> > > > >> > > > > - There's a current reviewer bandwidth problem >> >> that >> >> > > hurts >> >> > > >> > > > >> development >> >> > > >> > > > >> > > > velocity of the `iceberg-datafusion` crate >> currently >> >> > > hosted >> >> > > >> > > under >> >> > > >> > > > >> > > > `apache/iceberg-rust`. >> >> > > >> > > > >> > > > > - There is value in maintaining the >> >> > > `iceberg-datafusion` >> >> > > >> > crate >> >> > > >> > > > >> inside >> >> > > >> > > > >> > > > `apache/iceberg-rust` for testing, design and >> >> > governance. >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > How would that sound for the `iceberg-rust` >> >> community >> >> > > to >> >> > > >> > > grant a >> >> > > >> > > > >> couple >> >> > > >> > > > >> > > > of Apache DataFusion PMCs committer access to the >> >> > > >> repository, >> >> > > >> > > > >> limited to >> >> > > >> > > > >> > > > the `iceberg-datafusion` crate via a CODEOWNERS >> >> file, >> >> > so >> >> > > >> that >> >> > > >> > > they >> >> > > >> > > > >> can >> >> > > >> > > > >> > > help >> >> > > >> > > > >> > > > push reviews and PRs forward independently? >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > Based on review history, I'd propose Matt >> >> Butrovich >> >> > and >> >> > > >> Tim >> >> > > >> > > > >> Saucer as >> >> > > >> > > > >> > > > two good candidates, but this would be completely >> >> up to >> >> > > the >> >> > > >> > > > >> > > `iceberg-rust` >> >> > > >> > > > >> > > > community. >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > On 2026/08/26 17:56:19 Shawn Chang wrote: >> >> > > >> > > > >> > > > > > Hi all, >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > Summarizing the discussion so far, including >> a >> >> few >> >> > > >> points >> >> > > >> > > > >> raised in >> >> > > >> > > > >> > > > today’s >> >> > > >> > > > >> > > > > > community sync. >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > There seems to be general agreement that the >> >> > current >> >> > > >> > > > DataFusion >> >> > > >> > > > >> > > > integration >> >> > > >> > > > >> > > > > > has a velocity/reviewer bandwidth problem, >> and >> >> > moving >> >> > > >> it >> >> > > >> > to >> >> > > >> > > a >> >> > > >> > > > >> > > separate >> >> > > >> > > > >> > > > > > repository could help DataFusion contributors >> >> > iterate >> >> > > >> more >> >> > > >> > > > >> > > > independently. >> >> > > >> > > > >> > > > > > At the same time, several open questions >> remain: >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > - >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > Whether repo separation is the right >> >> solution, >> >> > > >> versus >> >> > > >> > > > >> expanding >> >> > > >> > > > >> > > > > > DataFusion reviewer/committer >> participation >> >> in >> >> > > >> > > > iceberg-rust. >> >> > > >> > > > >> > > > > > - >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > Where the boundary should be between >> engine >> >> > > specific >> >> > > >> > > > >> integration >> >> > > >> > > > >> > > and >> >> > > >> > > > >> > > > > > Iceberg core functionality, and how to >> avoid >> >> > > >> duplicated >> >> > > >> > > or >> >> > > >> > > > >> forked >> >> > > >> > > > >> > > > Iceberg >> >> > > >> > > > >> > > > > > implementations. (how to avoid the case >> where >> >> > > >> > > > >> Iceberg-datafusion >> >> > > >> > > > >> > > > moving >> >> > > >> > > > >> > > > > > much faster than the core and eventually >> >> need a >> >> > > >> forked >> >> > > >> > > core >> >> > > >> > > > >> API >> >> > > >> > > > >> > > > > > implementation) >> >> > > >> > > > >> > > > > > - >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > How compatibility and end-to-end >> correctness >> >> > > testing >> >> > > >> > > should >> >> > > >> > > > >> work >> >> > > >> > > > >> > > > across >> >> > > >> > > > >> > > > > > repositories, since iceberg-rust still >> >> relies on >> >> > > >> > > DataFusion >> >> > > >> > > > >> for >> >> > > >> > > > >> > > > integration >> >> > > >> > > > >> > > > > > testing. >> >> > > >> > > > >> > > > > > - >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > Where the integration should live and how >> to >> >> > keep >> >> > > it >> >> > > >> > > under >> >> > > >> > > > >> Apache >> >> > > >> > > > >> > > > > > governance while making it easy for both >> >> Iceberg >> >> > > and >> >> > > >> > > > >> DataFusion >> >> > > >> > > > >> > > > > > contributors to maintain. >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > So I think the main question is not only >> >> whether to >> >> > > >> move >> >> > > >> > the >> >> > > >> > > > >> code, >> >> > > >> > > > >> > > but >> >> > > >> > > > >> > > > how >> >> > > >> > > > >> > > > > > to improve development velocity without >> losing >> >> the >> >> > > >> close >> >> > > >> > > > design, >> >> > > >> > > > >> > > > testing, >> >> > > >> > > > >> > > > > > and governance relationship between the >> engine >> >> > > >> integration >> >> > > >> > > and >> >> > > >> > > > >> > > > iceberg-rust >> >> > > >> > > > >> > > > > > core. >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > Best, >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > Shawn >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > On Mon, Aug 24, 2026 at 4:14 AM Manu Zhang < >> >> > > >> > > > >> [email protected]> >> >> > > >> > > > >> > > > wrote: >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > > > +1 moving the DataFusion integration and >> tests >> >> > > into a >> >> > > >> > > > separate >> >> > > >> > > > >> > > > > > > apache-governed repository. Can we bring >> this >> >> > > >> discussion >> >> > > >> > > to >> >> > > >> > > > >> the >> >> > > >> > > > >> > > > Community >> >> > > >> > > > >> > > > > > > Sync[1] this week? >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > > 1. >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > >> >> > > >> > > > >> >> >> > > >> > > > >> >> > > >> > > >> >> > > >> > >> >> > > >> >> >> > > >> >> > >> >> >> https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit?tab=t.0 >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > > On Mon, Aug 24, 2026 at 3:37 PM Renjie Liu >> < >> >> > > >> > > > >> > > [email protected]> >> >> > > >> > > > >> > > > > > > wrote: >> >> > > >> > > > >> > > > > > > >> >> > > >> > > > >> > > > > > >> Hi, Matt: >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> Thanks for raising this. >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> 1) Apache governance: >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> I would +1 for putting this in an apache >> >> repo, >> >> > for >> >> > > >> > > example >> >> > > >> > > > a >> >> > > >> > > > >> sub >> >> > > >> > > > >> > > > repo of >> >> > > >> > > > >> > > > > > >> datafusion project. Shawn has stated most >> of >> >> the >> >> > > >> > reasons, >> >> > > >> > > > so >> >> > > >> > > > >> I >> >> > > >> > > > >> > > > don't want >> >> > > >> > > > >> > > > > > >> to repeat it again. >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> 2) Where do tests live/how tests should be >> >> > > >> maintained? >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> Initially I was thinking about putting >> >> > > sqllogictest >> >> > > >> in >> >> > > >> > > > >> > > > iceberg-rust, but >> >> > > >> > > > >> > > > > > >> after second thought I'm leaning towards >> to >> >> put >> >> > it >> >> > > >> in >> >> > > >> > the >> >> > > >> > > > >> new repo >> >> > > >> > > > >> > > > for two >> >> > > >> > > > >> > > > > > >> reasons: >> >> > > >> > > > >> > > > > > >> 1. It would be easier for developer of the >> >> > > >> > > > datafusion-iceberg >> >> > > >> > > > >> > > > integration >> >> > > >> > > > >> > > > > > >> to add tests >> >> > > >> > > > >> > > > > > >> 2. It would make the dependency graph and >> >> > version >> >> > > >> > release >> >> > > >> > > > >> easier. >> >> > > >> > > > >> > > > Though >> >> > > >> > > > >> > > > > > >> the dependency is on crate level rather >> than >> >> > repo >> >> > > >> > level, >> >> > > >> > > > the >> >> > > >> > > > >> > > > bi-direction >> >> > > >> > > > >> > > > > > >> dependency may make version management >> weird >> >> and >> >> > > >> > > difficult. >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> The downside of this approach is that >> it's a >> >> > > little >> >> > > >> > > > >> unfriendly for >> >> > > >> > > > >> > > > > > >> iceberg-rust developers, but I think it's >> >> less >> >> > > >> frequent >> >> > > >> > > for >> >> > > >> > > > >> > > > iceberg-rust >> >> > > >> > > > >> > > > > > >> developers to add sqllogictests compared >> with >> >> > > >> > > > >> datafusion-iceberg >> >> > > >> > > > >> > > > developers. >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> 3) Dropping DataFusion dependencies from >> >> > > >> pyiceberg-core >> >> > > >> > > > >> binding >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> I'm not quite familiar with this part, >> but it >> >> > > sounds >> >> > > >> > > > >> reasonable to >> >> > > >> > > > >> > > > me. >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> On Sat, Aug 22, 2026 at 6:50 AM Shawn >> Chang < >> >> > > >> > > > >> > > [email protected] >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > > >> wrote: >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >>> Hi Matt, >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> Thanks for raising this! I generally >> agree >> >> that >> >> > > we >> >> > > >> can >> >> > > >> > > > move >> >> > > >> > > > >> the >> >> > > >> > > > >> > > > > > >>> datafusion >> >> > > >> > > > >> > > > > > >>> integration to a separate repo if that >> helps >> >> > more >> >> > > >> > > > DataFusion >> >> > > >> > > > >> > > > experts to >> >> > > >> > > > >> > > > > > >>> work on the integrations >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> On the three considerations: >> >> > > >> > > > >> > > > > > >>> 1) Apache governance: >> >> > > >> > > > >> > > > > > >>> I think this is my biggest concern so >> far. >> >> This >> >> > > >> could >> >> > > >> > > not >> >> > > >> > > > >> only >> >> > > >> > > > >> > > > affect >> >> > > >> > > > >> > > > > > >>> contributors, but also whether the >> >> downstream >> >> > > users >> >> > > >> > > could >> >> > > >> > > > >> > > continue >> >> > > >> > > > >> > > > to use >> >> > > >> > > > >> > > > > > >>> the integration. >> >> > > >> > > > >> > > > > > >>> Even if the code remains Apache-licensed, >> >> > > >> governance, >> >> > > >> > > > >> release of >> >> > > >> > > > >> > > > > > >>> artifacts, >> >> > > >> > > > >> > > > > > >>> and contribution policies can matter for >> >> > > adoption. >> >> > > >> > > > >> > > > > > >>> We should explore more about the option >> to >> >> keep >> >> > > it >> >> > > >> > under >> >> > > >> > > > an >> >> > > >> > > > >> > > > > > >>> Apache-governed >> >> > > >> > > > >> > > > > > >>> repository. >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> 2) Where do tests live/how tests should >> be >> >> > > >> maintained? >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> I think this is closely related to the >> >> > governance >> >> > > >> > > > question. >> >> > > >> > > > >> To >> >> > > >> > > > >> > > me, >> >> > > >> > > > >> > > > the >> >> > > >> > > > >> > > > > > >>> broader question is: *how do we make it >> easy >> >> > for >> >> > > >> > people >> >> > > >> > > > >> from both >> >> > > >> > > > >> > > > the >> >> > > >> > > > >> > > > > > >>> Iceberg Rust and DataFusion communities >> to >> >> > > maintain >> >> > > >> > the >> >> > > >> > > > >> > > > integration?* >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> If we move it to a non-Apache >> repository, I >> >> > worry >> >> > > >> that >> >> > > >> > > it >> >> > > >> > > > >> could >> >> > > >> > > > >> > > > > > >>> eventually >> >> > > >> > > > >> > > > > > >>> look somewhat like the current Iceberg >> Java >> >> <> >> >> > > >> Trino >> >> > > >> > > > >> integration: >> >> > > >> > > > >> > > > the >> >> > > >> > > > >> > > > > > >>> integration primarily lives on the Trino >> >> side >> >> > and >> >> > > >> is >> >> > > >> > > > >> therefore >> >> > > >> > > > >> > > > mostly >> >> > > >> > > > >> > > > > > >>> maintained by people who are already >> deeply >> >> > > >> involved >> >> > > >> > in >> >> > > >> > > > >> Trino. >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> There is an important difference here, >> >> though. >> >> > > For >> >> > > >> > > Iceberg >> >> > > >> > > > >> Java, >> >> > > >> > > > >> > > > Spark is >> >> > > >> > > > >> > > > > > >>> arguably the primary engine integration >> and >> >> > has a >> >> > > >> > large >> >> > > >> > > > >> Iceberg >> >> > > >> > > > >> > > > > > >>> contributor >> >> > > >> > > > >> > > > > > >>> base around it, so having the Trino >> >> integration >> >> > > >> > > maintained >> >> > > >> > > > >> more >> >> > > >> > > > >> > > > > > >>> independently is relatively natural. In >> >> Iceberg >> >> > > >> Rust >> >> > > >> > > > today, >> >> > > >> > > > >> > > > DataFusion >> >> > > >> > > > >> > > > > > >>> has >> >> > > >> > > > >> > > > > > >>> a much more central role. It is by far >> the >> >> most >> >> > > >> mature >> >> > > >> > > > >> engine >> >> > > >> > > > >> > > > integration >> >> > > >> > > > >> > > > > > >>> in the project, is used by our >> SQLLogicTest >> >> > > >> > > > infrastructure, >> >> > > >> > > > >> and >> >> > > >> > > > >> > > > many >> >> > > >> > > > >> > > > > > >>> users >> >> > > >> > > > >> > > > > > >>> building on Iceberg Rust are also >> building >> >> on >> >> > > >> > > DataFusion. >> >> > > >> > > > >> The >> >> > > >> > > > >> > > > overlap >> >> > > >> > > > >> > > > > > >>> between the two communities is therefore >> >> much >> >> > > >> larger. >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> Because of that, I would prefer that >> >> extracting >> >> > > the >> >> > > >> > > > >> integration >> >> > > >> > > > >> > > > does not >> >> > > >> > > > >> > > > > > >>> turn it into something that is >> effectively >> >> > owned >> >> > > >> only >> >> > > >> > by >> >> > > >> > > > the >> >> > > >> > > > >> > > > DataFusion >> >> > > >> > > > >> > > > > > >>> side. Ideally, both Iceberg Rust >> >> contributors >> >> > and >> >> > > >> > > > DataFusion >> >> > > >> > > > >> > > > contributors >> >> > > >> > > > >> > > > > > >>> should be able to review changes, >> maintain >> >> > > >> > > compatibility, >> >> > > >> > > > >> and >> >> > > >> > > > >> > > > evolve the >> >> > > >> > > > >> > > > > > >>> integration together. >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> 3) Dropping DataFusion dependencies from >> >> > > >> > pyiceberg-core >> >> > > >> > > > >> binding: >> >> > > >> > > > >> > > I >> >> > > >> > > > >> > > > think >> >> > > >> > > > >> > > > > > >>> it >> >> > > >> > > > >> > > > > > >>> makes sense to make things simpler and >> have >> >> > left >> >> > > a >> >> > > >> > > comment >> >> > > >> > > > >> on the >> >> > > >> > > > >> > > > github >> >> > > >> > > > >> > > > > > >>> issue with more detailed thoughts: >> >> > > >> > > > >> > > > > > >>> >> >> > > https://github.com/apache/iceberg-rust/issues/3036 >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> Best, >> >> > > >> > > > >> > > > > > >>> Shawn >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> On Fri, Aug 21, 2026 at 11:23 AM Matt >> >> > Butrovich < >> >> > > >> > > > >> > > > [email protected]> >> >> > > >> > > > >> > > > > > >>> wrote: >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >>> > Hello all, >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > I've never tried emailing two different >> >> > project >> >> > > >> > lists >> >> > > >> > > at >> >> > > >> > > > >> once, >> >> > > >> > > > >> > > > but it >> >> > > >> > > > >> > > > > > >>> was >> >> > > >> > > > >> > > > > > >>> > suggested that I do so to try to track >> the >> >> > > >> > > conversation >> >> > > >> > > > >> across >> >> > > >> > > > >> > > > both >> >> > > >> > > > >> > > > > > >>> > communities. We'll see how this >> threads on >> >> > the >> >> > > >> > mailing >> >> > > >> > > > >> lists. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > There has been a GitHub Discussion >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > >> >> > > >> > > > >> >> >> > > >> > > > >> >> > > >> > > >> >> > > >> > >> >> > > >> >> >> > > >> >> > >> >> >> https://github.com/apache/iceberg-rust/discussions/2992#discussioncomment-18082533 >> >> > > >> > > > >> > > > > > >>> , >> >> > > >> > > > >> > > > > > >>> > a GitHub Issue >> >> > > >> > > > >> > > > >> https://github.com/apache/iceberg-rust/issues/3029, >> >> > and >> >> > > >> > > > >> > > > > > >>> > it's been a long topic of conversation >> in >> >> the >> >> > > >> past >> >> > > >> > two >> >> > > >> > > > >> weeks in >> >> > > >> > > > >> > > > both >> >> > > >> > > > >> > > > > > >>> the >> >> > > >> > > > >> > > > > > >>> > DataFusion and Iceberg Rust Community >> >> Calls >> >> > to >> >> > > >> > discuss >> >> > > >> > > > >> moving >> >> > > >> > > > >> > > the >> >> > > >> > > > >> > > > > > >>> > DataFusion integration from Iceberg >> Rust >> >> to a >> >> > > >> > separate >> >> > > >> > > > >> > > > repository. I >> >> > > >> > > > >> > > > > > >>> will >> >> > > >> > > > >> > > > > > >>> > try to summarize some of the major >> points >> >> as >> >> > I >> >> > > >> > > > understand >> >> > > >> > > > >> them, >> >> > > >> > > > >> > > > but the >> >> > > >> > > > >> > > > > > >>> > conversations are the ground truth and >> >> please >> >> > > >> feel >> >> > > >> > > free >> >> > > >> > > > to >> >> > > >> > > > >> > > > correct me >> >> > > >> > > > >> > > > > > >>> here. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > The DataFusion TableProvider >> integration >> >> in >> >> > > >> Iceberg >> >> > > >> > > Rust >> >> > > >> > > > >> makes >> >> > > >> > > > >> > > > > > >>> DataFusion >> >> > > >> > > > >> > > > > > >>> > a dependency for Iceberg Rust. The >> >> > integration >> >> > > >> > exists >> >> > > >> > > > for >> >> > > >> > > > >> > > > multiple >> >> > > >> > > > >> > > > > > >>> reasons: >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 1) an engine to execute Iceberg Rust's >> >> corpus >> >> > > of >> >> > > >> > > > >> sqllogictest >> >> > > >> > > > >> > > > files >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 2) a TableProvider integration for >> >> DataFusion >> >> > > >> users >> >> > > >> > to >> >> > > >> > > > >> interact >> >> > > >> > > > >> > > > with >> >> > > >> > > > >> > > > > > >>> > Iceberg tables >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > Some of the motivations to break out >> the >> >> > > >> > integration: >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 1) There have been a number of issues >> and >> >> > pull >> >> > > >> > > requests >> >> > > >> > > > >> against >> >> > > >> > > > >> > > > the >> >> > > >> > > > >> > > > > > >>> > DataFusion TableProvider in Iceberg >> Rust >> >> as >> >> > > users >> >> > > >> > want >> >> > > >> > > > to >> >> > > >> > > > >> add >> >> > > >> > > > >> > > > more >> >> > > >> > > > >> > > > > > >>> > features, and they often go stale. I >> don't >> >> > > >> believe >> >> > > >> > > there >> >> > > >> > > > >> are >> >> > > >> > > > >> > > many >> >> > > >> > > > >> > > > > > >>> > committers/PMC members familiar with or >> >> using >> >> > > the >> >> > > >> > > > >> DataFusion >> >> > > >> > > > >> > > > > > >>> integration. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 2) Iceberg Rust would like to stay as >> >> > > >> > engine-agnostic >> >> > > >> > > as >> >> > > >> > > > >> > > > possible. A >> >> > > >> > > > >> > > > > > >>> > recent DataFusion Ballista integration >> was >> >> > > >> declined >> >> > > >> > > for >> >> > > >> > > > >> this >> >> > > >> > > > >> > > > reason >> >> > > >> > > > >> > > > > > >>> > >> >> > > https://github.com/apache/iceberg-rust/pull/2613 >> >> > > >> . >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 3) Other projects that rely on both >> >> Iceberg >> >> > > Rust >> >> > > >> and >> >> > > >> > > > >> DataFusion >> >> > > >> > > > >> > > > (e.g., >> >> > > >> > > > >> > > > > > >>> > Comet) are blocked by Iceberg Rust >> >> upgrading >> >> > > its >> >> > > >> > > > >> DataFusion and >> >> > > >> > > > >> > > > Arrow >> >> > > >> > > > >> > > > > > >>> > dependencies before they can upgrade. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > Some considerations for both >> communities: >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 1) Where would this DataFusion >> >> TableProvider >> >> > > >> live? >> >> > > >> > > Most >> >> > > >> > > > >> > > > specifically, >> >> > > >> > > > >> > > > > > >>> > would it be an Apache-governed >> project? As >> >> > > folks >> >> > > >> > like >> >> > > >> > > > >> > > @andygrove >> >> > > >> > > > >> > > > point >> >> > > >> > > > >> > > > > > >>> out, >> >> > > >> > > > >> > > > > > >>> > this can affect whether some community >> >> > members >> >> > > >> could >> >> > > >> > > > >> contribute >> >> > > >> > > > >> > > > to it. >> >> > > >> > > > >> > > > > > >>> > There is a datafusion-contrib org for >> >> > > >> > > DataFusion-related >> >> > > >> > > > >> > > > projects to >> >> > > >> > > > >> > > > > > >>> have >> >> > > >> > > > >> > > > > > >>> > visibility but no Apache governance, >> but >> >> > there >> >> > > >> may >> >> > > >> > be >> >> > > >> > > > >> options >> >> > > >> > > > >> > > to >> >> > > >> > > > >> > > > put it >> >> > > >> > > > >> > > > > > >>> > under an Apache repository. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 2) How would Iceberg Rust continue to >> run >> >> > > >> > > sqllogictests >> >> > > >> > > > >> for >> >> > > >> > > > >> > > > regression >> >> > > >> > > > >> > > > > > >>> > testing? Does this live in a different >> >> > > repository >> >> > > >> > that >> >> > > >> > > > >> depends >> >> > > >> > > > >> > > > on this >> >> > > >> > > > >> > > > > > >>> new >> >> > > >> > > > >> > > > > > >>> > Iceberg Rust TableProvider crate? Would >> >> we be >> >> > > >> able >> >> > > >> > to >> >> > > >> > > > >> test Pull >> >> > > >> > > > >> > > > > > >>> Requests on >> >> > > >> > > > >> > > > > > >>> > Iceberg Rust with sqllogictests? >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > 3) @kevinqliu is familiar with the >> Python >> >> > > >> bindings >> >> > > >> > in >> >> > > >> > > > >> Iceberg >> >> > > >> > > > >> > > > Rust, >> >> > > >> > > > >> > > > > > >>> and I >> >> > > >> > > > >> > > > > > >>> > believe that DataFusion dependency >> might >> >> be >> >> > > >> removed >> >> > > >> > as >> >> > > >> > > > >> well, >> >> > > >> > > > >> > > but >> >> > > >> > > > >> > > > that >> >> > > >> > > > >> > > > > > >>> is >> >> > > >> > > > >> > > > > > >>> > not the core focus of this >> conversation. >> >> He >> >> > has >> >> > > >> also >> >> > > >> > > > >> proposed >> >> > > >> > > > >> > > > removing >> >> > > >> > > > >> > > > > > >>> that >> >> > > >> > > > >> > > > > > >>> > and using the DataFusion Python >> bindings >> >> > > >> directly. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > I'm sure I'm forgetting things, but >> this >> >> > email >> >> > > is >> >> > > >> > long >> >> > > >> > > > >> enough. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > Thanks everyone for the discussion thus >> >> far, >> >> > > and >> >> > > >> > > looking >> >> > > >> > > > >> > > forward >> >> > > >> > > > >> > > > to >> >> > > >> > > > >> > > > > > >>> more >> >> > > >> > > > >> > > > > > >>> > input on this. >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > -Matt >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> > >> >> > > >> > > > >> > > > > > >>> >> >> > > >> > > > >> > > > > > >> >> >> > > >> > > > >> > > > > > >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > Xuanwo >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > > https://xuanwo.io/ >> >> > > >> > > > >> > > > > >> >> > > >> > > > >> > > > >> >> > > >> > > > >> > > >> >> > > >> > > > >> > >> >> > > >> > > > >> >> >> > > >> > > > >> >> >> > > >> > >> >> > --------------------------------------------------------------------- >> >> > > >> > > > >> To unsubscribe, e-mail: >> >> > [email protected] >> >> > > >> > > > >> For additional commands, e-mail: >> >> > > [email protected] >> >> > > >> > > > >> >> >> > > >> > > > >> >> >> > > >> > > > >> >> > > >> > > >> >> > > >> > >> >> > > >> >> >> > > > >> >> > > >> >> > >> >> >> > >> >
