I added another comment [1] on the issue [2], but will share here as well for maximum visibility.
I have two PRs in Comet that I am looking for reviews on. The first PR adds an @IcebergApi annotation to all Java classes/methods/fields that are currently referenced from the Iceberg main branch, and also adds documentation to the contributors guide. #3237 <https://github.com/apache/datafusion-comet/pull/3237> The second PR builds on this and adds new dedicated unit tests for API stability. This PR is more consequential because it discusses deprecating/removing Comet's native_comet scan and how that relates to the Iceberg API. #3240 <https://github.com/apache/datafusion-comet/pull/3240> I am not an expert on the current integration, so it is possible that I may have misunderstood things. I would appreciate reviews from both the Iceberg and Comet communities! Thanks, Andy. [1] https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915 [2] https://github.com/apache/datafusion-comet/issues/2921 On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <[email protected]> wrote: > Thanks for starting the thread! I've added a comment on the Github issue. > Super exciting to see the iceberg-rust integration with Comet. > As mentioned, I'll help take a look at > https://github.com/apache/iceberg/pull/13786 and see if we can bring it > to the finish line. > > Best, > Kevin Liu > > On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <[email protected]> > wrote: > >> Thanks Andy for raising this. >> >> +1 for the iceberg-rust solution. This could benefit the whole oss >> community in the long term and avoid duplicated efforts. As with the >> technique challenges you mentioned, this seems more like a missing feature >> rather than limitation. >> >> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <[email protected]> >> wrote: >> >>> Thanks Andy for starting this discussion! >>> >>> I agree we should prioritize the iceberg-rust path as the long term >>> default for Comet/Iceberg scans, while it's still valuable to keep the >>> Iceberg Java integration as an option, especially for users who need >>> JVM-specific features. I'm hoping we can get consensus on a path to resolve >>> the shading issues (e.g. the approach proposed in >>> github.com/apache/iceberg/pull/13786) so the java option can move >>> forward. >>> >>> Thanks, >>> Huaxin >>> >>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <[email protected]> >>> wrote: >>> >>>> Thanks Andy for starting this thread! >>>> >>>> I've replied to the issue above but am posting here as well for better >>>> visibility: >>>> >>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the >>>> shading and circular dependency headaches of the Java approach. Easier >>>> maintenance typically lowers the entry barrier and encourages more >>>> community involvement, which is important for the project’s long-term >>>> health. >>>> >>>> It also makes sense to keep the Iceberg Java integration available as >>>> an experimental option for users who need its richer JVM-specific features >>>> while Rust becomes the long-term default. We don’t want to block >>>> contributions from anyone who truly needs the Java path. >>>> >>>> Best, >>>> Shawn >>>> >>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> There is a discussion in the DataFusion Comet community about the >>>>> future direction of our Iceberg support [1]. For example, should we >>>>> continue the efforts to integrate with the Iceberg Java project or focus >>>>> more on the iceberg-rust project. >>>>> >>>>> It would be great to also get input from the Iceberg community. >>>>> >>>>> Thanks, >>>>> >>>>> Andy. >>>>> >>>>> [1] https://github.com/apache/datafusion-comet/issues/2921 >>>>> >>>>
