Hi Anurag, I have no objection to my PR being reverted to allow for more discussion. I think that would make sense.
My only concern is that I am not sure how we would "fix the broken code". My understanding is that there was a PR that aimed to do that [1] but it was just closed after 7 months and 98 comments. Parth can likely add more context here though. I'm open to suggestions on the best way to handle this. Thanks, Andy. [1] https://github.com/apache/iceberg/pull/13786 On Fri, Mar 20, 2026 at 11:28 AM Anurag Mantripragada <[email protected]> wrote: > > Thanks for the PR, Andy. > > I see that it has been merged, but I feel we haven't yet had the opportunity > to discuss this in detail within the Iceberg community. Kevin and I were > discussing this offline and feel that removing the broken code in its current > state makes it difficult to restore in the future. > > I propose the following: > - Revert the removal PR. > - Fix the broken code and include it in a patch release so we freeze a > working version. > - Discuss the removal during the next Iceberg community sync on April 1. > - If we reach a consensus, re-apply the removal PR. > > What do you think? > > ~ Anurag > > On Wed, Mar 18, 2026 at 9:54 AM Andy Grove <[email protected]> wrote: >> >> I have a draft PR up to remove the current Comet integration. I am waiting >> for CI to finish before marking it as ready for review. >> >> https://github.com/apache/iceberg/pull/15674 >> >> On Wed, Mar 18, 2026 at 10:43 AM Kevin Liu <[email protected]> wrote: >>> >>> Thanks for the update, Andy! >>> It looks like most of the Comet code lives in the Spark integration [1]. >>> We're currently working on the next 1.11 release. I know the removal isn't >>> time-sensitive, but just wanted to mention it. >>> >>> Best, >>> Kevin Liu >>> >>> [1] >>> https://grep.app/search?f.repo=apache%2Ficeberg&f.repo.pattern=apache%2Ficeberg&q=comet >>> >>> On Wed, Mar 18, 2026 at 8:24 AM Andy Grove <[email protected]> wrote: >>>> >>>> Hi, >>>> >>>> The Comet community has made good progress with the iceberg-rust >>>> integration and we now believe that this is the best path forward for >>>> Comet, so are no longer going to be working on the integration with >>>> Iceberg-Java. I posted a comment on the discussion issue as well [1]. >>>> >>>> Thanks for the community support with all of this. We will follow up with >>>> a PR soon to remove the current (broken) Comet integration from >>>> Iceberg-Java. >>>> >>>> Thanks, >>>> >>>> Andy. >>>> >>>> [1] >>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-4048408537 >>>> >>>> On Fri, Feb 20, 2026 at 7:03 AM Péter Váry <[email protected]> >>>> wrote: >>>>> >>>>> Hi Team, >>>>> The File Format API is merged, so that should not block the progress on >>>>> the Comet integration progress anymore. >>>>> If you need help, let me know. >>>>> Thanks, >>>>> Peter >>>>> >>>>> Andy Grove <[email protected]> ezt írta (időpont: 2026. jan. 22., Cs, >>>>> 21:51): >>>>>> >>>>>> I added another comment [1] on the issue [2], but will share here as >>>>>> well for maximum visibility. >>>>>> >>>>>> I have two PRs in Comet that I am looking for reviews on. >>>>>> >>>>>> The first PR adds an @IcebergApi annotation to all Java >>>>>> classes/methods/fields that are currently referenced from the Iceberg >>>>>> main branch, and also adds documentation to the contributors guide. >>>>>> #3237 >>>>>> >>>>>> The second PR builds on this and adds new dedicated unit tests for API >>>>>> stability. This PR is more consequential because it discusses >>>>>> deprecating/removing Comet's native_comet scan and how that relates to >>>>>> the Iceberg API. >>>>>> #3240 >>>>>> >>>>>> I am not an expert on the current integration, so it is possible that I >>>>>> may have misunderstood things. I would appreciate reviews from both the >>>>>> Iceberg and Comet communities! >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Andy. >>>>>> >>>>>> [1] >>>>>> https://github.com/apache/datafusion-comet/issues/2921#issuecomment-3785878915 >>>>>> [2] https://github.com/apache/datafusion-comet/issues/2921 >>>>>> >>>>>> On Tue, Dec 23, 2025 at 1:20 PM Kevin Liu <[email protected]> wrote: >>>>>>> >>>>>>> Thanks for starting the thread! I've added a comment on the Github >>>>>>> issue. Super exciting to see the iceberg-rust integration with Comet. >>>>>>> As mentioned, I'll help take a look at >>>>>>> https://github.com/apache/iceberg/pull/13786 and see if we can bring it >>>>>>> to the finish line. >>>>>>> >>>>>>> Best, >>>>>>> Kevin Liu >>>>>>> >>>>>>> On Thu, Dec 18, 2025 at 6:30 PM Renjie Liu <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> Thanks Andy for raising this. >>>>>>>> >>>>>>>> +1 for the iceberg-rust solution. This could benefit the whole oss >>>>>>>> community in the long term and avoid duplicated efforts. As with the >>>>>>>> technique challenges you mentioned, this seems more like a missing >>>>>>>> feature rather than limitation. >>>>>>>> >>>>>>>> On Thu, Dec 18, 2025 at 5:58 AM huaxin gao <[email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Thanks Andy for starting this discussion! >>>>>>>>> >>>>>>>>> I agree we should prioritize the iceberg-rust path as the long term >>>>>>>>> default for Comet/Iceberg scans, while it's still valuable to keep >>>>>>>>> the Iceberg Java integration as an option, especially for users who >>>>>>>>> need JVM-specific features. I'm hoping we can get consensus on a path >>>>>>>>> to resolve the shading issues (e.g. the approach proposed in >>>>>>>>> github.com/apache/iceberg/pull/13786) so the java option can move >>>>>>>>> forward. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Huaxin >>>>>>>>> >>>>>>>>> On Wed, Dec 17, 2025 at 12:23 PM Shawn Chang <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Thanks Andy for starting this thread! >>>>>>>>>> >>>>>>>>>> I've replied to the issue above but am posting here as well for >>>>>>>>>> better visibility: >>>>>>>>>> >>>>>>>>>> I’m leaning toward prioritizing the iceberg-rust path to avoid the >>>>>>>>>> shading and circular dependency headaches of the Java approach. >>>>>>>>>> Easier maintenance typically lowers the entry barrier and encourages >>>>>>>>>> more community involvement, which is important for the project’s >>>>>>>>>> long-term health. >>>>>>>>>> >>>>>>>>>> It also makes sense to keep the Iceberg Java integration available >>>>>>>>>> as an experimental option for users who need its richer JVM-specific >>>>>>>>>> features while Rust becomes the long-term default. We don’t want to >>>>>>>>>> block contributions from anyone who truly needs the Java path. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Shawn >>>>>>>>>> >>>>>>>>>> On Wed, Dec 17, 2025 at 11:00 AM Andy Grove <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> There is a discussion in the DataFusion Comet community about the >>>>>>>>>>> future direction of our Iceberg support [1]. For example, should we >>>>>>>>>>> continue the efforts to integrate with the Iceberg Java project or >>>>>>>>>>> focus more on the iceberg-rust project. >>>>>>>>>>> >>>>>>>>>>> It would be great to also get input from the Iceberg community. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Andy. >>>>>>>>>>> >>>>>>>>>>> [1] https://github.com/apache/datafusion-comet/issues/2921
