Hi Vinish and Nick, here is the draft PR -> https://github.com/apache/incubator-xtable/pull/742
Looking forward to getting your feedback, especially on the Iceberg issue! :) On Thu, Sep 4, 2025 at 8:57 AM Mike Dias <[email protected]> wrote: > Hi Vinish, that's a good idea! > > I'll tidy up my branch and raise a draft pull request instead of an issue; > I think it will be easier to troubleshoot it by seeing the changes and > having a way to reproduce it. > > Thanks, > Mike Dias > > On Wed, Sep 3, 2025, 9:21 AM Vinish Reddy <[email protected]> wrote: > >> Hi Mike. >> >> I'm Vinish one of the PPMC members for XTable. Regarding the iceberg >> target issues you are seeing for the XTable Paimon source, can you create a >> GH issue? The XTable community can help in root causing the problem. >> https://github.com/apache/incubator-xtable/issues >> >> -Vinish >> >> On 2025/09/01 05:37:39 Mike Dias via user wrote: >> > Hi Nick, thanks for your reply! >> > >> > Yeah, I believe that creating the XTable source for Paimon is the best >> > approach: fewer hops, fewer dependencies, less maintenance overall. >> > >> > I have a prototype for the XTable Paimon source that works for Delta and >> > Hudi targets, but it does not work for the Iceberg target yet due to >> issues >> > with reading parquet fields with incorrect positions that I don't yet >> > understand why... >> > >> > Given that you have experience with Paimon and Iceberg, maybe I could >> get >> > some help troubleshooting that? That would be a huge help! >> > >> > >> > >> > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <[email protected]> >> wrote: >> > >> > > Hi Mike, >> > > >> > > I help maintain Paimon's Iceberg Compatibility. >> > > >> > > The fastest way to accomplish your goal is the first option. It >> should not >> > > require code changes to Paimon or XTable. That said the feature has >> some >> > > tradeoffs. >> > > >> > > XTable is greatly preferred to adding Delta specific code to Paimon. >> I do >> > > think it would benefit Paimon adoption to be available in XTable [1]. >> > > >> > > Paimon's Iceberg Compatibility works differently than XTable - it is a >> > > synchronous commit callback to translate metadata. This makes it >> really >> > > easy to enable and it simply supports latest snapshot queries. It >> does not >> > > yet support adding the full snapshot history (which I am working on >> > > https://github.com/apache/paimon/issues/6107) but still there will be >> > > some practical limitations around very long snapshot histories. >> > > >> > > Does the current feature set of Paimon-Iceberg compatibility support >> your >> > > use case? >> > > Would you be willing to contribute Paimon support in XTable? >> > > >> > > Nick >> > > >> > > [1] https://github.com/apache/incubator-xtable/issues/275 >> > > >> > > >> > > On 2025/08/08 05:44:03 Mike Dias via user wrote: >> > > > Hello, we are currently evaluating using Paimon as our primary >> format for >> > > > landing data via Flink, but we still want to maintain compatibility >> with >> > > > Delta Lake, as we use Delta Sharing >> > > > <https://github.com/delta-io/delta-sharing> features extensively. >> > > > >> > > > Right now, it seems that there are three options to achieve that: >> > > > >> > > > - Enable the Paimon -> Iceberg >> > > > <https://paimon.apache.org/docs/master/iceberg/overview/> >> > > Integration, >> > > > and then use XTable <https://xtable.apache.org/> to convert >> Iceberg >> > > > metadata to Delta Lake metadata. >> > > > - Implement a Paimon Source for XTable < >> https://xtable.apache.org/> >> > > and >> > > > use it to convert Delta Lake metadata directly from Paimon >> metadata >> > > > (read-optimize only). >> > > > - Implement a Commit Callback in Paimon that writes the Delta >> Lake >> > > > metadata directly, similar to what was done on the Iceberg >> integration >> > > > <https://paimon.apache.org/docs/master/iceberg/overview/>. >> > > > >> > > > We are playing with the first option right now, but we would like to >> > > avoid >> > > > that going forward because the Iceberg conversion is unnecessary >> for us. >> > > So >> > > > the question is, would the Paimon community accept contributions to >> > > enable >> > > > Delta Lake compatibility, just like Iceberg's, or would the >> community >> > > > prefer to rely on XTable for such cross-table format conversions? >> > > > >> > > > Keen to hear your thoughts! >> > > > >> > > > Thanks >> > > > Mike Dias >> > > > >> > >> > >> > >> > -- >> > Thanks, >> > Mike Dias >> > >> > -- Thanks, Mike Dias
