Hi Mike. I'm Vinish one of the PPMC members for XTable. Regarding the iceberg target issues you are seeing for the XTable Paimon source, can you create a GH issue? The XTable community can help in root causing the problem. https://github.com/apache/incubator-xtable/issues
-Vinish On 2025/09/01 05:37:39 Mike Dias via user wrote: > Hi Nick, thanks for your reply! > > Yeah, I believe that creating the XTable source for Paimon is the best > approach: fewer hops, fewer dependencies, less maintenance overall. > > I have a prototype for the XTable Paimon source that works for Delta and > Hudi targets, but it does not work for the Iceberg target yet due to issues > with reading parquet fields with incorrect positions that I don't yet > understand why... > > Given that you have experience with Paimon and Iceberg, maybe I could get > some help troubleshooting that? That would be a huge help! > > > > On Wed, Aug 27, 2025 at 6:36 AM Nick Del Nano <[email protected]> wrote: > > > Hi Mike, > > > > I help maintain Paimon's Iceberg Compatibility. > > > > The fastest way to accomplish your goal is the first option. It should not > > require code changes to Paimon or XTable. That said the feature has some > > tradeoffs. > > > > XTable is greatly preferred to adding Delta specific code to Paimon. I do > > think it would benefit Paimon adoption to be available in XTable [1]. > > > > Paimon's Iceberg Compatibility works differently than XTable - it is a > > synchronous commit callback to translate metadata. This makes it really > > easy to enable and it simply supports latest snapshot queries. It does not > > yet support adding the full snapshot history (which I am working on > > https://github.com/apache/paimon/issues/6107) but still there will be > > some practical limitations around very long snapshot histories. > > > > Does the current feature set of Paimon-Iceberg compatibility support your > > use case? > > Would you be willing to contribute Paimon support in XTable? > > > > Nick > > > > [1] https://github.com/apache/incubator-xtable/issues/275 > > > > > > On 2025/08/08 05:44:03 Mike Dias via user wrote: > > > Hello, we are currently evaluating using Paimon as our primary format for > > > landing data via Flink, but we still want to maintain compatibility with > > > Delta Lake, as we use Delta Sharing > > > <https://github.com/delta-io/delta-sharing> features extensively. > > > > > > Right now, it seems that there are three options to achieve that: > > > > > > - Enable the Paimon -> Iceberg > > > <https://paimon.apache.org/docs/master/iceberg/overview/> > > Integration, > > > and then use XTable <https://xtable.apache.org/> to convert Iceberg > > > metadata to Delta Lake metadata. > > > - Implement a Paimon Source for XTable <https://xtable.apache.org/> > > and > > > use it to convert Delta Lake metadata directly from Paimon metadata > > > (read-optimize only). > > > - Implement a Commit Callback in Paimon that writes the Delta Lake > > > metadata directly, similar to what was done on the Iceberg integration > > > <https://paimon.apache.org/docs/master/iceberg/overview/>. > > > > > > We are playing with the first option right now, but we would like to > > avoid > > > that going forward because the Iceberg conversion is unnecessary for us. > > So > > > the question is, would the Paimon community accept contributions to > > enable > > > Delta Lake compatibility, just like Iceberg's, or would the community > > > prefer to rely on XTable for such cross-table format conversions? > > > > > > Keen to hear your thoughts! > > > > > > Thanks > > > Mike Dias > > > > > > > -- > Thanks, > Mike Dias >
