> Another approach is to create a snapshot compatible way for Paimon to generate Iceberg, which is what we are working on.
Oh! Very interesting. Can you say more? And/or do you have links to Jira or anything? Thanks for your response! :) On Wed, May 29, 2024 at 7:41 AM Jingsong Li <jingsongl...@gmail.com> wrote: > Hi Andrew, > > It is difficult to move this mechanism to the Iceberg sink. The table > structure change in Iceberg's design requires generating a new > snapshot, which poses significant challenges to schema evolution. > > Another approach is to create a snapshot compatible way for Paimon to > generate Iceberg, which is what we are working on. > > Best, > Jingsong > > On Fri, May 24, 2024 at 8:11 PM Andrew Otto <o...@wikimedia.org> wrote: > > > > Hi! > > > > How coupled to Paimon catalogs and tables is the cdc part of Paimon? > RichCdcMultiplexRecord and related code seem incredibly useful even outside > of the context of the Paimon table format. > > > > I'm asking because the database sync action feature is amazing. At the > Wikimedia Foundation, we are on an all-in journey with Iceberg. I'm > wondering how hard it would be to extract the CDC logic from Paimon and > abstract the Sink bits. > > > > Could the table/database sync with schema evolution (without Flink job > restarts!) potentially work with the Iceberg sink? > > > > Thanks! > > -Andrew Otto > > Wikimedia Foundation > > > > > > > > >
