We are developing prototype in our internal. It takes about 2 to 3 months.
Andrew Otto <o...@wikimedia.org>于2024年5月29日 周三21:46写道： > > Another approach is to create a snapshot compatible way for Paimon to > > generate > Iceberg, which is what we are working on. > > Oh! Very interesting. Can you say more? And/or do you have links to Jira > or anything? > > Thanks for your response! :) > > > > On Wed, May 29, 2024 at 7:41 AM Jingsong Li <jingsongl...@gmail.com> > wrote: > >> Hi Andrew, >> >> It is difficult to move this mechanism to the Iceberg sink. The table >> structure change in Iceberg's design requires generating a new >> snapshot, which poses significant challenges to schema evolution. >> >> Another approach is to create a snapshot compatible way for Paimon to >> generate Iceberg, which is what we are working on. >> >> Best, >> Jingsong >> >> On Fri, May 24, 2024 at 8:11 PM Andrew Otto <o...@wikimedia.org> wrote: >> > >> > Hi! >> > >> > How coupled to Paimon catalogs and tables is the cdc part of Paimon? >> RichCdcMultiplexRecord and related code seem incredibly useful even outside >> of the context of the Paimon table format. >> > >> > I'm asking because the database sync action feature is amazing. At the >> Wikimedia Foundation, we are on an all-in journey with Iceberg. I'm >> wondering how hard it would be to extract the CDC logic from Paimon and >> abstract the Sink bits. >> > >> > Could the table/database sync with schema evolution (without Flink job >> restarts!) potentially work with the Iceberg sink? >> > >> > Thanks! >> > -Andrew Otto >> > Wikimedia Foundation >> > >> > >> > >> > >> >
