Awesome, I look forward to it! Thank you! On Mon, Jun 10, 2024 at 2:35 AM Jingsong Li <jingsongl...@gmail.com> wrote:
> We are developing prototype in our internal. > > It takes about 2 to 3 months. > > Andrew Otto <o...@wikimedia.org>于2024年5月29日 周三21:46写道： > >> > Another approach is to create a snapshot compatible way for Paimon to >> > generate >> Iceberg, which is what we are working on. >> >> Oh! Very interesting. Can you say more? And/or do you have links to >> Jira or anything? >> >> Thanks for your response! :) >> >> >> >> On Wed, May 29, 2024 at 7:41 AM Jingsong Li <jingsongl...@gmail.com> >> wrote: >> >>> Hi Andrew, >>> >>> It is difficult to move this mechanism to the Iceberg sink. The table >>> structure change in Iceberg's design requires generating a new >>> snapshot, which poses significant challenges to schema evolution. >>> >>> Another approach is to create a snapshot compatible way for Paimon to >>> generate Iceberg, which is what we are working on. >>> >>> Best, >>> Jingsong >>> >>> On Fri, May 24, 2024 at 8:11 PM Andrew Otto <o...@wikimedia.org> wrote: >>> > >>> > Hi! >>> > >>> > How coupled to Paimon catalogs and tables is the cdc part of Paimon? >>> RichCdcMultiplexRecord and related code seem incredibly useful even outside >>> of the context of the Paimon table format. >>> > >>> > I'm asking because the database sync action feature is amazing. At >>> the Wikimedia Foundation, we are on an all-in journey with Iceberg. I'm >>> wondering how hard it would be to extract the CDC logic from Paimon and >>> abstract the Sink bits. >>> > >>> > Could the table/database sync with schema evolution (without Flink job >>> restarts!) potentially work with the Iceberg sink? >>> > >>> > Thanks! >>> > -Andrew Otto >>> > Wikimedia Foundation >>> > >>> > >>> > >>> > >>> >>
