zhongyujiang commented on issue #3627: URL: https://github.com/apache/paimon/issues/3627#issuecomment-2219615167
Although moving large amounts of data is a heavy load, the load for metadata changes is not high, so metadata commits should be relatively fast. This is similar to concurrent writes, as long as the data involved in two commits does not conflict, the Sink job can complete the commit through retries. We have had successful practices with Iceberg in this area. Although I do not have extensive experience with concurrent writing in Paimon, based on the design of Paimon’s metadata, I believe it is feasible. @lijinsong could you please help confirm this? Thanks. > On Jul 10, 2024, at 09:59, PLASH SPEED ***@***.***> wrote: > > > @zhongyujiang <https://github.com/zhongyujiang> So.How? Changing the location means submitting a new metadata. This operation is always slow.If you don't stop streaming writes to this re-location-job, there is a high probability that it will fail. > > — > Reply to this email directly, view it on GitHub <https://github.com/apache/paimon/issues/3627#issuecomment-2219331119>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKHLOGGYKQQZKEXJONKOR5LZLSIR5AVCNFSM6AAAAABKBGAS2KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJZGMZTCMJRHE>. > You are receiving this because you were mentioned. > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
