openinx commented on issue #2610: URL: https://github.com/apache/iceberg/issues/2610#issuecomment-843743215
What's your iceberg table schema? Is it possible that the rows with the same primary key located in two different partitions/buckets ? If sure, then we may encounter the duplicated rows that from two different partitions/buckets because we won't dedup the same rows from different partitions , it is high cost to accomplish that. Another thing, all those change log events come from MYSQLbinlog or flink aggregate results ? If they are came from binlog, then each UPDATE will emit a `UPDATE_BEFORE` and a `UPDATE_AFTER`, currently we iceberg only support update (rather than UPSERT, that means we won't delete the existing rows explicitily but we have pending PR to do this) rows, in theory we won't encounter duplicate rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
