wuwenchi commented on PR #5061: URL: https://github.com/apache/iceberg/pull/5061#issuecomment-1161433895
For the data with id=2 1. Originally there is a piece of data, the operation is insert, which is `+I`, 2. The two data in the CDC data are `+I` and `-D` If the cdc data is regarded as a whole, and the cdc is inserted first and then deleted, the original piece of data should be left in the end. But in fact, on the whole, all operations on the data are `+I, +I, -D`, and the second `+I` was wrongly processed as an update to the original data, which eventually led to the disappearance of this data. (In fact, the second `+I` is an insert into the intermediate result set) So, in this case, should batch mode be used? Generate the final result directly for `select` in batch mode: ``` select t1.id as id ,t1.data as data from upsert_sample t1 left join upsert_test t2 on t1.id = t2.id where t1.data > t2.data or t2.data is null : +----+--------------------------------+--------------------------------+ | op | id | data | +----+--------------------------------+--------------------------------+ | +I | 1 | 20220607 | | +I | 3 | 20220505 | +----+--------------------------------+--------------------------------+ ``` This is just a temporary solution that I can think of. But in stream mode, there is indeed a problem with the result processed by iceberg now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
