voonhous commented on PR #19236: URL: https://github.com/apache/hudi/pull/19236#issuecomment-4925349173
Our original test did not capture this error because: 1. The topology never forced the dangerous path. With default shuffle parallelism, the bulk_insert scattered ids `0..5` across multiple file groups. The upsert of ids `0..2` only produced log files in their file groups. Compaction only rewrites file groups that have log files, so the untouched ids `3..5` sat in log-free file groups that compaction never touched. Their bytes "survived" because their base file was never rewritten at all under compaction. The test even asserted that a deltacommit existed and that some file slice had log files, but never that an untouched row and a log file shared a filegroup, which is the only combination that exercises the carry-over rewrite. 2. Even the rewritten rows didn't test the base-file read. Ids `0..2` got their new bytes from the log records during the merge, not from reading the base file. So whatever blob mode the base-file read used (the actual bug) was invisible: the buggy read's output for those rows was discarded in favor of the log records anyway. TLDR: New tests forces `3..5` into the same filegroup that needed to be re-read and rewritten into new base file under compaction but were never present in the log files, triggering the data loss error which we are fixing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
