vinishjail97 opened a new pull request, #824: URL: https://github.com/apache/incubator-xtable/pull/824
## What is the purpose of the pull request Fixes a bug where XTable's incremental Hudi → Delta sync can leave a stale `ADD` in the Delta log, causing `FILE_NOT_FOUND` on read. ## RCA (one line) Hudi `CLEAN` deletes an old Parquet file before XTable's next incremental sync runs, so `HudiDataFileExtractor.getUpdatesToPartition` can't see the deleted file via `getAllBaseFiles()` and skips emitting a `REMOVE` for it — leaving a stale `ADD` in the Delta log that points to a non-existent file; this PR recovers the deleted file path from the previous commit's metadata in the Hudi timeline (`HoodieWriteStat.prevCommit`) so the `REMOVE` is emitted correctly. ## Brief change log - `HudiDataFileExtractor.getUpdatesToPartition` now also takes `HoodieTimeline` and `List<HoodieWriteStat>` so it can build a `fileId → prevCommit` map. - After the base-file loop, when an `ADD` was emitted but no `REMOVE` was, look up the previous commit's metadata via `HoodieCommitMetadata.fromBytes(timeline.getInstantDetails(prevInstant).get(), ...)` and emit the `REMOVE` from `HoodieWriteStat.getPath()`. - Gracefully degrades (logs a warning and skips) if the previous commit has been archived or the read fails. ## Verify this pull request This change added tests and can be verified as follows: - Added `TestHudiDataFileExtractor` with three unit tests for `recoverRemovedFile`: previous commit present, previous commit archived, fileId not in previous commit metadata. - Added `ITConversionController.testIncrementalSyncEmitsRemoveWhenHudiCleanRunsBeforeSync`, an end-to-end regression test that reproduces the race (insert → sync → upsert → insert → clean → incremental sync) and asserts Delta equivalence at 120 records. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
