nsivabalan opened a new pull request, #20748: URL: https://github.com/apache/hudi/pull/20748
### Describe the issue this Pull Request addresses Incremental queries can fail with a FileNotFoundException on `.hoodie/<instant>.commit` or `.hoodie/<instant>.deltacommit` while a writer archives the table. An incremental query lists the active timeline once and reads the commit metadata of each included instant later. Readers hold no lease on the timeline, so the writer's archiver can move a listed instant out of the active timeline in between. MERGE_ON_READ incremental planning reads commit metadata twice, and either read can hit the race: 1. the file index lists the written partitions while Spark optimizes the plan (`BaseHoodieTableFileIndex#listPartitionPaths`, incremental branch); 2. the relation reads the included commits while it collects file splits (`MergeOnReadIncrementalRelation#commitsMetadata`). COPY_ON_WRITE (`IncrementalRelation`) reads the metadata of the commits to return, including replace commits, after listing. Today the query fails with `HoodieIOException: Could not read commit details from ...`, which names neither the cause nor the remedy. When the begin or end instant is archived before the query starts, Hudi already handles it: it falls back to a full table scan if `hoodie.datasource.read.incr.fallback.fulltablescan.enable` is set, and otherwise fails with `HoodieIncrementalPathNotFoundException`. ### Summary and Changelog An included instant whose commit metadata file is gone from the active timeline is now treated like an archived begin instant. - `MergeOnReadIncrementalRelation` and `IncrementalRelation` fall back to a full table scan when `hoodie.datasource.read.incr.fallback.fulltablescan.enable` is set. Otherwise they fail with `HoodieIncrementalPathNotFoundException`, which names that config. The MOR relation checks this before the existing missing-files check, because that check reads the metadata of every included commit. - `BaseHoodieTableFileIndex`: when the incremental partition listing hits a missing commit metadata file, it lists all partitions and leaves the decision to the relation. - New helper `ExceptionUtil#hasCause(Throwable, Class)`. Readers still do not read the archived timeline. I also considered an up-front check against the cleaner's retain point and rejected it. Such a check sends bounded queries that the incremental path can still serve to the full table scan, and the full table scan filters the latest snapshot by `_hoodie_commit_time`. A key written inside the range and updated after the end instant would then be dropped. New test `TestIncrementalReadConcurrentArchival` (COW and MOR). It builds the relation, lets it list the timeline, archives included instants, then runs the scan. - With the fallback enabled, it asserts the exact (commit time, record key) rows. Only the full table scan can return these rows, because the archived metadata can no longer be read. - With the fallback disabled, it asserts a `HoodieIncrementalPathNotFoundException` whose cause chain contains the `FileNotFoundException`. Verified: all 4 new cases fail without the main-code change, with `Could not read commit details`. With the change, they pass together with `TestIncrementalReadWithFullTableScan` and `TestIncrementalReadByStateTransitionTime` (8 tests). ### Impact Incremental reads no longer fail with a bare FileNotFoundException when the writer archives an included instant during query planning. If the fallback is enabled they do a full table scan, as they already do for an archived begin instant. Otherwise they fail with `HoodieIncrementalPathNotFoundException`. No public API or config change. Not covered: the cleaner deleting data files after planning and before executors read them. That still fails as it does today. ### Risk Level low. Behaviour changes only when an included instant's metadata file is missing, and that case always failed the query before. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
