nsivabalan opened a new pull request, #20749: URL: https://github.com/apache/hudi/pull/20749
### Describe the issue this Pull Request addresses Incremental queries can fail with a FileNotFoundException on `.hoodie/timeline/<instant>.commit` or `.deltacommit` while a writer archives the table. An incremental query lists the active timeline once and reads the commit metadata of each included instant later. Readers hold no lease on the timeline, so the writer's archiver can move a listed instant out of the active timeline in between. MERGE_ON_READ incremental planning reads commit metadata twice, and either read can hit the race: 1. the file index lists the written partitions while Spark optimizes the plan (`BaseHoodieTableFileIndex#listPartitionPaths`, incremental branch); 2. the relation reads the included commits while it collects file splits (`commitsMetadata` in the MOR incremental relations). The COPY_ON_WRITE relations read the metadata of the commits to return, including replace commits, after listing. Branch-0.x counterpart: #20748. ### Summary and Changelog **V1 relations (table version < 8):** `IncrementalRelationV1` and `MergeOnReadIncrementalRelationV1` treat an included instant whose metadata file is missing like an archived begin instant. - With `hoodie.datasource.read.incr.fallback.fulltablescan.enable` set, they fall back to a full table scan. - Otherwise they fail with `HoodieIncrementalPathNotFoundException`, which names that config. - The MOR relation checks this before the missing-files check, because that check reads the metadata of every included commit. **V2 relations:** these already read archived instants by design. When an included instant moves to the archived timeline after listing, `IncrementalRelationV2` and `MergeOnReadIncrementalRelationV2` now read its metadata from there. Each relation loads the archived timeline lazily, at most once. The reader is `TimelineUtils#readInstantContentOrArchived`, a new helper. If the instant is in neither timeline, the query fails with a `HoodieIOException` saying so. **File index:** when the incremental partition listing in `BaseHoodieTableFileIndex` hits a missing commit metadata file, it lists all partitions and leaves the decision to the relation. I considered an up-front check against the cleaner's retain point and rejected it. Such a check sends bounded queries that the incremental path can still serve to the full table scan. The full table scan filters the latest snapshot by `_hoodie_commit_time`, so a key written inside the range and updated after the end instant would be dropped. New test `TestIncrementalReadConcurrentArchival` (COW and MOR). It builds the relation, lets it list the timeline, archives included instants, then runs the scan. It covers: - V1 with the fallback enabled: the exact (commit time, record key) rows come back through the full table scan. - V1 with the fallback disabled: `HoodieIncrementalPathNotFoundException`, with the `FileNotFoundException` in its cause chain. - V2: the rows are read through the archived timeline. - V2: an instant that is missing from both timelines fails clearly. Verified: all 8 new cases fail without the main-code change, with `Could not read commit details`. With the change, they pass together with `TestIncrementalReadWithFullTableScan` and `TestIncrementalReadByStateTransitionTime` (19 tests). ### Impact Incremental reads no longer fail with a bare FileNotFoundException when the writer archives an included instant during query planning. No public API or config change. Not covered: the cleaner deleting data files after planning and before executors read them. That still fails as it does today. ### Risk Level low. Behaviour changes only when an included instant's metadata file is missing, and that case always failed the query before. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
