nsivabalan opened a new pull request, #20749:
URL: https://github.com/apache/hudi/pull/20749

   ### Describe the issue this Pull Request addresses
   
   Incremental queries can fail with a FileNotFoundException on 
`.hoodie/timeline/<instant>.commit` or `.deltacommit` while a writer archives 
the table.
   
   An incremental query lists the active timeline once and reads the commit 
metadata of each included instant later. Readers hold no lease on the timeline, 
so the writer's archiver can move a listed instant out of the active timeline 
in between. MERGE_ON_READ incremental planning reads commit metadata twice, and 
either read can hit the race:
   
   1. the file index lists the written partitions while Spark optimizes the 
plan (`BaseHoodieTableFileIndex#listPartitionPaths`, incremental branch);
   2. the relation reads the included commits while it collects file splits 
(`commitsMetadata` in the MOR incremental relations).
   
   The COPY_ON_WRITE relations read the metadata of the commits to return, 
including replace commits, after listing.
   
   Branch-0.x counterpart: #20748.
   
   ### Summary and Changelog
   
   **V1 relations (table version < 8):** `IncrementalRelationV1` and 
`MergeOnReadIncrementalRelationV1` treat an included instant whose metadata 
file is missing like an archived begin instant.
   - With `hoodie.datasource.read.incr.fallback.fulltablescan.enable` set, they 
fall back to a full table scan.
   - Otherwise they fail with `HoodieIncrementalPathNotFoundException`, which 
names that config.
   - The MOR relation checks this before the missing-files check, because that 
check reads the metadata of every included commit.
   
   **V2 relations:** these already read archived instants by design. When an 
included instant moves to the archived timeline after listing, 
`IncrementalRelationV2` and `MergeOnReadIncrementalRelationV2` now read its 
metadata from there. Each relation loads the archived timeline lazily, at most 
once. The reader is `TimelineUtils#readInstantContentOrArchived`, a new helper. 
If the instant is in neither timeline, the query fails with a 
`HoodieIOException` saying so.
   
   **File index:** when the incremental partition listing in 
`BaseHoodieTableFileIndex` hits a missing commit metadata file, it lists all 
partitions and leaves the decision to the relation.
   
   I considered an up-front check against the cleaner's retain point and 
rejected it. Such a check sends bounded queries that the incremental path can 
still serve to the full table scan. The full table scan filters the latest 
snapshot by `_hoodie_commit_time`, so a key written inside the range and 
updated after the end instant would be dropped.
   
   New test `TestIncrementalReadConcurrentArchival` (COW and MOR). It builds 
the relation, lets it list the timeline, archives included instants, then runs 
the scan. It covers:
   - V1 with the fallback enabled: the exact (commit time, record key) rows 
come back through the full table scan.
   - V1 with the fallback disabled: `HoodieIncrementalPathNotFoundException`, 
with the `FileNotFoundException` in its cause chain.
   - V2: the rows are read through the archived timeline.
   - V2: an instant that is missing from both timelines fails clearly.
   
   Verified: all 8 new cases fail without the main-code change, with `Could not 
read commit details`. With the change, they pass together with 
`TestIncrementalReadWithFullTableScan` and 
`TestIncrementalReadByStateTransitionTime` (19 tests).
   
   ### Impact
   
   Incremental reads no longer fail with a bare FileNotFoundException when the 
writer archives an included instant during query planning. No public API or 
config change.
   
   Not covered: the cleaner deleting data files after planning and before 
executors read them. That still fails as it does today.
   
   ### Risk Level
   
   low. Behaviour changes only when an included instant's metadata file is 
missing, and that case always failed the query before.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to