ad1happy2go opened a new issue, #20286: URL: https://github.com/apache/hudi/issues/20286
### Bug Description **What happened:** An incremental query with `hoodie.datasource.read.incr.fallback.fulltablescan.enable=true` whose start instant is archived returns no rows, or only the rows of some partitions. This happens when a completed rollback (or clean) older than every active commit is still on the active timeline. No error is raised. This timeline is common on tables written by 0.14.x. The table version 6 archiver only archives rollback instants once there are more than `hoodie.keep.max.commits` of them, so a single rollback left by a failed write stays on the active timeline long after the commits around it are archived. It is kept when the table is upgraded to version 8 or 9. Two places take the first instant of any action on the active timeline as the end of the archived history: - `BaseHoodieTableFileIndex.isBeforeTimelineStarts()` checks the whole active timeline. The rollback makes an archived start look active, so the file index lists only the partitions written by the active commits in range (`findInstantsInRange()` on the write timeline). The relation has already chosen a full table scan, so it reads 0 partitions, or only the partitions touched by active commits. - `CompletionTimeQueryViewV2` (table version 8 and above) starts its cursor at the first active instant and treats everything after it as active. Archived commits between the rollback and the first active commit are never loaded: `IncrementalQueryAnalyzer` resolves the archived range to no instants, so the V2 relation never falls back, and `getCompletionTime` reports those commits as pending. **What you expected:** The same rows as a snapshot read filtered on `_hoodie_commit_time` for the same window. **Steps to reproduce:** 1. With Hudi 0.14.1 (Spark 3.4), write commit 1 to a COW table partitioned on `p`, then a write that fails during the write stages (for example with `assert_true(id < 0)` on a column), then commits 2 to 20, each into one of five partitions `p0`..`p4` in turn. Use `hoodie.keep.min.commits=2`, `hoodie.keep.max.commits=3`, `hoodie.cleaner.commits.retained=1`. The write after the failed one rolls it back, and archival keeps the rollback but archives most commits. 2. Check `.hoodie`: the active timeline starts at the `.rollback` instant, while the first active `.commit` is commit 19. 3. With Hudi 1.1.1 or 1.2.1, run fallback incremental reads (`hoodie.datasource.read.incr.fallback.fulltablescan.enable=true`) and compare with the snapshot: | Window | Expected rows | Returned | |---|---|---| | commit 5 → commit 10 (both archived) | 500 | **0** | | commit 5 → latest | 1500 | **600** (2 of 5 partitions) | | commit 19 → latest (active) | 100 | 100 | The same table written with `--no-rollback` (no failed write) returns the expected rows. Reproduced on table version 6, and after upgrading it to version 9. ### Environment **Hudi version:** 1.1.1, 1.2.1 and master (table written by 0.14.1) **Query engine:** Spark 3.4 / 3.5 **Relevant configs:** `hoodie.datasource.query.type=incremental`, `hoodie.datasource.read.incr.fallback.fulltablescan.enable=true`, archived `hoodie.datasource.read.begin.instanttime`; COW and MOR ### Logs and Stack Trace No exception. The file index logs a full table listing with no partitions: ``` listing full table (0 partitions) ``` **Fixed by:** #20273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
