nsivabalan opened a new pull request, #19568:
URL: https://github.com/apache/hudi/pull/19568
### Describe the issue this Pull Request addresses
**A CoW incremental query returns zero rows whenever its projection excludes
`_hoodie_commit_time`.** `count()` is the most visible instance, and it fails
silently — a wrong answer that reads as "no new data" rather than an error.
Reproduced on unmodified `master` (`65cf7e8ede7d`), table version 10, on a
plain CoW table with no special configuration:
```
collect() -> 2 rows (correct)
count() -> 0 rows (WRONG)
select("k").collect() -> 0 rows (WRONG)
select("k").count() -> 0 rows (WRONG)
snapshot count() -> 4 rows (correct — snapshot reads are unaffected)
```
Not specific to a table version, a table type beyond CoW, or any meta-fields
configuration.
### Root cause
An incremental query filters on `_hoodie_commit_time` — `IsNotNull` plus an
`In` over the instants in range — and those filters are pushed into the file
reader as `requiredFilters`. The reader evaluates a pushed predicate against
**the schema it was asked to read**, which contains only the columns the query
projects.
When the projection omits `_hoodie_commit_time`, the predicate cannot be
satisfied and every row is dropped.
`collect()` works only *incidentally*: it happens to project every column,
including the filtered one. It is not a working case so much as a case that has
not failed yet. `count()` projects nothing at all, and `select(subset)`
projects only what the user asked for — both hit it.
Verified by instrumenting `readBaseFile` rather than by inspection. The
filters are identical in all three cases; only the schema differs:
| Query | schema passed to the reader | filters | rows |
|---|---|---|---|
| `collect()` | all 7 columns, incl. `_hoodie_commit_time` |
`IsNotNull(_hoodie_commit_time)`, `In(...)` | 2 ✅ |
| `count()` | `[]` | *identical* | 0 ❌ |
| `select("k")` | `[k]` | *identical* | 0 ❌ |
Worth noting what is **not** the bug, since both look suspicious and both
are correct:
- `isCount` excludes incremental deliberately
(`HoodieFileGroupReaderBasedFileFormat:254`). You cannot answer an incremental
`count()` from row counts, because only some rows fall in the commit range.
- The fallthrough to the plain Parquet reader for column-less CoW reads is
intentional — see HUDI-8079 (`31eea3b9a851`), which only restructured an inner
`if` into the guard.
Both mechanisms are right in isolation. The defect is the seam between them:
**a required filter references a column, and nothing guarantees that column is
in the schema handed to the reader.**
### Summary and Changelog
`readBaseFile` now reads any column a required filter references but the
projection omits, then projects it back out before returning so the caller's
schema stays exact.
Two details worth review:
- **Partition columns are excluded.** The caller appends those from
partition values rather than reading them from the file, so adding one here
would duplicate the field and shift every ordinal after it. Not reachable by
the repro above — it needs a filter over a partition column — but wrong
regardless.
- **Candidate columns come from the full table schema**, not the requested
schema. On the failing path the requested schema is itself empty, so it cannot
supply the column the filter needs. (My first attempt sourced them from the
requested schema; it compiled, read correctly, and did nothing.)
### Impact
**Behavior**: incremental queries that previously returned zero rows now
return the correct rows. No other query shape changes.
**Strict no-op for every non-incremental read.** Snapshot, CDC and bootstrap
relations all supply `Seq.empty` for required filters, so there is nothing to
reference and nothing to add. No extra column is read on the ordinary `count()`
path — which matters, since that is the most common query in Hudi.
**Performance**: on an incremental read whose projection omits the filtered
column, one extra column is read and then projected away. That is the minimum
needed to evaluate the predicate correctly.
**API / storage**: unchanged.
### Risk Level
medium
The change is small and its no-op property is structural rather than
incidental. Raised above "low" only because `readBaseFile` is on the shared
read path for every Spark query — snapshot, incremental, CDC, bootstrap and MoR
all route through it — so the blast radius of a mistake here is wide even
though this change is narrow.
### Tests
`TestIncrementalQueryProjection`, four cases:
1. `count()` agrees with `collect()`
2. projected reads agree with unprojected ones
3. a projected read returns the **right rows**, not merely the right number
— a filter that silently matched everything would pass a count-only assertion
4. snapshot reads are unaffected, guarding the no-op property
**Verified not vacuous:** with the fix reverted, exactly the three
incremental cases fail and the snapshot guard still passes. That check mattered
here — the first version of this fix compiled cleanly and did nothing, and a
test run only against the fixed build would have looked identical.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
