awadhesh14 opened a new pull request, #19897:
URL: https://github.com/apache/hudi/pull/19897
### Describe the issue this Pull Request addresses
Closes #18727
Queries with empty projections against Lance-backed Hudi tables, such as
`df.count()`, `COUNT(*)`, `COUNT(1)`, `SELECT 1`, and `EXISTS(...)`, could
return incorrect results or fail.
For empty data projections, `SparkLanceReaderBase` previously returned
`Iterator.empty` when `requiredSchema.isEmpty && partitionSchema.isEmpty`,
causing queries that should produce one result per input row to see zero rows.
Other empty-schema cases reached `LanceRecordIterator` with a physical Lance
batch containing columns while the Spark schema contained none, resulting in
errors such as:
```
Lance batch column count X does not match expected Spark schema size 0
for file: ...lance
at org.apache.hudi.io.storage.LanceRecordIterator.hasNext
```
This can also occur under schema evolution when an older Lance file slice
does not contain any of the columns requested by Spark.
### Summary and Changelog
1. **DataSource-level empty projection handling**:
- Replaced the premature `Iterator.empty` path in
`SparkLanceReaderBase.scala` with an `iteratorSchema.isEmpty` branch.
- The branch uses the Lance file's metadata row count via
`lanceReader.numRows()` without reading or decoding data columns.
- Returns the corresponding number of initialized zero-field
`UnsafeRow`s, reusing a single row instance per iterator.
2. **Schema evolution and null padding**:
- When an older file slice does not contain a newly requested column,
`filterSchemaByFileSchema` can produce an empty `iteratorSchema`.
- The reader still emits the rows from that file using the metadata row
count.
- Spark's existing `generateNullPaddingProjection` then supplies `NULL`
values for the missing columns.
3. **Resource lifecycle**:
- Tracks the `LanceFileReader` explicitly so it can be closed if an
exception occurs before the iterator is constructed.
- Preserves normal iterator-owned cleanup for the Lance reader and Arrow
allocator.
- `LanceRecordIterator` remains unchanged and continues to handle Arrow
batch decoding.
4. **Integration regression tests**:
- Added end-to-end coverage in `TestLanceDataSource.scala` for:
- unpartitioned and partitioned empty projections
- `COUNT(*)`, `COUNT(1)`, and `SELECT 1`
- `EXISTS` and `LIMIT 1`
- data and partition predicates
- counts after deletes, including the zero-row case
- schema evolution where older file slices are missing newly added
columns
- both Copy-on-Write and Merge-on-Read tables
### Impact
- Fixes incorrect empty-projection results for Lance-backed Hudi tables.
- Avoids reading and decoding data columns for empty projections by using
the Lance file's metadata row count.
- Correctly preserves rows from older file slices during schema evolution
and produces `NULL` for columns that are absent from those slices.
- Keeps the implementation localized to the Spark Lance DataSource layer,
with changes limited to two files.
### Risk Level
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]