awadhesh14 opened a new pull request, #19897:
URL: https://github.com/apache/hudi/pull/19897
### Describe the issue this Pull Request addresses
Queries with empty projections against Lance-backed Hudi tables (such as
`df.count()`, `COUNT(*)`, `COUNT(1)`, `SELECT 1`, and `EXISTS(...)`) previously
returned 0 rows or failed.
`SparkLanceReaderBase` previously had a short-circuit returning
`Iterator.empty` when `requiredSchema.isEmpty && partitionSchema.isEmpty`.
Furthermore, if `requiredSchema.isEmpty` or if an older file slice lacked all
requested columns under schema evolution, `filterSchemaByFileSchema` produces
an empty `iteratorSchema`, which caused schema-validation checks in
`LanceRecordIterator` to fail or misalign.
### Summary and Changelog
1. **Metadata-only empty projection iterator**:
- Replaced premature `Iterator.empty` in `SparkLanceReaderBase.scala`
with a branch on `iteratorSchema.isEmpty` to
`LanceRecordIterator.emptyProjectionIterator(...)`.
- Obtains the exact row count from Lance file metadata via
`lanceReader.numRows()`, eliminating data-column I/O for count and
empty-projection queries.
- Yields valid, zero-field `UnsafeRow`s using a reusable, fully
initialized singleton (`row.pointTo(new byte[0], 0)`).
2. **Schema evolution and null padding**:
- When an older file slice lacks newly added columns requested by Spark,
`filterSchemaByFileSchema` produces an empty `iteratorSchema`. The reader emits
$N$ metadata rows with zero data-column I/O, which are then correctly populated
with `NULL`s via
`SparkSchemaTransformUtils.generateNullPaddingProjection(iteratorSchema,
requiredSchema)`.
3. **Safety & Invariants**:
- Guarded reader lifecycle in `SparkLanceReaderBase.scala` so that
`lanceReader` is closed on exception paths if an exception occurs before the
iterator is initialized.
- Handled zero-field Spark schemas in
`LanceRecordIterator.buildColumnVectors` returning `new ColumnVector[0]`.
- Verified the MOR merge invariant: uncompacted log files on MOR trigger
file-group reading which injects mandatory merge keys (`_hoodie_record_key`,
precombine, delete markers), correctly routing through the merge path rather
than this metadata-only shortcut.
4. **Comprehensive Tests**:
- Added unit tests in `TestLanceRecordIterator` covering
`createEmptyUnsafeRow`, lifecycle, zero-row edge cases, and empty schema batch
handling with clean Arrow buffer management.
- Added unit test in `TestSparkSchemaTransformUtils` verifying null
padding from an empty input schema.
- Added integration regression tests in `TestLanceDataSource` covering
unpartitioned count, partitioned count, zero-row counts after delete, and
multi-commit schema evolution with missing columns across COW and MOR.
### Impact
- Fixes correctness for `df.count()`, `COUNT(*)`, `SELECT 1`, and empty
projections on Lance tables.
- Drastically speeds up count queries on Lance tables by avoiding
unnecessary data-column scans and deriving cardinality directly from file
metadata (`numRows()`).
- Correctly handles schema evolution queries requesting only newly added
columns against older Lance files.
### Risk Level
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
