awadhesh14 opened a new pull request, #19897:
URL: https://github.com/apache/hudi/pull/19897

   ### Describe the issue this Pull Request addresses
   
   Closes #18727
   
   Queries with empty projections against Lance-backed Hudi tables, such as 
`df.count()`, `COUNT(*)`, `COUNT(1)`, `SELECT 1`, and `EXISTS(...)`, could 
return incorrect results or fail.
   
   For empty data projections, `SparkLanceReaderBase` previously returned 
`Iterator.empty` when `requiredSchema.isEmpty && partitionSchema.isEmpty`, 
causing queries that should produce one result per input row to see zero rows.
   
   Other empty-schema cases reached `LanceRecordIterator` with a physical Lance 
batch containing columns while the Spark schema contained none, resulting in 
errors such as:
   ```
   Lance batch column count X does not match expected Spark schema size 0
     for file: ...lance
     at org.apache.hudi.io.storage.LanceRecordIterator.hasNext
   ```
   This can also occur under schema evolution when an older Lance file slice 
does not contain any of the columns requested by Spark.
   
   ### Summary and Changelog
   
   1. **DataSource-level empty projection handling**:
      - Replaced the premature `Iterator.empty` path in 
`SparkLanceReaderBase.scala` with an `iteratorSchema.isEmpty` branch.
      - The branch uses the Lance file's metadata row count via 
`lanceReader.numRows()` without reading or decoding data columns.
      - Returns the corresponding number of initialized zero-field 
`UnsafeRow`s, reusing a single row instance per iterator.
   
   2. **Schema evolution and null padding**:
      - When an older file slice does not contain a newly requested column, 
`filterSchemaByFileSchema` can produce an empty `iteratorSchema`.
      - The reader still emits the rows from that file using the metadata row 
count.
      - Spark's existing `generateNullPaddingProjection` then supplies `NULL` 
values for the missing columns.
   
   3. **Resource lifecycle**:
      - Tracks the `LanceFileReader` explicitly so it can be closed if an 
exception occurs before the iterator is constructed.
      - Preserves normal iterator-owned cleanup for the Lance reader and Arrow 
allocator.
      - `LanceRecordIterator` remains unchanged and continues to handle Arrow 
batch decoding.
   
   4. **Integration regression tests**:
      - Added end-to-end coverage in `TestLanceDataSource.scala` for:
        - unpartitioned and partitioned empty projections
        - `COUNT(*)`, `COUNT(1)`, and `SELECT 1`
        - `EXISTS` and `LIMIT 1`
        - data and partition predicates
        - counts after deletes, including the zero-row case
        - schema evolution where older file slices are missing newly added 
columns
        - both Copy-on-Write and Merge-on-Read tables
   
   ### Impact
   
   - Fixes incorrect empty-projection results for Lance-backed Hudi tables.
   - Avoids reading and decoding data columns for empty projections by using 
the Lance file's metadata row count.
   - Correctly preserves rows from older file slices during schema evolution 
and produces `NULL` for columns that are absent from those slices.
   - Keeps the implementation localized to the Spark Lance DataSource layer, 
with changes limited to two files.
   
   ### Risk Level
   
   low
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to