ericyuan915 opened a new pull request, #19210:
URL: https://github.com/apache/hudi/pull/19210
closes #19208
### Describe the issue this Pull Request addresses
NestedColumnReader#readRow collapses a present row whose children are all
null into a NULL row. That loop iterated up to rowPosition.getPositionsCount(),
which on a full, non-final vectorized batch is one larger than the materialized
child vectors: the Dremel level stream carries a one-record lookahead
(NestedPrimitiveColumnReader#readAndNewVector reads one value past the batch,
and #getLevelDelegation keeps that trailing level for the next batch). Reading
a nested ROW column from a COW base file with more rows than the 2048-row batch
(RecordIterators.DEFAULT_BATCH_SIZE) therefore indexed one element past a
shorter child vector and threw ArrayIndexOutOfBoundsException.
### Summary and Changelog
Clamp the collapse loop to the shortest vector it indexes (the row vector
and every child), unwrapping ParquetDecimalVector (a non-AbstractHeapVector
DECIMAL child) via a new vectorLength helper; the phantom trailing position is
never surfaced downstream (ParquetColumnarRowSplitReader caps the batch at
num). Applied identically to all five flink modules
(1.18.x/1.19.x/1.20.x/2.0.x/2.1.x). Adds integration test
ITTestHoodieDataSource#testParquetNestedRowExceedingReadBatch covering both a
heap-vector nested row and an isolated decimal-only nested row across the batch
boundary.
### Impact
Reads of nested ROW columns from COW base files larger than one vectorized
read batch no longer crash with `ArrayIndexOutOfBoundsException`.
### Risk Level
Low
Bounded change to a single read loop; covered by a new integration test.
<!-- Accepted values: none, low, medium or high. Other than `none`, explain
the risk.
If medium or high, explain what verification was done to mitigate the
risks. -->
### Documentation Update
None.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
