0lai0 opened a new pull request, #1402:
URL: https://github.com/apache/mahout/pull/1402
### Related Issues
<!-- Closes #123 -->
Closes #1401
Part of #1338
### Changes
- [x] Bug fix
- [ ] New feature
- [ ] Refactoring
- [ ] Documentation
- [ ] Test
- [ ] CI/CD pipeline
- [ ] Other
### Why
<!-- Why is this change needed? -->
Reading a nullable `List<T>` column with a null outer row (e.g. `[[1, 2],
null, [3, 4]]`) fails with:
`InvalidInput("Inconsistent sample sizes: expected 2, got 0")`
Root cause: sample-size validation calls `ListArray::value_length(i)`
without checking `is_null(i)`. For a null outer row, Arrow returns 0, which the
reader treats as a length-0 list.
If a non-null row was seen first → validation fails with `expected N, got 0`
If the null row is row 0 → sample_size is seeded to 0 and corrupts all
subsequent rows
This is pre-existing (not introduced by #1393) and affects three readers:
- ParquetReader
- ParquetStreamingReader
- ArrowIPCReader
`NullHandling::FillZero` does not help because the failure happens at
list-length validation, before value filling runs.
### How
<!-- What was done? -->
**Null outer row semantics**
| Policy | Behavior |
|--------|----------|
| `Reject` | Return `InvalidInput` with a clear message |
| `FillZero` | Fill `sample_size` zeros once `sample_size` is known |
| `sample_size` unknown | Skip the null row (no zeros, not counted in
`num_samples`) — e.g. leading all-null batch before any non-null row
establishes size |
**Code changes**
1. **`ParquetReader`** (`parquet.rs`)
- Add `is_null(i)` guard in sample-size validation loop
- Split fast path (no null outer rows) vs slow path (per-row null
handling per `NullHandling`)
2. **`ParquetStreamingReader`** (`parquet.rs`)
- Seed `sample_size` from first non-null row instead of row 0
- Skip all-null batches when size is unknown
- Same null-handling policy as batch reader
- Defensive `.max(1)` on sample-size divisor in `read_batch`
3. **`ArrowIPCReader`** (`arrow_ipc.rs`)
- Two-phase List handling
- Phase 1: validate size from non-null rows only
- Phase 2: collect data with the same null policy
## Checklist
- [x] Added or updated unit tests for all changes
- [x] Added or updated documentation for all changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
