lichuang opened a new pull request, #12150:
URL: https://github.com/apache/arrow-rs/pull/12150
# Which issue does this PR close?
- Closes #11902
# Rationale for this change
`MutableArrayData::try_with_capacities` used a single `use_nulls` flag for
the whole recursive output tree. A nullable parent therefore forced every child
to allocate a validity bitmap, even when the child was `DataType::Null` or when
the child source had no nulls and its values were already covered by the parent
null mask.
For nested types such as `FixedSizeList<FixedSizeList<Null>>`, fixed-size
list capacities multiply, so a valid input with large list sizes could request
an enormous null bitmap for a child that stores no data. This is an avoidable
allocation and memory-exhaustion vector.
# What changes are included in this PR?
- Never allocate a null bitmap for `DataType::Null`.
- Decide child null-bitmap allocation from the child source data and child
type instead of blindly propagating the parent `use_nulls` flag.
- Allow parent nulls to be represented without forcing child validity
bitmaps for types that can be filled with valid placeholders.
- Keep the existing null handling for `Dictionary`, `Union` and
`RunEndEncoded`, where placeholder values or top-level validity are not
sufficient.
- Add an internal `try_extend_nulls_inner(len, mark_null)` path:
- `mark_null = true` marks appended slots as null.
- `mark_null = false` appends valid placeholders covered by a parent null
mask.
- `DataType::Null` only updates length and null count.
- Update `Struct` and `FixedSizeList` null extension to use the placeholder
path for children rather than requiring child validity bitmaps.
- Update the related documentation comments.
# Are these changes tested?
Yes. New regression tests cover:
- `DataType::Null` never allocating a null bitmap, including when `use_nulls
= true`.
- The issue reproducer with nested `FixedSizeList<FixedSizeList<Null>>`,
asserting that neither the inner list nor the null leaf allocates a null bitmap.
- A nullable `Struct` whose non-null child does not allocate a null bitmap.
- Extending the nested fixed-size list with nulls, freezing the result, and
validating the resulting `ArrayData`.
The `arrow-data` tests were run with and without `force_validate`, and the
existing `arrow` array transform tests were also run. Clippy and formatting
checks pass.
# Are there any significant user-facing changes?
No public API changes.
One behavior difference: `try_extend_nulls` on a `MutableArrayData` of type
`DataType::Null` now succeeds even when it was created with `use_nulls =
false`, because a `Null` array can always represent nulls and needs no validity
bitmap.
# AI usage disclosure
This change and its tests were prepared with AI assistance. I reviewed the
implementation, understand the change, and verified the relevant tests, Clippy,
and formatting locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
