max-parke-scale opened a new issue, #51029:
URL: https://github.com/apache/arrow/issues/51029
### Describe the bug, including details regarding any error messages,
version, and platform.
Casting a map array still hard-aborts the process (SIGABRT, not a raised
exception) with `Map array keys array should have no nulls` when the keys child
carries a validity bitmap — even an all-valid one. This is a residual case of
#38553: the fix in #41957 changed `MapArray::ValidateChildData` to use
`MayHaveNulls()`, which resolved the no-bitmap repro on that issue, but
`MayHaveNulls()` answers true whenever a validity bitmap is present and the
null count is unknown — it never counts. A cast's output starts with an unknown
null count, so the fatal `ARROW_CHECK_OK(ValidateChildData(...))` in
`MapArray::SetData` (`array_nested.cc:912` as of current main) kills the
process.
Three-line reproduction, all public API, on pyarrow 24.0.0 (the current
release):
```python
import pyarrow as pa
m = pa.array([{"a": "1"}, {"b": "2"}], type=pa.map_(pa.string(),
pa.string()))
m.filter(pa.array([False, True])).cast(pa.map_(pa.large_string(),
pa.large_string()))
```
```
/Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/array/array_nested.cc:912:
Check failed: _s.ok() Operation failed: ValidateChildData(data->child_data)
Bad status: Invalid: Map array keys array should have no nulls
```
Exit code 134. The `filter` call itself succeeds and the filtered array
passes `validate(full=True)`.
`filter` matters only because it materializes new child arrays that carry an
all-valid validity bitmap; `slice` re-views the original buffers without one,
which is why the slice→cast variant works after #41957. The bitmap alone is
sufficient — same abort with no filter involved:
```python
import pyarrow as pa
keys = pa.array(["a", "b"])
validity = pa.array([True, True]).buffers()[1] # all-valid bitmap
keys = pa.Array.from_buffers(pa.string(), 2, [validity, keys.buffers()[1],
keys.buffers()[2]])
m = pa.MapArray.from_arrays(pa.array([0, 1, 2], type=pa.int32()), keys,
pa.array(["1", "2"]))
m.cast(pa.map_(pa.large_string(), pa.large_string())) # aborts
```
Forcing null-count computation on the input (`m.keys.null_count` → 0) does
not help: the cast produces a fresh keys `ArrayData` whose count is unknown
again.
Real-world impact: pyiceberg's scan path does filter-then-cast (row-filtered
Parquet read, then cast to large types), so any row-filtered scan that projects
a map column kills the worker process.
Two candidate fixes, possibly both:
- `MapArray::ValidateChildData` (`array_nested.cc:899`/`905`) could
force-compute the null count (`GetNullCount() != 0`) rather than ask
`MayHaveNulls()`; counting an all-valid bitmap is cheap next to aborting the
process.
- The `ARROW_CHECK_OK` at `array_nested.cc:912` turns an `Invalid` status
into process death even when the data is genuinely invalid; a returned/raised
error would be recoverable.
Environment: pyarrow 24.0.0 (PyPI wheel), Python 3.13.13, macOS 26.4.1 arm64.
### Component(s)
C++, Python
🤖 — posted via [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
