max-parke-scale opened a new issue, #4114:
URL: https://github.com/apache/iceberg-python/issues/4114
### Apache Iceberg version
0.11.1 (also present in 0.12.0 and on main)
### Please describe the bug 🐞
`ArrowProjectionVisitor.map()` rebuilds a map whose values are structs with
`pa.MapArray.from_arrays(map_array.offsets, key_result, value_result,
arrow_field)`. The offsets array carries no nulls, so the map's validity bitmap
is dropped and a null map reads back as an empty one. Maps with non-struct
values go through `map_array.cast()` and keep their nulls. This is the map
counterpart of #3833; #4065 fixes `list()` only.
```python
import pyarrow as pa
from pyiceberg.io.pyarrow import _to_requested_schema
from pyiceberg.schema import Schema
from pyiceberg.types import MapType, NestedField, StringType, StructType
value = StructType(NestedField(3, "source", StringType(), required=False))
schema = Schema(
NestedField(1, "m", MapType(4, StringType(), 5, value,
value_required=False), required=False)
)
source = pa.struct([("source", pa.string())])
batch = pa.record_batch({"m": pa.array([[("d", {"source": "x"})], None],
type=pa.map_(pa.string(), source))})
print(_to_requested_schema(schema, schema, batch).column(0).to_pylist())
# [[('d', {'source': 'x'})], []] expected: [[('d', {'source': 'x'})], None]
```
Keeping the original validity and offsets buffers, as #4065 does for lists,
fixes it:
```python
projected = pa.MapArray.from_arrays(map_array.offsets, key_result,
value_result, arrow_field)
return pa.Array.from_buffers(
projected.type, len(map_array), map_array.buffers()[:2],
map_array.null_count, map_array.offset, [projected.values],
)
```
🧑💻🤖 — posted via [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
