andygrove opened a new issue, #5097:
URL: https://github.com/apache/datafusion-comet/issues/5097
### Describe the bug
`cast_map_to_map` in `native/spark-expr/src/conversion_funcs/cast.rs`
rebuilds the map's keys, values, entries struct, and offsets by hand, mainly to
reconcile Parquet field naming ("key_value") with Spark naming ("entries"). In
doing so it diverges from arrow's map cast in two ways:
1. It passes `None` for the entries-struct null buffer, dropping
`map_array.entries().nulls()` if present.
2. It always propagates the source's `sorted` flag (`*from_sorted`) and
ignores the target's `_to_sorted`.
Arrow's Map-to-Map `cast_with_options` preserves both correctly.
### Steps to reproduce
Latent today: Spark-produced maps do not carry null map entries, so the
dropped null buffer is not currently observable through normal plans. The
incorrect `sorted` flag is metadata-only.
### Expected behavior
For the rename-only case (`from_key_type == to_key_type && from_value_type
== to_value_type`, the common Parquet case), delegate to arrow's map cast (or
the existing zero-copy `relabel_array` in `core/src/parquet/cast_column.rs`),
which handles both issues. Keep the hand-rolled recursion only when key or
value types genuinely change and need Spark cast semantics.
### Additional context
Found during an audit of native code that replicates existing arrow-rs
kernels. Filed as a bug because the current code drops data (a null buffer)
relative to its input, even though no user-visible query is known to be
affected yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
