Baunsgaard opened a new pull request, #3658:
URL: https://github.com/apache/iceberg-rust/pull/3658
## Which issue does this PR close?
- Closes #3557.
## What changes are included in this PR?
Parquet can only read a map when both its key and value are projected. When
a map value is a struct and none of its leaves match the table (a name-mapped
physical name that is not in the mapping, or embedded ids that are not in the
table schema), only the key was selected. Converting that pre-projected schema
failed with `Map field must have exactly 2 fields`. Iceberg Java reads the same
files.
1. Convert each selected leaf on its own instead of converting a
pre-projected schema that can contain a key-only map. Duplicate field ids in
the file are still rejected.
2. After the mask is built, walk the Parquet schema (detecting maps the same
way parquet-rs does). If a map's key is read but none of its value leaves are,
also read the value leaves. They match no table field, so the value reads as a
struct of NULLs (#3437).
3. Only struct values are rescued. A primitive value is matched by position
and would be cast to the table type, so that case keeps failing as before.
4. Leaves rejected by `type_promotion_is_valid` are never rescued. If every
value leaf is rejected, the read fails with `DataInvalid` instead of a Parquet
"partial projection" error.
Only reads that failed before select extra columns. A map whose value is
read but whose key is not still fails as on main.
Selected leaf indices come from Arrow `filter_leaves`, while map key/value
ranges are computed from the Parquet schema. Those two leaf numberings disagree
only when a nested `ListView` (or similar) appears before the map in the file.
That mismatch already mis-projects columns on `apache/iceberg-rust main` today
(#3609); open PR #3624 fixes it. This PR does not change that behavior.
Related open work:
- #3438 covers the embedded-id form of this failure, not the name-mapped
case in #3557.
- #3624 avoids the "Map field must have exactly 2 fields" conversion error
by flattening selected leaves, but without the map-value rescue #3557 still
fails in Parquet with "partial projection of MapArray".
### When the read failed
Typical case:
```text
Table (with name mapping): File (no field ids):
2: m map< m: map<
3: key string, key: string,
4: value struct< value: struct<
5: x int old_x: int // not in mapping
>> >>
```
Here x should result in null, not as it is now crashing.
## Are these changes tested?
Yes, in `arrow::reader::projection`:
- `test_name_mapped_map_value_with_unmapped_child_reads_null`: name-mapped
`x` reads values; `old_x` (#3557) reads a struct of NULLs.
- `test_map_value_with_unmatched_embedded_id_reads_null`: file id 5, table
id 6. Also checks that a read projecting only `id` leaves the unprojected map
alone.
- `test_map_value_with_invalid_promotion_is_not_cast`: int to string is
rejected with `DataInvalid`; with an unmatched sibling the map reads and the
rejected leaf is NULL.
- `test_map_primitive_value_with_unmatched_id_is_not_cast`: primitive value
with a mismatched id errors instead of casting `"12"` to `12`.
The first three fail without the fix; the last checks that a mismatched
primitive value is not added to the projection.
## AI Disclosure
Cursor assisted with the implementation, regression tests, and local review.
I reviewed the resulting changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
