JingsongLi commented on code in PR #9789:
URL: https://github.com/apache/paimon/pull/9789#discussion_r4005690072
##########
paimon-python/pypaimon/read/table_read.py:
##########
@@ -1020,9 +1033,50 @@ def _widen_to_top_level_for_merge(self) ->
List[DataField]:
raise ValueError(
"Nested projection top-level field %r not found in "
"table schema" % (top_name,))
+ paths = paths_by_top[top_name]
+ if (isinstance(field.type, MapType)
+ and all(len(path) > 1 for path in paths)
+ and not self._map_has_aggregator(top_name)):
+ keys = []
+ for path in paths:
+ if path[1] not in keys:
+ keys.append(path[1])
+ try:
+ field = map_selected_keys_field(field, keys)
Review Comment:
The ROW and row-sidecar cases now pass. I think key projection should
support `MAP<STRING, BLOB>` as well, since this is an existing supported table
type.
There is still a failure on `88cafac`: create a data-evolution table with
`payload MAP<STRING, BLOB>` and write `{'k': b'hello', 'v': b'world'}`. Reading
the complete MAP succeeds, but `with_projection(["payload['k']"])` raises
`ArrowTypeError`. Projecting both the complete MAP and the key also succeeds.
The selected-key ROW type reaches `FormatBlobReader`, so it no longer
recognizes the field as a MAP of BLOBs and tries to read it as a scalar BLOB.
The failure happens before the new conversion in `DataFileBatchReader`. Please
preserve the physical MAP schema for BLOB decoding and then extract the
selected key, or use the complete-MAP fallback for this layout.
Please also add an end-to-end regression test that writes a real
MAP-of-BLOBs table and verifies key-only projection returns the expected bytes.
Cover missing keys, null values, and null/empty MAPs, and compare the results
with complete-MAP reads so the test exercises the actual BLOB reader path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
