leaves12138 commented on code in PR #887:
URL: https://github.com/apache/paimon-rust/pull/887#discussion_r4056693341
##########
crates/paimon/src/arrow/shredding/map.rs:
##########
@@ -1078,6 +1104,57 @@ impl MapShreddingReadPlan {
contexts,
}))
}
+
+ /// Return the physical struct children needed for one logical field.
+ /// `None` means the whole root field must be decoded.
+ pub(crate) fn projected_physical_children(&self, field_name: &str) ->
Option<HashSet<String>> {
+ let field_index = self
+ .logical_fields
+ .iter()
+ .position(|field| field.name() == field_name)?;
+ let ctx = self.contexts.get(&field_index)?;
+ ctx.selected_names.as_ref()?;
+
+ let mut children = HashSet::from([FIELD_MAPPING_NAME.to_string()]);
+ children.extend(
+ ctx.selected_columns
+ .iter()
+ .map(|index| physical_column_name(*index)),
+ );
+ if ctx.include_overflow {
+ children.insert(OVERFLOW_NAME.to_string());
+ }
+ Some(children)
+ }
+}
+
+fn selected_map_keys(field: &DataField) -> Result<Option<HashSet<String>>> {
+ let Some(description) = field.description() else {
+ return Ok(None);
+ };
+ let Some(encoded) = description.strip_prefix(SELECTED_KEYS_PREFIX) else {
+ return Ok(None);
+ };
+ if encoded.is_empty() {
Review Comment:
[P2] Preserve the valid empty-string MAP key
An empty encoded suffix does not necessarily mean an empty key list:
`project_nested_field` encodes the valid selection `[""]` as
`__PAIMON_MAP_SELECTED_KEYS:`. PyPaimon's `map_selected_keys_field` accepts
this selection, and `map_selected_keys` decodes the suffix to `[""]`, but this
check rejects it.
I reproduced this against the real extension with an append table containing
`attrs: MAP<STRING, INT>`, `fields.attrs.map.storage-layout=shared-shredding`,
and a row `{'attrs': {'': 10, 'other': 11}}`. `with_projection(["attrs['']"])`
returns `10` on the Python reader (and works on ordinary non-shredded MAP
files), but the native shared-shredding reader raises `Selected-key MAP field
'attrs' has no keys` during batch iteration. This is after the native reader
has been returned, so it does not fall back to Python.
Please decode this as the single empty-string key to match the existing
convention, or retain the full-MAP fallback for this key. A shared-shredding
regression test selecting only `attrs['']` would cover the failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
