JingsongLi commented on code in PR #1069:
URL: https://github.com/apache/paimon-rust/pull/1069#discussion_r4215545688
##########
crates/paimon/src/arrow/shredding/variant.rs:
##########
@@ -1189,12 +1342,30 @@ fn has_dictionary_variant_metadata(fields: &Fields) ->
bool {
if key.as_ref() == &ArrowDataType::Int32 && value.as_ref() ==
&ArrowDataType::Binary)
}
+fn is_dictionary_metadata(field: &ArrowField) -> bool {
+ field.name() == VARIANT_METADATA_FIELD_NAME
+ && matches!(field.data_type(), ArrowDataType::Dictionary(key, value)
+ if key.as_ref() == &ArrowDataType::Int32 && value.as_ref() ==
&ArrowDataType::Binary)
+}
+
+/// Shredded Variant whose metadata child the Parquet reader kept
dictionary-encoded.
+fn is_dictionary_shredded_variant_array(array: &dyn Array) -> bool {
+ let ArrowDataType::Struct(fields) = array.data_type() else {
+ return false;
+ };
+ is_shredded_variant_array(array) && fields.iter().any(|field|
is_dictionary_metadata(field))
Review Comment:
[P1] Recognize dictionary Variant storage after typed_value is pruned
A residual-only extraction clips the physical shredded root to [metadata,
value], removing typed_value. This predicate then returns false; storage
recognition also rejects Dictionary metadata, and plain recognition expects
[value, metadata] in that order. The reader skips assembly. For a
schema-evolved file, reconciliation treats the source as the synthetic
extraction ROW and casts metadata into the requested result.
I reproduced through a real filesystem SQL table with
data-evolution.enabled/row-tracking.enabled and only age shredded as INT:
insert {"age":27,"rest":"A"} and {"age":32,"rest":"B"}; ALTER TABLE ... ADD
COLUMN extra INT; SELECT id, try_variant_get(payload, '$.rest', 'string') FROM
... ORDER BY id. Instead of A/B, both rows return the metadata string
"\u0001\u0002\u0000\u0003\u0007agerest". Including payload in the projection
(complete decoding) returns A/B. The exact baseline implementation passes the
same probe; the clean current-main integration retains the wrong values. Please
recognize dictionary metadata/value storage by child names even without
typed_value, materialize it for the fallback, and add this
schema-evolution/residual-only SQL regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
