sundapeng commented on code in PR #1069:
URL: https://github.com/apache/paimon-rust/pull/1069#discussion_r4216076402
##########
crates/paimon/src/arrow/shredding/variant.rs:
##########
@@ -1189,12 +1342,30 @@ fn has_dictionary_variant_metadata(fields: &Fields) ->
bool {
if key.as_ref() == &ArrowDataType::Int32 && value.as_ref() ==
&ArrowDataType::Binary)
}
+fn is_dictionary_metadata(field: &ArrowField) -> bool {
+ field.name() == VARIANT_METADATA_FIELD_NAME
+ && matches!(field.data_type(), ArrowDataType::Dictionary(key, value)
+ if key.as_ref() == &ArrowDataType::Int32 && value.as_ref() ==
&ArrowDataType::Binary)
+}
+
+/// Shredded Variant whose metadata child the Parquet reader kept
dictionary-encoded.
+fn is_dictionary_shredded_variant_array(array: &dyn Array) -> bool {
+ let ArrowDataType::Struct(fields) = array.data_type() else {
+ return false;
+ };
+ is_shredded_variant_array(array) && fields.iter().any(|field|
is_dictionary_metadata(field))
Review Comment:
Thanks, confirmed and fixed in 3637ae8. A residual-only extraction clips the
shredded root to `metadata` and `value`, but the reader had already switched
`metadata` to a dictionary because the file schema has `typed_value`; none of
the storage predicates recognized the clipped struct, so it skipped assembly.
`is_dictionary_variant_storage_array` now recognizes Variant storage with
dictionary metadata by child names (metadata plus `value` and/or `typed_value`,
in any order), so the column reaches `assemble_variant_extraction_array`, whose
row-wise fallback materializes the metadata.
I added your scenario as a SQL regression,
`residual_only_extraction_after_add_column_reads_values`: a data-evolution
table that shreds only `age`, then ADD COLUMN, then `try_variant_get(payload,
'$.rest', 'string')`. Before the fix it returned the metadata bytes for both
rows. There is also a unit test for the clipped dictionary struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
