sainimal1ba-hue opened a new pull request, #12086: URL: https://github.com/apache/arrow-rs/pull/12086
# Which issue does this PR close? - Closes #12078 # Rationale for this change right now shred_variant expects every field in a target struct to have a specific type, that makes it hard to shred a field on its own while keeping its contents as a variant, without defining its internal structure. for example if an object has id, payload, and debug, we should be able to shred id as an integer and payload as a variant and leave debug in the parents residual value. The current code doesn't recognize the VariantType extension when building object fields, so I updated it to handle both typed and untyped children. # What changes are included in this PR? 1. typed and untyped object fields: the object builder used VariantToShreddedVariantRowBuilder for every child, which assumes each field has a typed_value column. i added an ObjectFieldBuilder enum to handle two cases: - Typed keeps using the existing shredding logic - Untyped stores the original variant value directly, no typed_value column so typed behavior stays the same, and variant fields can now be shredded without specifying their internal schema. 2. how the field builders get created: before, every field's data type went straight into the typed builder. now the code checks for the VariantType extension first, if it's there it makes an untyped builder, otherwise the typed one like before. the object traversal logic stays the same for both, so there's no separate path for untyped fields. 3. missing fields and variant nulls: the untyped builder reuses the existing null handling. a missing field gets a null entry in the value column, and a field that explicitly holds a variant null gets that null serialized as a value. so missing and null stay distinct. 4. building the shredded child arrays: the object builder used to always pass a typed_value column when building each ShreddedVariantFieldArray. i moved that into ObjectFieldBuilder::finish(), which now builds the right thing for typed or untyped children. that drops the assumption that every child needs a typed_value column and avoids duplicating the array construction code. 5. keeping arrow field metadata: ShreddingField only stored the data type and nullability, so rebuilding the schema lost extension metadata like VariantType. it now keeps the whole arrow Field, and when the schema is built it clones the original field and just updates the name, so the metadata survives. the existing data type based inputs work like before. # Are these changes tested? yes, i added two regression tests: - test_untyped_variant_child_shredding checks that untyped children have a value column but no typed_value column. it also covers residual fields, missing values, explicit variant nulls, and round tripping through unshred_variant. - test_schema_builder_preserves_variant_extension checks that ShreddedSchemaBuilder keeps the variant extension for both direct and nested fields, and that the resulting fields use the expected typed or untyped representation. all 373 unit tests and 15 doc tests passed locally, and clippy and formatting checks passed too. # Are there any significant user-facing changes? yes, shred_variant now supports object fields annotated with VariantType, so selected fields can be stored separately without a predefined internal schema. existing typed shredding behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
