sdf-jkl opened a new issue, #11950: URL: https://github.com/apache/arrow-rs/issues/11950
### Is your feature request related to a problem or challenge? Applications such as datafusion-variant want to select a physical layout for Variant columns when writing Parquet—for example, shredding `event.service` as Utf8 and `event.status` as Int64. This is already possible by calling `shred_variant`, constructing a RecordBatch with the resulting storage arrays and Variant fields, and passing it to `ArrowWriter`. Each application must coordinate the transformation, output schema, and metadata itself. Writer-level configuration or a reusable writer adapter would make this easier. ### Describe the solution you'd like Provide a way to associate an explicit shredding schema with a Variant column and write batches containing the original Variant arrays. For example, the caller could supply a logical target DataType built with `ShreddedSchemaBuilder`; the exact API is open for discussion. The integration should reuse the existing shredding kernels, derive the corresponding output schema and Variant metadata, and use the selected layout consistently across batches in a file. Unselected fields and incompatible values should remain in the residual Variant data, preserving logical values and null semantics. Keep the current behavior when no configuration is supplied. Define the policy for already-shredded input, since `shred_variant` currently rejects it; changing layouts may require `unshred_variant` first. Automatic schema selection from sampled data is separate work. ### Describe alternatives you've considered ### Additional context - [datafusion-variant integration request](https://github.com/datafusion-contrib/datafusion-variant/issues/104). - [Existing shredding kernel and schema builder](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet-variant-compute/src/shred_variant.rs). - [ArrowWriter schema initialization](https://github.com/apache/arrow-rs/blob/fd2b5d156d80c986ff9ba5ace2db84500edf50ec/parquet/src/arrow/arrow_writer/mod.rs#L250): the writer currently derives its file schema from the supplied schema before writing batches. - #8922 / #8940 introduced the schema builder; this request concerns applying that schema during writing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
