rustyconover opened a new issue, #54958: URL: https://github.com/apache/arrow/issues/54958
### Describe the enhancement requested In IPC, a dictionary-encoded column is a single `Field` whose type is the dictionary value type, and the only metadata slot is `Field.custom_metadata`. So `dictionary<values=extension<X>>` and `extension<X, storage=dictionary<...>>` serialize identically, and readers cannot tell them apart. For example, pyarrow cannot read back its own IPC stream of `pa.dictionary(pa.int32(), pa.uuid())` (#49704). The dev@ discussion (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding an optional `custom_metadata: [KeyValue]` to `DictionaryEncoding`. It would carry the custom metadata of the dictionary value type, as the IPC counterpart of `ArrowSchema.dictionary->metadata` in the C Data Interface. This issue tracks the format/specification change (format version 1.6). The implementations are tracked separately: - C++: #49704 / #53994 - Go: apache/arrow-go#1381 - Rust (reader): apache/arrow-rs#11622 - nanoarrow: apache/arrow-nanoarrow#972 ### Component(s) Format -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
