rustyconover opened a new issue, #1380: URL: https://github.com/apache/arrow-go/issues/1380
### Describe the enhancement requested Support the proposed `DictionaryEncoding.custom_metadata` member of the IPC format (apache/arrow#49704, Arrow format version 1.6), which holds the metadata of a dictionary's *value* type. In IPC a dictionary-encoded field is a single `Field` whose type is the value type. Its only metadata slot was `Field.custom_metadata`, so `dictionary<values=extension<X>>` and `extension<X, storage=dictionary<...>>` serialize identically. In arrow-go today: - a dictionary of `arrow.uuid` values is written with the UUID keys on the `Field`, and reading it back fails with `invalid storage type for UUIDType: dictionary`; - writing the schema of an extension type whose storage is a dictionary panics (`interface conversion: arrow.DataType is *extensions.OpaqueType, not *arrow.DictionaryType`). The dev@ discussion (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding `custom_metadata: [KeyValue]` to `DictionaryEncoding`. The format change is a draft in apache/arrow pending a vote, and needs a second reference implementation alongside C++. arrow-go would: - write a dictionary value type's extension keys to `DictionaryEncoding.custom_metadata`; - apply them to the value type when reading; - support the matching `"dictionary"."metadata"` key in the integration JSON format; - register the canonical extension types in the integration test programs. ### Component(s) Integration, Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
