Hi Dewey, It came up here:
https://github.com/duckdb/duckdb-nanoarrow/pull/129 I was working on adding dictionary write support to DuckDB's nanoarrow extension. The motivating use case was exporting a UUID type. I couldn't actually write the Arrow extension type metadata for the UUID DuckDB column. So it would be a dictionary of FixedSizeBinary(16) with the UUID extension type: dictionary<values=arrow.uuid, indices=int32>. Thanks for looking at this code! Rusty -- Rusty Conover Query.Farm Founder https://query.farm On Fri, Oct 16, 2026, at 11:33 PM, Dewey Dunnington wrote: > Thank you Rusty for taking this on! The fact that IPC can't represent > this is a bit of a hole in Arrow's type model...in addition to > theoretical completeness, I believe Rusty found that this came up > exporting DuckDB types via Arrow where this can come up. In the > spatial world, there are certain types of geometry columns that > contain a lot of repeated points and we've wondered if dictionary > encoding (e.g., a dictionary-encoded geoarrow.point extension type) > would be a way to handle that. > > I will work through reviewing these next week! > > Cheers, > > -dewey > > On Fri, Oct 16, 2026 at 12:33 PM Rusty Conover via dev > <[email protected]> wrote: >> >> Hi all, >> >> Following up on the consensus for option 1 (an optional `custom_metadata` on >> `DictionaryEncoding` for the dictionary value type), there are now draft PRs: >> >> - Format + C++: https://github.com/apache/arrow/pull/53994 >> - Go: https://github.com/apache/arrow-go/pull/1381 >> - Rust: https://github.com/apache/arrow-rs/pull/11622 >> - nanoarrow: https://github.com/apache/arrow-nanoarrow/pull/972 >> >> The new member has the same semantics as `Field.custom_metadata`, including >> the reserved `ARROW:extension:*` keys, but applies to the dictionary values >> rather than to the field. It is the IPC counterpart of >> `ArrowSchema.dictionary->metadata` in the C Data Interface. It is appended >> to `DictionaryEncoding` as format version 1.6, and `MetadataVersion` stays >> V5. Readers that predate it see `dictionary<storage type>`. >> >> With the drafts, C++ and Go produce and consume a new archery case, >> `dictionary_extension`, over IPC and C Data, and arrow-rs and nanoarrow read >> it. One implementation limitation: the C++ and Go dictionary types can only >> attach a registered extension type to the values, so they drop other >> value-type metadata on read. nanoarrow keeps all of it. arrow-rs maps it >> into the field's metadata until it can represent metadata on dictionary >> values (apache/arrow-rs#7982). >> >> Readers get no fallback for files written by existing writers, where the >> extension keys sit on the Field, so those read the same as today. >> >> If there are no objections, I'd like to start a vote on the format change in >> a week or so. >> >> Have a great weekend! >> >> Rusty
