CurtHagenlocher opened a new pull request, #476:
URL: https://github.com/apache/arrow-dotnet/pull/476
> [!NOTE]
> Draft until the `[VOTE]` on dev@ for format version 1.6 passes. It
implements the format change proposed in apache/arrow#53994, which may still
change before the vote. Companion PRs: apache/arrow-go#1381,
apache/arrow-rs#11622, apache/arrow-nanoarrow#972.
## What's Changed
In IPC, `dictionary<values=extension<X>>` and `extension<X,
storage=dictionary<...>>` serialize identically today, with
`ARROW:extension:name` in `Field.custom_metadata` (apache/arrow#49704). Format
1.6 adds an optional `DictionaryEncoding.custom_metadata` that holds the
dictionary value type's metadata. It's the IPC counterpart of
`ArrowSchema.dictionary->metadata`, which #474 already handles for C Data.
### Format
- `format/Schema.fbs` gets the new `custom_metadata: [ KeyValue ]` member of
`DictionaryEncoding`, as proposed upstream.
- `Flatbuf/DictionaryEncoding.cs` was regenerated with flatc 23.5.9, the
version of the checked-in code. The diff is limited to the new member.
### Writer
- When a dictionary's value type is an `ExtensionType`, its
`ARROW:extension:name` and `ARROW:extension:metadata` are written to
`DictionaryEncoding.custom_metadata` instead of `Field.custom_metadata`.
- If the field metadata names that same extension, those keys are dropped
from the field. Earlier versions of the IPC reader left them there, and on the
field they would mean an extension over the whole dictionary. Other field
metadata is kept.
- An extension type whose storage is a dictionary keeps its keys on the
field. Writing one used to throw in `DictionaryMemo.AddField`, which only
accepted a `DictionaryType` field.
- A dictionary whose value type is an extension type with nested storage now
writes the storage's children.
### Reader
- `DictionaryEncoding.custom_metadata` resolves the value type against the
registry before it's wrapped in the `DictionaryType`. `DictionaryType` has
nowhere to keep value-type metadata, so, as in Arrow C++, other keys and an
unresolved extension name are dropped.
- `Field.custom_metadata` is then resolved against the whole type, so on a
dictionary-encoded field it names an extension whose storage is the dictionary,
as in Arrow C++ and arrow-go.
- **Breaking change:** a field-level extension name on a dictionary isn't
applied to the values any more. Files with a dictionary of an extension type
written by earlier Apache.Arrow versions, and by every other writer before 1.6,
read as a dictionary of the storage type, with the keys left in the field
metadata. As decided in #475, there's no fallback. This needs a release note.
## Tests
- **.NET:**
- the flatbuffer the writer produces, including dropping field metadata
that names the value type;
- round trips with and without a registry;
- a dictionary of an extension type nested in a list;
- field-level metadata on a dictionary not being applied to the values;
- an extension type whose storage is a dictionary.
- **Full suite:** passes on net8.0, net462 and net472, pyarrow tests
included.
- **Cross-implementation, run manually:**
- `flatc --json` decodes a schema written by .NET with the new member as
expected.
- .NET reads a schema message that `flatc` built from JSON.
- pyarrow 24, which predates 1.6, reads .NET output as
`dictionary<values=fixed_size_binary[16]>`.
There's no Go toolchain here, so I couldn't test against
apache/arrow-go#1381.
## Not included
- **Integration test app:** it doesn't support the archery
`dictionary_extension` case yet, so archery has to keep skipping .NET for it.
The app has no extension types; it passes field-level extension metadata
through unchanged. A dictionary's value metadata has nowhere to live in the
.NET object model, so the app needs a pass-through extension type before it can
produce the case.
Closes #475.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
