zhuqi-lucas opened a new issue, #10427: URL: https://github.com/apache/arrow-rs/issues/10427
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** `RowConverter` accepts types on encode whose **decode output type differs from the encode input type**: dictionaries (and dictionary-containing nested types) are flattened to their value type on decode via the internal `corrected_type` step. For example: - `Dictionary(Int32, Utf8)` decodes as `Utf8` - `ListView(Dictionary(Int32, Utf8))` decodes as `ListView(Utf8)` - `List(Dictionary(Int32, Utf8))` decodes as `List(Utf8)` This mapping is arrow-row internal knowledge with **no public API to query it**. Downstreams that need to round-trip arrays through the row format (e.g. DataFusion's group-by, which stores group keys as rows and re-emits them as arrays) must mirror decoder internals by hand to know which output types to expect and reconstruct. This has produced two real bugs of the same class: 1. #10413 — `decode_fixed_size_list` diverged from the other list-like decoders by *not* applying the correction, panicking on `FixedSizeList<Dictionary>` (fixed by #10414). 2. [apache/datafusion#23523 review](https://github.com/apache/datafusion/pull/23523#discussion_r3643717742) — DataFusion accepted `ListView<Dictionary>` for its rows-backed group storage but missed the reconstruction arm for view-lists, silently emitting `ListView(Utf8)` where the schema declared `ListView(Dictionary(Int32, Utf8))`. Both stem from the same gap: each downstream has to re-derive, per container type, "what type will decode hand me back?" — and every missed container is a bug. **Describe the solution you'd like** Expose the mapping as a public API, e.g.: ```rust /// Returns the data type produced by `RowConverter::convert_rows` for /// an input column of `data_type` — identical to the input except that /// dictionary-encoded values (at any nesting level) are replaced by /// their value type. pub fn decoded_type(data_type: &DataType) -> DataType ``` (placement flexible: free function in `arrow_row`, method on `SortField`, or `RowConverter::decoded_types(&self) -> Vec<DataType>`). Together with a **property test in arrow-row** asserting the contract for every supported type shape: ```rust for dt in representative_supported_shapes() { let array = make_test_array(&dt); let rows = converter.convert_columns(&[array])?; let decoded = converter.convert_rows(&rows)?; assert_eq!(decoded[0].data_type(), &decoded_type(&dt)); } ``` The invariant test is arguably the bigger win: it would have caught #10413 automatically (panic = contract violation), and prevents any future decoder from silently diverging from the others the way `decode_fixed_size_list` did. With this API, downstream code reduces to a mechanical rule: `decoded_type(t) == t` → array round-trips as-is; otherwise recurse over the structure and re-encode exactly where the two types differ. **Describe alternatives you've considered** - **A boolean `round_trips_losslessly(&DataType) -> bool`** — simpler, but only says *whether* the type changes, not *what comes back*, so downstreams still can't derive reconstruction mechanically. - **Making decode preserve dictionary types** — would eliminate the mapping entirely, but re-encoding dictionaries on decode is expensive, changes long-standing semantics, and the flattened output is often what callers want. - **Status quo** — downstreams mirror decoder internals; two bugs in two months suggests this doesn't scale. **Additional context** - #10413 / #10414 (the `decode_fixed_size_list` divergence) - [apache/datafusion#23523](https://github.com/apache/datafusion/pull/23523) — the DataFusion PR where the second instance surfaced, with an end-to-end reproducer in the review thread Happy to implement this (API + property test) if the direction sounds good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
