zhuqi-lucas opened a new issue, #10413:
URL: https://github.com/apache/arrow-rs/issues/10413
## Describe the bug
`RowConverter::convert_rows` panics when the target schema contains a
`FixedSizeList` whose element type is (or transitively contains) a
`Dictionary`. The row-format encoder happily accepts these types
(`supports_fields` returns `true`), but the reverse trip through `convert_rows`
fails with a schema-mismatch:
```text
InvalidArgumentError("FixedSizeListArray expected data type
Dictionary(Int32, Utf8) got Utf8 for \"item\"")
```
The other list-like decoders — `List`, `LargeList`, `ListView`,
`LargeListView`, and `Map` — round-trip cleanly on the same input. Only
`decode_fixed_size_list` is affected.
## To Reproduce
Encode a one-row `FixedSizeList<Dictionary<Int32, Utf8>, 2>` and convert it
back:
```rust
use std::sync::Arc;
use arrow_array::{
ArrayRef, DictionaryArray, FixedSizeListArray, StringArray,
types::Int32Type,
};
use arrow_row::{RowConverter, SortField};
use arrow_schema::{DataType, Field};
fn main() {
let dict_dt = DataType::Dictionary(Box::new(DataType::Int32),
Box::new(DataType::Utf8));
let element_field = Arc::new(Field::new("item", dict_dt.clone(), true));
let fsl_dt = DataType::FixedSizeList(Arc::clone(&element_field), 2);
// supports_fields returns `true` — this schema is considered supported
assert!(RowConverter::supports_fields(&[SortField::new(fsl_dt.clone())]));
// Build one row = ["a", "b"] as FixedSizeList<Dict<Int32, Utf8>, 2>
let values = Arc::new(StringArray::from(vec!["a", "b"]));
let keys = arrow_array::Int32Array::from(vec![0, 1]);
let dict = DictionaryArray::<Int32Type>::try_new(keys, values).unwrap();
let fsl = FixedSizeListArray::new(Arc::clone(&element_field), 2,
Arc::new(dict), None);
let converter =
RowConverter::new(vec![SortField::new(fsl_dt.clone())]).unwrap();
let rows = converter.convert_columns(&[Arc::new(fsl) as
ArrayRef]).unwrap();
// 💥 boom
let _ = converter.convert_rows(&rows).unwrap();
}
```
Panics on the last line with:
```text
InvalidArgumentError("FixedSizeListArray expected data type
Dictionary(Int32, Utf8) got Utf8 for \"item\"")
```
Same panic for any type containing an `FSL<Dict>` anywhere in its subtree —
`FSL<Struct<Dict>>`, `FSL<List<Dict>>`, `List<FSL<Dict>>`, `Struct<FSL<Dict>>`,
etc.
## Expected behavior
Either
- (a) `convert_rows` round-trips the dictionary correctly, producing a
`FixedSizeListArray` whose element is `Dictionary<Int32, Utf8>` (matching the
declared schema), or
- (b) `convert_rows` returns an error rather than panicking, and
`RowConverter::supports_fields` returns `false` for these shapes so callers can
fall back.
Ideally (a) — the same behaviour the other list-like decoders already
exhibit.
## Root cause
`arrow-row/src/list.rs::decode_fixed_size_list` at [line 424-430 on
main](https://github.com/apache/arrow-rs/blob/main/arrow-row/src/list.rs#L424-L430)
builds the returned array with the **declared** `element_field` (which still
carries `Dictionary<...>`) but the child data comes from
`converter.convert_raw`, which returns the **flattened** (native) type:
```rust
let mut children = unsafe { converter.convert_raw(&mut child_rows,
validate_utf8) }?;
assert_eq!(children.len(), 1);
FixedSizeListArray::try_new_with_length(
Arc::clone(element_field), // declared: Dict<Int32, Utf8>
*size,
children.pop().unwrap(), // actual: Utf8
nulls,
num_rows,
)
```
`try_new_with_length` validates that `element_field.data_type() ==
child.data_type()` and returns `InvalidArgumentError`. Callers that `.expect()`
the result (as any consumer of `RowConverter::convert_rows` naturally does) get
a panic.
For contrast, the generic `list::decode<L: GenericListArrayOrMap>` path
(which handles `List` / `LargeList` / `ListView` / `LargeListView` / `Map`)
already applies a **`corrected_type`** step to reconcile declared-vs-actual —
see [list.rs:259-320 on
main](https://github.com/apache/arrow-rs/blob/main/arrow-row/src/list.rs#L259-L320).
The `Codec::Struct` decode path in `lib.rs` does the same via
`corrected_fields`.
`decode_fixed_size_list` is the only list-like decoder that skipped this
pattern.
## Proposed fix
Adopt the same `corrected_type` shape used elsewhere in the file:
```rust
let mut children = unsafe { converter.convert_raw(&mut child_rows,
validate_utf8) }?;
assert_eq!(children.len(), 1);
// Since RowConverter flattens certain data types (i.e. Dictionary),
// we need to use the child's actual data type instead of the
// declared element_field's — mirrors decode<L> at list.rs:259-320.
let corrected_element_field = Arc::new(
element_field
.as_ref()
.clone()
.with_data_type(children[0].data_type().clone()),
);
FixedSizeListArray::try_new_with_length(
corrected_element_field,
*size,
children.pop().unwrap(),
nulls,
num_rows,
)
```
Downstream callers that care about the declared type (e.g. DataFusion's
`RowsGroupColumn::rows_to_array`) already have re-encoding helpers
(`encode_array_if_necessary`) that map the flattened output back onto the
declared dictionary type. What they need from arrow-row is a non-panicking,
self-consistent decoded array — this fix delivers that.
## Impact / who this affects
DataFusion hit this in
[#23523](https://github.com/apache/datafusion/pull/23523) (nested-type support
in `GroupValuesColumn`) where a `GROUP BY fsl_col` with `fsl_col:
FixedSizeList<Dictionary<Int32, Utf8>>` would panic on emit. Kosiew's review
caught it during a repro; the DataFusion PR is landing a defensive blacklist in
the meantime, but the right long-term fix is here in arrow-row.
Similar shapes exist in real workloads: sensor-vector columns
(`FixedSizeList<Dict<...>, N>`), one-hot / embedding batches, and any
Iceberg/DeltaLake table with fixed-arity vector columns backed by
dictionary-encoded strings.
## Version info
- Reproduces on `arrow-row = 59.1.0` and on current `apache/arrow-rs:main`
(verified against `arrow-row/src/list.rs` on main).
## Additional context
Happy to submit a PR with the fix + a round-trip regression test if the
direction above looks right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
