duongcongtoai commented on issue #17445:
URL: https://github.com/apache/datafusion/issues/17445#issuecomment-3406447917
```
let (dictionary, dict_concat) = match &data_type {
DataType::Dictionary(_, _) => {
// If more than one dictionary, concatenate dictionaries
together
let dict_concat = !arrays
.windows(2)
.all(|a|
a[0].child_data()[0].ptr_eq(&a[1].child_data()[0]));
match dict_concat {
false => (Some(arrays[0].child_data()[0].clone()),
false),
true => {
if let Capacities::Dictionary(_, _) = capacities {
panic!("dictionary capacity not yet supported")
}
let dictionaries: Vec<_> =
arrays.iter().map(|array|
&array.child_data()[0]).collect();
let lengths: Vec<_> = dictionaries
.iter()
.map(|dictionary| dictionary.len())
.collect();
let capacity = lengths.iter().sum();
let mut mutable =
MutableArrayData::new(dictionaries, false, capacity);
for (i, len) in lengths.iter().enumerate() {
mutable.extend(i, 0, *len)
}
(Some(mutable.freeze()), true)
}
}
}
_ => (None, false),
};
```
I found this logic relevant, if dict_concat is set to true, the dict belongs
to each children array is considered distinct dictionary and the function
`MutableArrayData::with_capacities` will try to concat them, which may result
into DictionaryKeyOverFlowError
```
let extend_values = match &data_type {
DataType::Dictionary(_, _) => {
let mut next_offset = 0;
let extend_values: Result<Vec<_>, _> = arrays
.iter()
.map(|array| {
let offset = next_offset;
let dict_len = array.child_data()[0].len();
if dict_concat {
next_offset += dict_len;
}
build_extend_dictionary(array, offset, offset +
dict_len)
.ok_or(ArrowError::DictionaryKeyOverflowError)
})
.collect();
extend_values.expect("MutableArrayData::new is infallible")
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
