tustvold opened a new issue, #2687:
URL: https://github.com/apache/arrow-rs/issues/2687
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
`DynComparator` does not inspect null masks. This is by design as it isn't
provided a `SortOptions` specifying how to order them.
Unfortunately the values array of a dictionary is permitted to contain
nulls, which causes the comparator to behave
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
The builders will never put a null in a dictionary, but if you manually
construct the DictionaryArray from ArrayData it is possible.
```
let values = Int32Array::from_iter([None, Some(1)]).into_data();
let keys =
Int32Array::from_iter([Some(0), None])
.into_data();
let data = keys
.into_builder()
.data_type(DataType::Dictionary(
Box::new(DataType::Int32),
Box::new(DataType::Int32),
))
.child_data(vec![values])
.build()
.unwrap();
let dictionary = DictionaryArray::<Int32Type>::from(data);
```
If you then construct a comparator on this dictionary it will say that the
two rows are not equal, when they are technically both null. If you remove the
`Some(1)` it might even panic.
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
This should be handled correctly
**Additional context**
<!--
Add any other context about the problem here.
-->
Longer term I intend for these to be deprecated and replaced with the row
format #2677, and so perhaps we just defer on fixing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
