Jefffrey commented on code in PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#discussion_r3813099517
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or
`DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent.
`is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical
nulls can still
+ // be empty.
Review Comment:
i dont understand how it can be conservative
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or
`DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent.
`is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical
nulls can still
+ // be empty.
+ if !a.is_nullable() && !b.is_nullable() {
try_binary_no_nulls(len, a, b, op)
} else {
- let nulls =
- NullBuffer::union(a.logical_nulls().as_ref(),
b.logical_nulls().as_ref()).unwrap();
+ let Some(nulls) = NullBuffer::union(a.logical_nulls().as_ref(),
b.logical_nulls().as_ref())
Review Comment:
theres an unfortunate side effect in that we calculate the logical_nulls
buffer twice; in terms of correctness this fix is good for ensuring we treat
nulls properly, but there is a cost to calculating this logical nulls buffer
for dictionaries specifically 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
