hcrosse opened a new issue, #1462:
URL: https://github.com/apache/arrow-go/issues/1462
### Describe the bug, including details regarding any error messages,
version, and platform.
`is_null` returns all false for a `null` array, even though every element is
null. A filter on `is_null` or `is_not_null` then keeps or drops the wrong
rows, with no error.
I reproduced this on v18.8.0 and on main at `c620eaa3`, on macOS with Go
1.27.3. Sparse union, dense union and run-end encoded arrays also return all
false when their children contain null values. `is_not_null` returns all true
for these inputs. Slices are affected the same way.
**Reproduction**
Save this as `main.go` in a module that requires
`github.com/apache/arrow-go/v18 v18.8.0`, or in a new subdirectory of an
arrow-go checkout at `c620eaa363c1b68ed93a810da7a8d46bd436f122`, then run `go
run .` from that directory:
```go
package main
import (
"context"
"fmt"
"strings"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/compute"
"github.com/apache/arrow-go/v18/arrow/memory"
)
func main() {
i8, i32 := arrow.PrimitiveTypes.Int8, arrow.PrimitiveTypes.Int32
js := func(dt arrow.DataType, s string) arrow.Array {
a, _, _ := array.FromJSON(memory.DefaultAllocator, dt,
strings.NewReader(s))
return a
}
vals := js(i32, `[1, null]`)
sparse, _ := array.NewSparseUnionFromArrays(js(i8, `[0, 0]`),
[]arrow.Array{vals})
dense, _ := array.NewDenseUnionFromArrays(js(i8, `[0, 0]`), js(i32,
`[0, 1]`), []arrow.Array{vals})
ree := array.NewRunEndEncodedArray(js(i32, `[2, 3]`), vals, 3, 0)
for _, arr := range []arrow.Array{array.NewNull(3), sparse, dense, ree}
{
got, _ := compute.CallFunction(context.Background(), "is_null",
nil, compute.NewDatum(arr))
fmt.Printf("%-40s is_null = %v\n", arr.DataType(),
got.(*compute.ArrayDatum).MakeArray())
}
}
```
Actual output:
```text
null is_null = [false false false]
sparse_union<0: type=int32, nullable=0> is_null = [false false]
dense_union<0: type=int32, nullable=0> is_null = [false false]
run_end_encoded<run_ends: int32, values: int32> is_null = [false false false]
```
Expected output:
```text
null is_null = [true true true]
sparse_union<0: type=int32, nullable=0> is_null = [false true]
dense_union<0: type=int32, nullable=0> is_null = [false true]
run_end_encoded<run_ends: int32, values: int32> is_null = [false false true]
```
pyarrow `26.0.0` returns these expected values. Its `is_valid`, the
counterpart of `is_not_null`, returns their inverse.
**Cause**
`isNullExec` and `isNotNullExec` read only buffer 0 and treat a missing
validity buffer as all valid. They are registered for any input type:
https://github.com/apache/arrow-go/blob/c620eaa363c1b68ed93a810da7a8d46bd436f122/arrow/compute/internal/kernels/scalar_comparisons.go#L718-L766
None of these four types has a validity bitmap in buffer 0. The `null` type
records its nulls in the null count. Union nulls come from the selected child,
and run-end encoded nulls come from the values child.
Arrow C++'s
[`SetLogicalNullBits`](https://github.com/apache/arrow/blob/0ab78e2df489055b71a7d26a666a52823f87f01c/cpp/src/arrow/compute/kernels/scalar_validity.cc#L120-L134)
handles `Type::NA` as all nulls and has separate branches for sparse union,
dense union and run-end encoded inputs.
https://github.com/apache/arrow-go/issues/1452 covers the separate
dictionary case.
I have a patch with tests and can open a PR.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
