hcrosse opened a new issue, #1484:
URL: https://github.com/apache/arrow-go/issues/1484
### Describe the bug, including details regarding any error messages,
version, and platform.
`is_null` reports NaN as not null even when `NullOptions.NanIsNull` is true.
The option is accepted but has no effect on floating point arrays, scalars or
dictionary values.
I reproduced this on main at `66722e30` and on v18.8.0, on macOS arm64 with
Go 1.26.10.
**Reproduction**
Save this as `main.go` in a new subdirectory of an arrow-go checkout at
`66722e30df98ddd6f7bb46023fa13287b84308f6`, then run `go run .` from that
directory:
```go
package main
import (
"context"
"fmt"
"math"
"strings"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/compute"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/arrow-go/v18/arrow/scalar"
)
func main() {
ctx := context.Background()
opts := &compute.NullOptions{NanIsNull: true}
run := func(label string, d compute.Datum) {
out, err := compute.CallFunction(ctx, "is_null", opts, d)
if err != nil {
panic(err)
}
defer out.Release()
if a, ok := out.(*compute.ArrayDatum); ok {
arr := a.MakeArray()
defer arr.Release()
fmt.Println(label, arr)
return
}
fmt.Println(label, out.(*compute.ScalarDatum).Value)
}
arr, _, _ := array.FromJSON(memory.DefaultAllocator,
arrow.PrimitiveTypes.Float64,
strings.NewReader(`[1, "NaN", null, 2]`))
defer arr.Release()
run("float64 [1, NaN, null, 2]:", compute.NewDatumWithoutOwning(arr))
run("float64 scalar NaN:",
compute.NewDatum(scalar.NewFloat64Scalar(math.NaN())))
dict, _ := array.DictArrayFromJSON(memory.DefaultAllocator,
&arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Int8,
ValueType: arrow.PrimitiveTypes.Float64},
`[0, 1, null]`, `[1, "NaN"]`)
defer dict.Release()
run("dictionary indices [0, 1, null], values [1, NaN]:",
compute.NewDatumWithoutOwning(dict))
}
```
It prints:
```text
float64 [1, NaN, null, 2]: [false false true false]
float64 scalar NaN: false
dictionary indices [0, 1, null], values [1, NaN]: [false false true]
```
**Expected behavior**
With `NanIsNull` set, each NaN position should be true:
```text
float64 [1, NaN, null, 2]: [false true true false]
float64 scalar NaN: true
dictionary indices [0, 1, null], values [1, NaN]: [false true true]
```
Arrow C++ [checks for NaN in floating point arrays and dictionary
values](https://github.com/apache/arrow/blob/1ebb9379b2f635755e773b79e88cb99852c70455/cpp/src/arrow/compute/kernels/scalar_validity.cc#L88-L178)
when `nan_is_null` is true.
**Cause**
`is_null` has no kernel initializer, so its options never reach the kernel,
and the kernel only reads the validity bitmap:
https://github.com/apache/arrow-go/blob/66722e30df98ddd6f7bb46023fa13287b84308f6/arrow/compute/internal/kernels/scalar_comparisons.go#L851-L871
https://github.com/apache/arrow-go/blob/66722e30df98ddd6f7bb46023fa13287b84308f6/arrow/compute/internal/kernels/scalar_comparisons.go#L903-L915
I have a patch with tests and can open a PR.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
