hcrosse opened a new issue, #1452:
URL: https://github.com/apache/arrow-go/issues/1452
### Describe the bug, including details regarding any error messages,
version, and platform.
`is_null` treats a valid dictionary index as non-null even when it points to
a null value. `is_not_null` has the same problem. A filter such as
`is_null(col)` on a dictionary column whose values include null therefore
returns the wrong rows, with no error.
I reproduced this on main at `22c9f26d`. Dictionary values `["a", null]`
with indices `[0, 1, null]` represent `["a", null, null]`, but the null checks
treat the second row as non-null.
**Reproduction**
Save this as `main.go` in a new subdirectory of an arrow-go checkout at
`22c9f26d`, then run `go run .` from that directory:
```go
package main
import (
"context"
"fmt"
"strings"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/compute"
"github.com/apache/arrow-go/v18/arrow/memory"
)
func main() {
mem := memory.DefaultAllocator
dictType := &arrow.DictionaryType{IndexType:
arrow.PrimitiveTypes.Int32, ValueType: arrow.BinaryTypes.String}
indices, _, _ := array.FromJSON(mem, arrow.PrimitiveTypes.Int32,
strings.NewReader(`[0, 1, null]`))
values, _, _ := array.FromJSON(mem, arrow.BinaryTypes.String,
strings.NewReader(`["a", null]`))
arr := array.NewDictionaryArray(dictType, indices, values)
fmt.Println("input: ", arr)
for _, fn := range []string{"is_null", "is_not_null"} {
out, err := compute.CallFunction(context.Background(), fn, nil,
compute.NewDatum(arr))
if err != nil {
panic(err)
}
fmt.Printf("%-12s %v\n", fn+":",
out.(*compute.ArrayDatum).MakeArray())
}
// equal decodes the dictionary, so row 1 is null here.
out, err := compute.CallFunction(context.Background(), "equal", nil,
compute.NewDatum(arr), compute.NewDatum("a"))
if err != nil {
panic(err)
}
fmt.Printf("%-12s %v\n", "equal \"a\":",
out.(*compute.ArrayDatum).MakeArray())
}
```
Output on main:
```text
input: { dictionary: ["a" (null)]
indices: [0 1 (null)] }
is_null: [false false true]
is_not_null: [true true false]
equal "a": [true (null) (null)]
```
**Expected behavior**
`is_null` should return `[false true true]`, and `is_not_null` should return
`[true false false]`.
pyarrow `26.0.1` gives the expected results for the same input. `is_null`
returns `[False, True, True]`, and `is_valid`, the counterpart of
`is_not_null`, returns `[True, False, False]`. `equal` returns `[True, None,
None]`.
**Cause and scope**
`isNullExec` and `isNotNullExec` read only the index validity buffer. Their
`InputAny` registration lets dictionary arrays reach them without decoding:
https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/compute/internal/kernels/scalar_comparisons.go#L718-L761
Arrow C++ handles dictionary values in `SetLogicalNullBits` through
`dict_util::SetLogicalNullBits`:
https://github.com/apache/arrow/blob/0ab78e2df489055b71a7d26a666a52823f87f01c/cpp/src/arrow/compute/kernels/scalar_validity.cc#L135-L140
In-memory arrays and IPC can carry dictionaries with null values.
Dictionaries read from Parquet cannot, because Parquet stores nulls in the
definition levels. I tested in-memory arrays, not IPC reads.
https://github.com/apache/arrow-go/pull/1449 lets Substrait field references
pass dictionary columns through, so expression filters reach these kernels with
dictionary input.
I have a patch with tests and can open a PR.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
