hcrosse opened a new issue, #1454:
URL: https://github.com/apache/arrow-go/issues/1454
### Describe the bug, including details regarding any error messages,
version, and platform.
`compute.CallFunction` panics on `is_null` and `is_not_null` with decimal
input, and on `is_nan` with decimal or float dictionary input. In iceberg-go, a
decimal null filter can crash the process instead of returning an error.
I reproduced the calls below on v18.8.0 and main at
`22c9f26df0abbc2066f5d9ae45f31720d29a4300`, on Linux amd64 with Go 1.26.3.
`is_null` and `is_not_null` also panic on `decimal256` arrays.
**Reproduction**
Save this as `main.go` in a module that requires
`github.com/apache/arrow-go/v18 v18.8.0`, or in a new subdirectory of a
checkout at `22c9f26d`, then run `go run .`. The program recovers each panic so
all four calls run:
```go
package main
import (
"context"
"fmt"
"strings"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/compute"
"github.com/apache/arrow-go/v18/arrow/memory"
)
func call(fn string, arr arrow.Array) {
defer func() {
if r := recover(); r != nil {
fmt.Printf("%s(%s): panic: %v\n", fn, arr.DataType(), r)
}
}()
out, err := compute.CallFunction(context.Background(), fn, nil,
compute.NewDatum(arr))
if err != nil {
fmt.Printf("%s(%s): error: %v\n", fn, arr.DataType(), err)
return
}
defer out.Release()
fmt.Printf("%s(%s): %v\n", fn, arr.DataType(),
out.(*compute.ArrayDatum).MakeArray())
}
func main() {
mem := memory.DefaultAllocator
dec, _, _ := array.FromJSON(mem, &arrow.Decimal128Type{Precision: 10,
Scale: 2},
strings.NewReader(`["1.50", null, "2.25"]`))
dict, _ := array.DictArrayFromJSON(mem,
&arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Int8,
ValueType: arrow.PrimitiveTypes.Float64}, `[0, 1, null]`,
`[1.5, 2.5]`)
call("is_null", dec)
call("is_not_null", dec)
call("is_nan", dec)
call("is_nan", dict)
}
```
Output on both versions:
```text
is_null(decimal(10, 2)): panic: runtime error: index out of range [1] with
length 1
is_not_null(decimal(10, 2)): panic: runtime error: index out of range [1]
with length 1
is_nan(decimal(10, 2)): panic: runtime error: index out of range [1] with
length 1
is_nan(dictionary<values=float64, indices=int8, ordered=false>): panic:
runtime error: index out of range [1] with length 1
```
**Expected behavior**
None of these calls should panic. For the decimal input above, `is_null`
should return `[false true false]` and `is_not_null` should return `[true false
true]`. `is_nan` already has a decimal kernel that returns false for every
value. It has no kernel for dictionary input, so `is_nan` on the float
dictionary should either decode it or return an error.
**Cause**
All three functions are registered as `compareFunction`:
https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/compute/scalar_compare.go#L165-L185
Its `DispatchBest` runs helpers that assume two arguments even though these
calls have one:
https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/compute/scalar_compare.go#L38-L58
- Decimal input reaches `castBinaryDecimalArgs`, which reads `vals[1]`:
https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/compute/utils.go#L260-L261.
- The float dictionary reaches `replaceNullWithOtherType`, which also reads
`vals[1]`:
https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/compute/utils.go#L115-L127.
The null-check kernels accept any input type, and `is_nan` has float and
decimal kernels. The decimal calls panic before exact dispatch can match those
kernels. The float dictionary has no exact kernel, so it reaches the fallback
that would decode it, and panics there:
https://github.com/apache/arrow-go/blob/22c9f26df0abbc2066f5d9ae45f31720d29a4300/arrow/compute/internal/kernels/scalar_comparisons.go#L747-L812
https://github.com/apache/arrow-go/issues/1452 is a separate bug in the same
null checks: they ignore null values inside a dictionary.
**Impact on iceberg-go**
I also reproduced process crashes on iceberg-go v0.7.0 and main `8ef0a092`,
both using arrow-go v18.8.0, with decimal `IsNull` and `NotNull` scan filters,
alone or inside `AND`. The scan calls `ExecuteScalarExpression` in a worker
goroutine with no `recover`, so the panic ends the process and the caller never
receives an error:
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/arrow_scanner.go#L1651-L1670
On iceberg-go main, decimal `IN` and `NOT IN` filters crash too, because
their translation includes `is_null`:
https://github.com/apache/iceberg-go/blob/8ef0a092fcbaf70f99aa8935481bc95071eb41b1/table/substrait/substrait.go#L413-L427
I have a patch with tests and can open a PR.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
