ShivamSharma43 opened a new pull request, #1620:
URL: https://github.com/apache/arrow-go/pull/1620
### Rationale for this change
`run_end_decode` gives wrong results or panics on a run-end encoded array
that is sliced so it ends before its last run does:
```go
ree, _, _ := array.FromJSON(mem,
arrow.RunEndEncodedOf(arrow.PrimitiveTypes.Int32, arrow.BinaryTypes.String),
strings.NewReader(`["a", "a", "b", "b", "c"]`))
slice := array.NewSlice(ree, 0, 3) // ["a", "a", "b"]
compute.RunEndDecodeArray(ctx, slice)
// panic: runtime error: index out of range [4] with length 4
ree, _, _ = array.FromJSON(mem,
arrow.RunEndEncodedOf(arrow.PrimitiveTypes.Int32, arrow.PrimitiveTypes.Int64),
strings.NewReader(`[null, null, 1, 1, 2]`))
out, _ := compute.RunEndDecodeArray(ctx, array.NewSlice(ree, 0, 3))
// out is [(null) (null) 1], but out.NullN() is 0 instead of 2
```
The decoders find the first run in the slice with `sort.Search`, but then
loop over every remaining run end in the run ends child. Nothing stops the loop
at `offset + length`, so the last run in the slice is decoded at its full
length and every run after it is decoded too. For string and binary values the
writes go past the end of the output offsets and panic. For boolean values the
bitmap writes can panic the same way. For other fixed width values the extra
values land in the buffer padding when there is room, and the extra runs are
counted as valid, so the output null count is wrong (it can even go negative).
The existing `TestDecodeWithOffset` only slices off the front of the array,
so the slice always ended at the last run end.
### What changes are included in this PR?
`newDecoder` now trims the run ends to the run that holds the last logical
value of the slice, and the four loops in the boolean, fixed width and binary
decoders (including the size computation in `decodeBinary.PreallocOutput`)
clamp each run end to `offset + length`. These are the only loops over run ends
in the kernels package.
### Are these changes tested?
Yes. `TestDecodeSliceEndingMidRun` is added to `RunEndEncodeDecodeSuite`, so
it runs for every value type and run end type in `TestRunEndFunctions`. It
decodes the slices `[0, n-1)` and `[1, n-1)` and checks the values and the null
count against the same slice of the plain array. On `main`, the `uint32 with
nulls` cases fail with the wrong null count (expected 3, got 2; expected 2, got
1), and the boolean and string cases panic with `index out of range`. All cases
pass with this change.
`go test ./arrow/compute/ ./arrow/compute/internal/kernels/` passes, as do
runs with `-tags assert,test -short` and with `-tags assert,test,noasm -short`.
`go vet`, gofmt and golangci-lint report nothing for the changed packages.
### Are there any user-facing changes?
Yes, a bug fix. `RunEndDecode` now works on run-end encoded arrays sliced so
that they end in the middle of a run or before the last run.
AI assistance: I used Claude Code (Anthropic) to find this bug and write the
fix and the test. I reviewed the code and ran the tests locally.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]