hcrosse opened a new pull request, #1465: URL: https://github.com/apache/arrow-go/pull/1465
### Rationale for this change Filters on `is_null` or `is_not_null` keep or drop the wrong rows for null, union and run-end encoded arrays. These types have no validity bitmap, so the kernels treat every slot as valid, as reported in https://github.com/apache/arrow-go/issues/1462. ### What changes are included in this PR? These inputs now use their logical validity, matching Arrow C++'s [`SetLogicalNullBits`](https://github.com/apache/arrow/blob/0ab78e2df489055b71a7d26a666a52823f87f01c/cpp/src/arrow/compute/kernels/scalar_validity.cc#L120-L178). A null array is all null. A union slot takes the validity of the child slot it selects, with nested unions and run-end encoded children resolving their own logical nulls. A run-end encoded slot takes the validity of its run's value. Two C++ quirks are kept. A dictionary child of a union is checked only by its index validity, so a null dictionary value there counts as valid. The values of a top-level run-end encoded array are checked only by their validity bitmap, so `null` or union values there count as valid. Inputs that have a validity bitmap keep the existing path. A dictionary whose values are the `null` type now reports every slot as null. Those values have no validity bitmap, so the dictionary path from https://github.com/apache/arrow-go/pull/1461 treated them as valid. This follows C++'s `IsValidIsNullDictionaryOfNullType` test, added in [`scalar_validity_test.cc`](https://github.com/apache/arrow/blob/00c32a69cb0dbe7a45a8c764638611b7bb021410/cpp/src/arrow/compute/kernels/scalar_validity_test.cc#L205-L210) with the `list_contains` change. It is not in a release yet. pyarrow 21, 23 and 25 report only index nulls here, while 26.0.0 and 26.0.1 segfault on this input. ### Are these changes tested? I added: - `TestNullType` for whole and sliced null arrays. - `TestSparseUnion` for non-identity type codes and sliced children, and `TestDenseUnion` for selected child offsets and slices. - `TestRunEndEncoded` for `int16`, `int32` and `int64` run ends, sliced values and run ends, a boundary slice, and the C++ validity-bitmap quirk. - `TestNestedUnion` for nested unions, including run-end encoded and dictionary children. - `TestLogicalNullScalars` for null, union and run-end encoded scalars. - `TestDictionaryNullTypeValues` for whole and sliced dictionaries with `null` values. Expected values come from pyarrow `26.0.0`, except the dictionary with `null` values, which follows the C++ test. All new cases fail on main except those pinning the deliberate quirks and the run-end encoded scalar cases. Those pass on main and fail when their behavior is deliberately changed. On macOS arm64 with Go 1.26.10, `gofmt`, `go vet`, `go test ./arrow/compute/...` with and without `-tags assert,test`, `go test -race ./arrow/compute/`, `golangci-lint` and the Release Audit Tool passed. ### Are there any user-facing changes? Yes. `is_null` and `is_not_null` results change for null, union and run-end encoded inputs, and dictionaries whose values are the `null` type. There is no API change. Closes https://github.com/apache/arrow-go/issues/1462. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
