CaptainAni187 opened a new pull request, #1367: URL: https://github.com/apache/arrow-go/pull/1367
### Rationale for this change Fixes #1366. Writing a `list<timestamp>` column whose only row is a null list with statistics enabled produced `min = MaxInt64` and `max = MinInt64` with `HasMinMax() == true`, so DuckDB failed to read the file. `UpdateSpaced` computed the non-null count as `len(values) - numNull`. For a nested column the spaced values only cover the leaf slots, but `numNull` also counts the null list, so the count was `0 - 1 = -1`. That skipped the `notnull == 0` early return and set min/max from no values. The same miscount also hid real values: a null list next to `[5]` gave `1 - 1 = 0`, so the `5` never reached min/max. ### What changes are included in this PR? `UpdateSpaced` counts the non-null values with `bitutil.CountSetBits` over `validBits`, the bitmap it already uses for `getMinMaxSpaced`. The change is in `statistics_types.gen.go.tmpl`, and `statistics_types.gen.go` is regenerated with `go generate`. ### Are these changes tested? Yes. `TestWriteArrowNestedNullListStatistics` writes a column with only a null list and one with a null list followed by `[5]`, on both data page versions. It checks the null count, the number of values and that min/max are absent or `5`. All four cases fail on main and pass with the fix. `go test ./parquet/... ./arrow/compute/...` passes with the testing submodules, and golangci-lint reports no issues. ### Are there any user-facing changes? Nested columns with null lists no longer get invalid min/max statistics, and their non-null values are included in min/max. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
