zeroshade opened a new pull request, #1608: URL: https://github.com/apache/arrow-go/pull/1608
### Rationale for this change Builds on #1601; only the last commit is new here. Repeated columns still read the wrong rows in three cases, all of which are also on main: - When V1 pages aren't aligned to rows, which is the writer's default without a page index, `Skip` stops at the end of a page once it has counted enough rows. The rest of the last row continues on the next page and is never skipped. - Without an offset index, `SeekToRow` counts V1 `num_values` as rows. For a repeated column that is the number of levels, so the seek lands on the wrong page and skips the wrong number of rows. - `ReadBatch` with nil repetition levels doesn't advance the repetition decoder, so later reads and skips on that page get the wrong levels. ### What changes are included in this PR? - `skipRows` keeps skipping after a page ends inside the last row it counted, until the next row starts. - Without an offset index, `SeekToPageWithRow` stops at the first V1 data page of a repeated column, and the column reader counts rows from there. Earlier V2 pages are still skipped by their row counts. Encrypted page readers now record the column's maximum repetition level as well. - `ReadBatch` without repetition levels caches the rest of the page's repetition levels, using the same cache that `skipRows` uses. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. `Skip`, `SeekToRow` and `ReadBatch` without repetition levels now return the right rows for repeated columns with V1 pages. There is no API change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
