hcrosse commented on code in PR #1601:
URL: https://github.com/apache/arrow-go/pull/1601#discussion_r4607774977
##########
parquet/file/column_reader.go:
##########
@@ -707,19 +708,23 @@ func (c *columnChunkReader) skipRows(nrows int64) (int64,
error) {
} else {
// with repetition levels, we have to check them to
determine
// how many rows to skip. we can't just skip the number
of values
- // because there could be multiple values per row. So
we read in
- // the repetition levels for the entire page at once
and then go
- // through them to find the right row.
- repLvls := c.getRepLvlBuffer(c.numBuffered)
- nreps, _, err := c.repetitionDecoder.Decode(repLvls)
- if err != nil {
- c.err = err
- return nrows - toSkip, err
+ // because there could be multiple values per row. So
we decode the
+ // rest of the page's repetition levels once, at their
page
+ // positions, and then go through them to find the
right row.
+ if len(c.repLvlBuffer) == 0 {
+ buf := c.getRepLvlBuffer(c.numBuffered)
+ nreps, _, err :=
c.repetitionDecoder.Decode(buf[c.numDecoded:])
Review Comment:
This also happens on main, and without `Skip`. `determineNumToRead` only
reads repetition levels when `repLvls` is not nil, so after `ReadBatch` with
nil `repLvls` the next `ReadBatch` with `repLvls` gets the page's first levels
again. On main, reading one row of the test data with nil `repLvls` and then
the rest with `repLvls` returns `[0 0 1 0 ...]` instead of `[0 1 0 1 ...]`.
That is a separate bug in `ReadBatch`, so I left it out of this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]