tschaub opened a new issue, #37402:
URL: https://github.com/apache/arrow/issues/37402
### Describe the usage question you have. Please include as many useful
details as possible.
I've seen a suggestion on Stack Overflow by @zeroshade to use
`pqarrow.RecordReader` to simplify reading records with repeated or nested
values. I'm looking at the tests for examples, but struggling with some basic
usage.
It looks like I can read all records if I use a batch size of 1. For
example, this test passes:
```go
func TestRecordReading(t *testing.T) {
f, fileErr := os.Open("../testdata/cases/example-v1.0.0-beta.1.parquet")
require.NoError(t, fileErr)
reader, readerErr := file.NewParquetReader(f)
require.NoError(t, readerErr)
defer reader.Close()
pqReader, pqErr := pqarrow.NewFileReader(reader,
pqarrow.ArrowReadProperties{BatchSize: 1}, memory.DefaultAllocator)
require.NoError(t, pqErr)
recordReader, rrErr := pqReader.GetRecordReader(context.Background(),
nil, nil)
require.NoError(t, rrErr)
numRows := 0
for {
rec, err := recordReader.Read()
if err == io.EOF {
assert.Nil(t, rec)
break
}
assert.NoError(t, err)
numRows += 1
}
assert.Equal(t, reader.NumRows(), int64(numRows))
}
```
However, if I instead use `pqarrow.ArrowReadProperties{BatchSize: 1}` or any
other batch size, I get `io.EOF` before all records are read.
Is this the right pattern for reading records from a Parquet file? If does
the total number of records in the file need to be an integer multiple of the
batch size used in order to read all records?
### Component(s)
Go, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
