zeroshade commented on code in PR #1626:
URL: https://github.com/apache/arrow-go/pull/1626#discussion_r4635286131
##########
parquet/file/page_reader.go:
##########
@@ -442,36 +443,61 @@ type serializedPageReader struct {
decompressBuffer *memory.Buffer
dataPageBuffer *memory.Buffer
dictPageBuffer *memory.Buffer
+ bufferPool chan pageReaderBuffers
err error
}
+type pageReaderBuffers struct {
+ decompress, data *memory.Buffer
+}
+
func (p *serializedPageReader) Close() error {
if p.curPage != nil {
// free a direct PageReader user's streaming value stream
p.curPage.Release()
p.curPage = nil
}
if p.decompressBuffer != nil {
- p.decompressBuffer.Release()
+ const maxCachedBufferSize = 1 << 20
Review Comment:
This limit is exactly the default data page size. With default writer
settings, an int64 column's buffers come to exactly 1,048,576 bytes, so they
are cached (the check is `<=`). Pages that go slightly over, which is common
for variable-width columns and for files from other writers, won't be cached at
all. A slightly higher limit (2 MiB, say) would cover them at the cost of a
larger per-file cap, or the trade-off could be noted here.
##########
parquet/file/file_reader.go:
##########
@@ -138,6 +138,9 @@ func NewParquetReader(r parquet.ReaderAtSeeker, opts
...ReadOption) (*Reader, er
return nil, err
}
}
+ if _, ok := f.props.Allocator().(*memory.GoAllocator); ok &&
f.fileDecryptor == nil && f.NumRowGroups() > 1 {
Review Comment:
Could you add a comment explaining why this is limited to the Go allocator?
Nothing empties the cache when the file is closed, and page readers can close
after the file does, so cached buffers are never `Release`d. That's only safe
when the garbage collector reclaims them. Enabling this for another allocator
would leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]