zeroshade commented on code in PR #1626:
URL: https://github.com/apache/arrow-go/pull/1626#discussion_r4635286131


##########
parquet/file/page_reader.go:
##########
@@ -442,36 +443,61 @@ type serializedPageReader struct {
        decompressBuffer *memory.Buffer
        dataPageBuffer   *memory.Buffer
        dictPageBuffer   *memory.Buffer
+       bufferPool       chan pageReaderBuffers
        err              error
 }
 
+type pageReaderBuffers struct {
+       decompress, data *memory.Buffer
+}
+
 func (p *serializedPageReader) Close() error {
        if p.curPage != nil {
                // free a direct PageReader user's streaming value stream
                p.curPage.Release()
                p.curPage = nil
        }
        if p.decompressBuffer != nil {
-               p.decompressBuffer.Release()
+               const maxCachedBufferSize = 1 << 20

Review Comment:
   This limit is exactly the default data page size. With default writer 
settings, an int64 column's buffers come to exactly 1,048,576 bytes, so they 
are cached (the check is `<=`). Pages that go slightly over, which is common 
for variable-width columns and for files from other writers, won't be cached at 
all. A slightly higher limit (2 MiB, say) would cover them at the cost of a 
larger per-file cap, or the trade-off could be noted here.



##########
parquet/file/file_reader.go:
##########
@@ -138,6 +138,9 @@ func NewParquetReader(r parquet.ReaderAtSeeker, opts 
...ReadOption) (*Reader, er
                        return nil, err
                }
        }
+       if _, ok := f.props.Allocator().(*memory.GoAllocator); ok && 
f.fileDecryptor == nil && f.NumRowGroups() > 1 {

Review Comment:
   Could you add a comment explaining why this is limited to the Go allocator? 
Nothing empties the cache when the file is closed, and page readers can close 
after the file does, so cached buffers are never `Release`d. That's only safe 
when the garbage collector reclaims them. Enabling this for another allocator 
would leak.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to