paveon opened a new pull request, #2392: URL: https://github.com/apache/iceberg-go/pull/2392
## Why Reading a Parquet data file through `blobfs` costs far more requests than the file needs: - `Open` calls `bucket.NewReader`, which starts a GET of the whole object. The scan reads the file only through `ReadAt`, so that body is never read; `Close` then drops it, which also discards the connection instead of returning it to the pool. - `ReadAt` issues one ranged GET per call. The Parquet reader calls it for the footer length, the footer, and each column chunk, so a file of a wide table costs hundreds of GETs however small it is. For tables written in many small files (streaming appends before compaction), the scan, and so compaction, is dominated by S3 round trips. ## Change - `Open` reads only the object's attributes (size, modification time), so missing objects still fail at `Open` and `Stat`/`Seek(0, io.SeekEnd)` need no request. - `Read` starts a streaming GET from the current offset on first use; a `Seek` to another offset closes it, and the next `Read` starts a new one. - `ReadAt` on an object of at most 8 MiB downloads it once, on the first call, and serves every call from memory until `Close`. Larger objects keep one ranged GET per call, so memory for large files is unchanged. ## Tests - `TestOpenReadAtGets`: `Open` and `Stat` issue no GET; three `ReadAt`s cost one GET for a small object and three for a large one. - `TestOpenReadSeek`: `Seek(0, io.SeekEnd)` issues no GET, reads after seeks return the right bytes, invalid seeks fail. - The existing `ReadAt` cleanup and error tests pass unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
