bhavya-sl opened a new issue, #55324: URL: https://github.com/apache/arrow/issues/55324
### Describe the enhancement requested `ReaderProperties::set_footer_read_size()` was added to C++ in GH-45015 (PR #45016), but there is no Python binding for it. I checked pyarrow 17.0.0 and 26.0.0 (current latest): it is reachable from neither `pyarrow.parquet.ParquetFile(...)` nor `pyarrow.dataset.ParquetFragmentScanOptions(...)`, and `grep -rn footer_read_size python/` finds nothing. `kDefaultFooterReadSize` is 64 KiB. When the footer is larger, `ParseMetaData()` issues a second `ReadAt` for the remainder — the `footer_read_size >= (metadata_len + kFooterSize)` branch in `cpp/src/parquet/file_reader.cc`. On high-latency object storage that second read costs a full extra round trip per file, and the extra bytes it would have cost to avoid it are essentially free. Measured against S3 from an EC2 instance in the same region (TTFB ~51 ms, ~74 MiB/s per connection), on files with a few hundred row groups and 13 columns: | file | row groups | footer size | one 64 KiB ranged read | one 512 KiB ranged read | `ParquetFile(...).metadata` | |---|---|---|---|---|---| | A | 189 | 242 KB | 33.0 ms | 34.7 ms | **58.5 ms** | | B | 83 | 106 KB | 32.0 ms | 32.7 ms | **59.2 ms** | | C | 161 | 202 KB | 42.6 ms | 34.7 ms | **67.9 ms** | A 512 KiB ranged read costs the same as a 64 KiB one — this is pure latency, the size is irrelevant at that scale — while opening the file and parsing its footer costs ~1.8x a single read, consistent with it being two round trips. Over a query touching 200 such files that is roughly 6 s of avoidable latency. Footers in this range are not unusual: a few hundred row groups x a dozen columns, with statistics, is enough to exceed 64 KiB. Writing fewer row groups is not an alternative when row-group count is exactly the pruning granularity the query depends on. ### Suggested shape A passthrough in the same style as the existing `thrift_string_size_limit` / `thrift_container_size_limit` options, which already follow this pattern: ```python pq.ParquetFile(path, filesystem=fs, footer_read_size=512 * 1024) ds.ParquetFragmentScanOptions(footer_read_size=512 * 1024) ``` Happy to put up a PR if that shape sounds right. ### Component(s) Python, Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
