abaybek opened a new issue, #50619: URL: https://github.com/apache/arrow/issues/50619
### Describe the enhancement requested The Parquet C++ library already lets you control how much [`SizeStatistics`](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift) metadata is written, via `WriterProperties::Builder::set_size_statistics_level()` (added in #40594). The available levels are: - `None` — write no size statistics - `ColumnChunk` — write size statistics only at the column-chunk level (`ColumnMetaData.size_statistics`) - `PageAndColumnChunk` — also write page-level histograms into `ColumnIndex` / `OffsetIndex` (**this is the C++ default**) However, **PyArrow does not currently expose this setting.** There is no way from `pyarrow.parquet.write_table`, `ParquetWriter`, or the dataset writer to change the size-statistics level, so every file written by PyArrow always includes `SizeStatistics` at the `PageAndColumnChunk` level. We have hit one concrete problem with it being unconditionally written: **Reader compatibility.** Some third-party / older Parquet readers do not handle the newer `SizeStatistics` thrift fields gracefully and fail to read files that PyArrow produces. Being able to write with `size_statistics_level="none"` produces files those readers can consume, without having to downgrade any other writer settings. The C++ capability exists and is stable; this request is purely to surface it through the Python binding layer. ### Component(s) Python, Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
