Awesome06 opened a new pull request, #55453:
URL: https://github.com/apache/arrow/pull/55453
* Closes: #55272
### Rationale for this change
Parquet V2 data pages are widely supported across the ecosystem. They
provide explicit page-level null counts (`num_nulls`) in page headers and
enable per-page selective compression (`min_compression_savings`), allowing
pages that do not benefit from compression to remain uncompressed.
Benchmarks across TPC-H, SSB, and TPC-DS (discussed on GH-55272) showed that
V2 data pages yield comparable file sizes (0.0% to 1.2% delta depending on
schema sparsity) while improving read throughput by 12% to 35% on scanned
columns.
### What changes are included in this PR?
* **C++ Core (`properties.h`)**:
* Changed the default `data_page_version` in `WriterProperties::Builder`
from `ParquetDataPageVersion::V1` to `ParquetDataPageVersion::V2`.
* Updated docstrings to indicate `V2` default.
* **C++ Tests (`properties_test.cc`, `column_writer_test.cc`)**:
* Updated `TestWriterProperties.Basics` to expect
`ParquetDataPageVersion::V2`.
* Updated `MinCompressionSavingsRequiresDataPageV2` to verify default `V2`
allows `min_compression_savings` without throwing, while explicitly configured
`V1` raises an exception.
* In `WriteLargeDictEncodedPage`, updated expected page type to
`PageType::DATA_PAGE_V2`.
* **C++ Documentation (`parquet.rst`)**:
* Documented that `data_page_version` defaults to V2, with V1 retained for
older reader compatibility.
* **Python API (`core.py`, `_dataset_parquet.pyx`, `_parquet.pyx`)**:
* Changed default `data_page_version` from `'1.0'` to `'2.0'` in
`ParquetWriter`, `write_table`, and dataset write options.
* Adjusted `_parquet.pyx` validation so `min_compression_savings > 0.01`
only rejects explicit `data_page_version="1.0"`.
* **Python Tests (`test_parquet_writer.py`)**:
* Updated `test_min_compression_savings_rejects_v1` to explicitly specify
`data_page_version="1.0"`.
* Added `test_min_compression_savings_default_v2` to verify default write
paths accept `min_compression_savings` without error.
### Are these changes tested?
Yes. Updated existing C++ properties and writer unit tests, and added Python
test coverage for default V2 writer behavior.
### Are there any user-facing changes?
Yes. Files written with default writer properties in C++ and PyArrow will
now use Parquet Data Page V2 instead of V1. Users requiring V1 for legacy
compatibility can pass `data_page_version="1.0"` (Python) or
`.data_page_version(ParquetDataPageVersion::V1)` (C++).
### Was AI used for this PR?
In accordance to the [AI generation
guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code),
please disclose below whether and how AI was used in this PR.
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [ ] AI
- [ ] Not reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
