JigaoLuo opened a new issue, #53993:
URL: https://github.com/apache/arrow/issues/53993
### Describe the enhancement requested
Parquet C++ already provides page-level access via
`RowGroupReader::GetColumnPageReader()` and `PageReader::NextPage()` (see
`cpp/src/parquet/column_page.h`), but PyArrow only exposes the Arrow-level
reader (`read_table`, `read_row_group`, `iter_batches`). From Python, the
smallest unit a Parquet file can be read at is a row group.
#### Motivation: reducing read amplification
A row group is typically tens to hundreds of MB and contains many pages per
column. When a Python user only needs a small part of a column chunk (a point
lookup, a narrow row range, one page's statistics), PyArrow still has to fetch
and decode the entire row group. On object storage this turns a
sub-millisecond, few-KB page read into a multi-MB request. The page is the
natural I/O unit of the format, and the C++ reader already works at that
granularity; it is only the Python binding that hides it.
#### Other use cases
- Inspecting/debugging files at page level (page type, encoding,
`num_values`, page statistics), similar to `parquet-cli pages`
- Custom or experimental decoding on top of PyArrow's I/O and decompression
- Page-level processing without full Arrow materialization. Downstream
projects currently re-implement page parsing in Python to get this (e.g.
apache/paimon#9850)
#### Prior art
The Lance file format's Python binding (pylance) works at page granularity.
`LanceFileReader.metadata()` exposes each column's page layout (buffer
positions/sizes and encoding), and the same page-level reader is what makes
`LanceFileReader.take_rows()` / `LanceDataset.take()` efficient random access
from Python without reading whole column chunks. Parquet C++ already has the
equivalent reader; exposing it in PyArrow would give Python users the same
building block.
#### Rough API idea (open to suggestions)
```python
pf = pq.ParquetFile("data.parquet")
for page in pf.iter_pages(row_group=0, column=0):
page.page_type # DICTIONARY_PAGE / DATA_PAGE / DATA_PAGE_V2
page.encoding
page.num_values
page.buffer # pyarrow.Buffer, decompressed page body
```
The C++ functionality exists and is stable; this would be a read-only
wrapper in the Python binding layer, with no new C++ code required. I'm happy
to work on a PR.
### Component(s)
Python, Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
