etseidl commented on code in PR #10719: URL: https://github.com/apache/arrow-rs/pull/10719#discussion_r3823382904
########## parquet/src/file/metadata/mod.rs: ########## @@ -134,36 +134,273 @@ use std::sync::Arc; pub use writer::ParquetMetaDataWriter; pub(crate) use writer::ThriftMetadataWriter; -/// Page level statistics for each column chunk of each row group. +/// Encapsulates the Parquet [Page Index] for efficient page-level data skipping /// -/// This structure is an in-memory representation of multiple [`ColumnIndex`] -/// structures in a parquet file footer, as described in the Parquet [PageIndex -/// documentation]. Each [`ColumnIndex`] holds statistics about all the pages in a -/// particular column chunk. +/// The Page Index is optional metadata that enables query engines to skip irrelevant +/// data pages during scans, significantly improving I/O efficiency. It consists of two +/// complementary structures: /// -/// `column_index[row_group_number][column_number]` holds the -/// [`ColumnIndex`] corresponding to column `column_number` of row group -/// `row_group_number`. +/// * **[`ColumnIndex`]**: Per-page min/max value boundaries that enable predicate-based +/// page filtering. Allows determining which pages might contain rows matching a query +/// predicate without reading the actual data pages. /// -/// For example `column_index[2][3]` holds the [`ColumnIndex`] for the fourth -/// column in the third row group of the parquet file. +/// * **[`OffsetIndex`]**: Physical locations and sizes of data pages, plus the first row +/// index of each page. Used to locate and read only the pages identified as relevant +/// by the ColumnIndex. /// -/// [PageIndex documentation]: https://github.com/apache/parquet-format/blob/master/PageIndex.md -/// [`ColumnIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData -pub type ParquetColumnIndex = Vec<Vec<ColumnIndexMetaData>>; - -/// [`OffsetIndexMetaData`] for each data page of each row group of each column +/// Together, these indexes enable: +/// - Single-row lookups reading only one data page per column (on sorted columns) +/// - Range scans reading only pages containing values in the query range +/// - Efficient cross-column filtering by skipping corresponding row ranges +/// +/// # Structure +/// +/// Both indexes are organized as a two-level structure: +/// - First level: indexed by row group number Review Comment: addressed this in https://github.com/apache/arrow-rs/pull/10719/commits/a4b7303c381a6a7cc13afdb9119370d951e8365c -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
