alamb commented on code in PR #10719:
URL: https://github.com/apache/arrow-rs/pull/10719#discussion_r3817498357
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -1313,12 +1313,11 @@ impl<T: ChunkReader + 'static> ReaderPageIterator<T> {
fn next_page_reader(&self, rg_idx: usize) ->
Result<SerializedPageReader<T>> {
let rg = self.metadata.row_group(rg_idx);
let column_chunk_metadata = rg.column(self.column_idx);
- let offset_index = self.metadata.offset_index();
- // `offset_index` may not exist and `i[rg_idx]` will be empty.
- // To avoid `i[rg_idx][self.column_idx`] panic, we need to filter out
empty `i[rg_idx]`.
- let page_locations = offset_index
- .filter(|i| !i[rg_idx].is_empty())
- .map(|i| i[rg_idx][self.column_idx].page_locations.clone());
+ let page_locations = self
+ .metadata
+ .page_index()
+ .map(|i| i.page_locations(rg_idx, self.column_idx).cloned())
Review Comment:
this clone is unfortunate (it clones all the page locations into a new Vec)
-- I realize it is what the previous code did, but I wonder if there is some
way to avoid it
It may also be related to
- https://github.com/apache/arrow-rs/issues/7582
Where @zhuqi-lucas and others have been looking for a way to load some but
not all page indexes (or load them on demand, from a cache, etc).
Maybe it is time to sprinkle on some `Arc` 🤔
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -134,36 +134,273 @@ use std::sync::Arc;
pub use writer::ParquetMetaDataWriter;
pub(crate) use writer::ThriftMetadataWriter;
-/// Page level statistics for each column chunk of each row group.
+/// Encapsulates the Parquet [Page Index] for efficient page-level data
skipping
///
-/// This structure is an in-memory representation of multiple [`ColumnIndex`]
-/// structures in a parquet file footer, as described in the Parquet [PageIndex
-/// documentation]. Each [`ColumnIndex`] holds statistics about all the pages
in a
-/// particular column chunk.
+/// The Page Index is optional metadata that enables query engines to skip
irrelevant
+/// data pages during scans, significantly improving I/O efficiency. It
consists of two
+/// complementary structures:
///
-/// `column_index[row_group_number][column_number]` holds the
-/// [`ColumnIndex`] corresponding to column `column_number` of row group
-/// `row_group_number`.
+/// * **[`ColumnIndex`]**: Per-page min/max value boundaries that enable
predicate-based
+/// page filtering. Allows determining which pages might contain rows
matching a query
+/// predicate without reading the actual data pages.
///
-/// For example `column_index[2][3]` holds the [`ColumnIndex`] for the fourth
-/// column in the third row group of the parquet file.
+/// * **[`OffsetIndex`]**: Physical locations and sizes of data pages, plus
the first row
+/// index of each page. Used to locate and read only the pages identified as
relevant
+/// by the ColumnIndex.
///
-/// [PageIndex documentation]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-/// [`ColumnIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-pub type ParquetColumnIndex = Vec<Vec<ColumnIndexMetaData>>;
-
-/// [`OffsetIndexMetaData`] for each data page of each row group of each column
+/// Together, these indexes enable:
+/// - Single-row lookups reading only one data page per column (on sorted
columns)
+/// - Range scans reading only pages containing values in the query range
+/// - Efficient cross-column filtering by skipping corresponding row ranges
+///
+/// # Structure
+///
+/// Both indexes are organized as a two-level structure:
+/// - First level: indexed by row group number
Review Comment:
👍
(though this might end up being an internal implementation detail that would
be better put closer to the field definitions)
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -134,36 +134,273 @@ use std::sync::Arc;
pub use writer::ParquetMetaDataWriter;
pub(crate) use writer::ThriftMetadataWriter;
-/// Page level statistics for each column chunk of each row group.
+/// Encapsulates the Parquet [Page Index] for efficient page-level data
skipping
///
-/// This structure is an in-memory representation of multiple [`ColumnIndex`]
-/// structures in a parquet file footer, as described in the Parquet [PageIndex
-/// documentation]. Each [`ColumnIndex`] holds statistics about all the pages
in a
-/// particular column chunk.
+/// The Page Index is optional metadata that enables query engines to skip
irrelevant
+/// data pages during scans, significantly improving I/O efficiency. It
consists of two
+/// complementary structures:
///
-/// `column_index[row_group_number][column_number]` holds the
-/// [`ColumnIndex`] corresponding to column `column_number` of row group
-/// `row_group_number`.
+/// * **[`ColumnIndex`]**: Per-page min/max value boundaries that enable
predicate-based
+/// page filtering. Allows determining which pages might contain rows
matching a query
+/// predicate without reading the actual data pages.
///
-/// For example `column_index[2][3]` holds the [`ColumnIndex`] for the fourth
-/// column in the third row group of the parquet file.
+/// * **[`OffsetIndex`]**: Physical locations and sizes of data pages, plus
the first row
+/// index of each page. Used to locate and read only the pages identified as
relevant
+/// by the ColumnIndex.
///
-/// [PageIndex documentation]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-/// [`ColumnIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-pub type ParquetColumnIndex = Vec<Vec<ColumnIndexMetaData>>;
-
-/// [`OffsetIndexMetaData`] for each data page of each row group of each column
+/// Together, these indexes enable:
+/// - Single-row lookups reading only one data page per column (on sorted
columns)
+/// - Range scans reading only pages containing values in the query range
+/// - Efficient cross-column filtering by skipping corresponding row ranges
+///
+/// # Structure
+///
+/// Both indexes are organized as a two-level structure:
+/// - First level: indexed by row group number
+/// - Second level: indexed by column number within that row group
+///
+/// Each entry is `Option<T>` because:
+/// - The entire page index might be absent (old files, disabled during write)
+/// - Individual columns might lack indexes (unsupported types, statistics
disabled)
+///
+/// # Example: Checking if Page Index is Available
///
-/// This structure is the parsed representation of the [`OffsetIndex`] from the
-/// Parquet file footer, as described in the Parquet [PageIndex documentation].
+/// ```
+/// use parquet::file::metadata::ParquetMetaData;
+/// # use parquet::errors::Result;
+///
+/// fn check_page_index_availability(metadata: &ParquetMetaData) -> Result<()>
{
+/// if let Some(page_index) = metadata.page_index() {
+/// println!("Page index present:");
+/// println!(" Has offset indexes: {}",
page_index.has_offset_indexes());
+/// println!(" Has column indexes: {}",
page_index.has_column_indexes());
+///
+/// // Check availability for first row group, first column
+/// if let Some(col_idx) = page_index.column_index(0, 0) {
+/// println!(" Column index found for row group 0, column 0");
+/// println!(" Number of pages: {}", col_idx.num_pages());
+/// }
///
-/// `offset_index[row_group_number][column_number]` holds
-/// the [`OffsetIndexMetaData`] corresponding to column
-/// `column_number`of row group `row_group_number`.
+/// if let Some(offset_idx) = page_index.offset_index(0, 0) {
+/// println!(" Offset index found for row group 0, column 0");
+/// println!(" Number of pages: {}",
offset_idx.page_locations().len());
+/// }
+/// } else {
+/// println!("No page index available");
+/// }
+/// Ok(())
+/// }
+/// ```
+///
+/// # Example: Using Page Index for Predicate Pushdown
///
-/// [PageIndex documentation]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-/// [`OffsetIndex`]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-pub type ParquetOffsetIndex = Vec<Vec<OffsetIndexMetaData>>;
+/// ```
+/// use parquet::file::metadata::ParquetMetaData;
+/// use parquet::file::page_index::column_index::ColumnIndexMetaData;
+/// # use parquet::errors::Result;
+///
+/// /// Identifies which pages in a column might contain values >= min_value
+/// fn find_relevant_pages(
Review Comment:
this is a (very) cool example
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -134,36 +134,273 @@ use std::sync::Arc;
pub use writer::ParquetMetaDataWriter;
pub(crate) use writer::ThriftMetadataWriter;
-/// Page level statistics for each column chunk of each row group.
+/// Encapsulates the Parquet [Page Index] for efficient page-level data
skipping
///
-/// This structure is an in-memory representation of multiple [`ColumnIndex`]
-/// structures in a parquet file footer, as described in the Parquet [PageIndex
-/// documentation]. Each [`ColumnIndex`] holds statistics about all the pages
in a
-/// particular column chunk.
+/// The Page Index is optional metadata that enables query engines to skip
irrelevant
+/// data pages during scans, significantly improving I/O efficiency. It
consists of two
+/// complementary structures:
///
-/// `column_index[row_group_number][column_number]` holds the
-/// [`ColumnIndex`] corresponding to column `column_number` of row group
-/// `row_group_number`.
+/// * **[`ColumnIndex`]**: Per-page min/max value boundaries that enable
predicate-based
+/// page filtering. Allows determining which pages might contain rows
matching a query
+/// predicate without reading the actual data pages.
///
-/// For example `column_index[2][3]` holds the [`ColumnIndex`] for the fourth
-/// column in the third row group of the parquet file.
+/// * **[`OffsetIndex`]**: Physical locations and sizes of data pages, plus
the first row
+/// index of each page. Used to locate and read only the pages identified as
relevant
+/// by the ColumnIndex.
///
-/// [PageIndex documentation]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-/// [`ColumnIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-pub type ParquetColumnIndex = Vec<Vec<ColumnIndexMetaData>>;
-
-/// [`OffsetIndexMetaData`] for each data page of each row group of each column
+/// Together, these indexes enable:
+/// - Single-row lookups reading only one data page per column (on sorted
columns)
+/// - Range scans reading only pages containing values in the query range
+/// - Efficient cross-column filtering by skipping corresponding row ranges
+///
+/// # Structure
+///
+/// Both indexes are organized as a two-level structure:
+/// - First level: indexed by row group number
+/// - Second level: indexed by column number within that row group
+///
+/// Each entry is `Option<T>` because:
+/// - The entire page index might be absent (old files, disabled during write)
+/// - Individual columns might lack indexes (unsupported types, statistics
disabled)
+///
+/// # Example: Checking if Page Index is Available
///
-/// This structure is the parsed representation of the [`OffsetIndex`] from the
-/// Parquet file footer, as described in the Parquet [PageIndex documentation].
+/// ```
+/// use parquet::file::metadata::ParquetMetaData;
+/// # use parquet::errors::Result;
+///
+/// fn check_page_index_availability(metadata: &ParquetMetaData) -> Result<()>
{
+/// if let Some(page_index) = metadata.page_index() {
+/// println!("Page index present:");
+/// println!(" Has offset indexes: {}",
page_index.has_offset_indexes());
+/// println!(" Has column indexes: {}",
page_index.has_column_indexes());
+///
+/// // Check availability for first row group, first column
+/// if let Some(col_idx) = page_index.column_index(0, 0) {
+/// println!(" Column index found for row group 0, column 0");
+/// println!(" Number of pages: {}", col_idx.num_pages());
+/// }
///
-/// `offset_index[row_group_number][column_number]` holds
-/// the [`OffsetIndexMetaData`] corresponding to column
-/// `column_number`of row group `row_group_number`.
+/// if let Some(offset_idx) = page_index.offset_index(0, 0) {
+/// println!(" Offset index found for row group 0, column 0");
+/// println!(" Number of pages: {}",
offset_idx.page_locations().len());
+/// }
+/// } else {
+/// println!("No page index available");
+/// }
+/// Ok(())
+/// }
+/// ```
+///
+/// # Example: Using Page Index for Predicate Pushdown
///
-/// [PageIndex documentation]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-/// [`OffsetIndex`]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-pub type ParquetOffsetIndex = Vec<Vec<OffsetIndexMetaData>>;
+/// ```
+/// use parquet::file::metadata::ParquetMetaData;
+/// use parquet::file::page_index::column_index::ColumnIndexMetaData;
+/// # use parquet::errors::Result;
+///
+/// /// Identifies which pages in a column might contain values >= min_value
+/// fn find_relevant_pages(
+/// metadata: &ParquetMetaData,
+/// row_group_idx: usize,
+/// column_idx: usize,
+/// min_value: i32,
+/// ) -> Vec<usize> {
+/// let mut relevant_pages = Vec::new();
+///
+/// let Some(page_index) = metadata.page_index() else {
+/// // No page index - must read all pages
+/// return relevant_pages;
+/// };
+///
+/// let Some(column_index) = page_index.column_index(row_group_idx,
column_idx) else {
+/// // No column index - must read all pages
+/// return relevant_pages;
+/// };
+///
+/// // Check each page's statistics
+/// match column_index {
+/// ColumnIndexMetaData::INT32(index) => {
+/// for (page_num, max_value) in
index.max_values_iter().enumerate() {
+/// // Page might contain matching rows if its max >= our min
+/// if let Some(max) = max_value {
+/// if *max >= min_value {
+/// relevant_pages.push(page_num);
+/// }
+/// }
+/// }
+/// }
+/// _ => {
+/// // Wrong column type - read all pages
+/// }
+/// }
+///
+/// relevant_pages
+/// }
+/// ```
+///
+/// [Page Index]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
+/// [`ColumnIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
+/// [`OffsetIndex`]: crate::file::page_index::offset_index::OffsetIndexMetaData
+#[derive(Debug, Clone, PartialEq)]
+pub struct PageIndex {
+ column_indexes: Option<Vec<Vec<Option<ColumnIndexMetaData>>>>,
Review Comment:
Given the comment above about cloning / handling incremental index loading,
we might consider
```rust
column_indexes: Option<Vec<Arc<[Option<ColumnIndexMetaData>]>>
```
So that the indexes could be returned as a
`&Arc<[Option<ColumnIndexMetaData]>` (and thus cheaply cloned) as well as
potenitally being able to reuse existing column metadata)
However I am not sure it really matters and don't feel super strongly
##########
parquet/src/arrow/arrow_reader/statistics.rs:
##########
@@ -1904,8 +1899,7 @@ impl<'a> StatisticsConverter<'a> {
/// See docs on [`Self::data_page_mins`] for details.
pub fn data_page_maxes<I>(
&self,
- column_page_index: &ParquetColumnIndex,
- column_offset_index: &ParquetOffsetIndex,
+ page_index: &PageIndex,
Review Comment:
that is looking much nicer
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -134,36 +134,273 @@ use std::sync::Arc;
pub use writer::ParquetMetaDataWriter;
pub(crate) use writer::ThriftMetadataWriter;
-/// Page level statistics for each column chunk of each row group.
+/// Encapsulates the Parquet [Page Index] for efficient page-level data
skipping
///
-/// This structure is an in-memory representation of multiple [`ColumnIndex`]
-/// structures in a parquet file footer, as described in the Parquet [PageIndex
-/// documentation]. Each [`ColumnIndex`] holds statistics about all the pages
in a
-/// particular column chunk.
+/// The Page Index is optional metadata that enables query engines to skip
irrelevant
+/// data pages during scans, significantly improving I/O efficiency. It
consists of two
+/// complementary structures:
///
-/// `column_index[row_group_number][column_number]` holds the
-/// [`ColumnIndex`] corresponding to column `column_number` of row group
-/// `row_group_number`.
+/// * **[`ColumnIndex`]**: Per-page min/max value boundaries that enable
predicate-based
+/// page filtering. Allows determining which pages might contain rows
matching a query
+/// predicate without reading the actual data pages.
///
-/// For example `column_index[2][3]` holds the [`ColumnIndex`] for the fourth
-/// column in the third row group of the parquet file.
+/// * **[`OffsetIndex`]**: Physical locations and sizes of data pages, plus
the first row
+/// index of each page. Used to locate and read only the pages identified as
relevant
+/// by the ColumnIndex.
///
-/// [PageIndex documentation]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-/// [`ColumnIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
-pub type ParquetColumnIndex = Vec<Vec<ColumnIndexMetaData>>;
-
-/// [`OffsetIndexMetaData`] for each data page of each row group of each column
+/// Together, these indexes enable:
+/// - Single-row lookups reading only one data page per column (on sorted
columns)
+/// - Range scans reading only pages containing values in the query range
+/// - Efficient cross-column filtering by skipping corresponding row ranges
+///
+/// # Structure
+///
+/// Both indexes are organized as a two-level structure:
+/// - First level: indexed by row group number
+/// - Second level: indexed by column number within that row group
+///
+/// Each entry is `Option<T>` because:
+/// - The entire page index might be absent (old files, disabled during write)
+/// - Individual columns might lack indexes (unsupported types, statistics
disabled)
+///
+/// # Example: Checking if Page Index is Available
///
-/// This structure is the parsed representation of the [`OffsetIndex`] from the
-/// Parquet file footer, as described in the Parquet [PageIndex documentation].
+/// ```
+/// use parquet::file::metadata::ParquetMetaData;
+/// # use parquet::errors::Result;
+///
+/// fn check_page_index_availability(metadata: &ParquetMetaData) -> Result<()>
{
+/// if let Some(page_index) = metadata.page_index() {
+/// println!("Page index present:");
+/// println!(" Has offset indexes: {}",
page_index.has_offset_indexes());
+/// println!(" Has column indexes: {}",
page_index.has_column_indexes());
+///
+/// // Check availability for first row group, first column
+/// if let Some(col_idx) = page_index.column_index(0, 0) {
+/// println!(" Column index found for row group 0, column 0");
+/// println!(" Number of pages: {}", col_idx.num_pages());
+/// }
///
-/// `offset_index[row_group_number][column_number]` holds
-/// the [`OffsetIndexMetaData`] corresponding to column
-/// `column_number`of row group `row_group_number`.
+/// if let Some(offset_idx) = page_index.offset_index(0, 0) {
+/// println!(" Offset index found for row group 0, column 0");
+/// println!(" Number of pages: {}",
offset_idx.page_locations().len());
+/// }
+/// } else {
+/// println!("No page index available");
+/// }
+/// Ok(())
+/// }
+/// ```
+///
+/// # Example: Using Page Index for Predicate Pushdown
///
-/// [PageIndex documentation]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-/// [`OffsetIndex`]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
-pub type ParquetOffsetIndex = Vec<Vec<OffsetIndexMetaData>>;
+/// ```
+/// use parquet::file::metadata::ParquetMetaData;
+/// use parquet::file::page_index::column_index::ColumnIndexMetaData;
+/// # use parquet::errors::Result;
+///
+/// /// Identifies which pages in a column might contain values >= min_value
+/// fn find_relevant_pages(
+/// metadata: &ParquetMetaData,
+/// row_group_idx: usize,
+/// column_idx: usize,
+/// min_value: i32,
+/// ) -> Vec<usize> {
+/// let mut relevant_pages = Vec::new();
+///
+/// let Some(page_index) = metadata.page_index() else {
+/// // No page index - must read all pages
+/// return relevant_pages;
+/// };
+///
+/// let Some(column_index) = page_index.column_index(row_group_idx,
column_idx) else {
+/// // No column index - must read all pages
+/// return relevant_pages;
+/// };
+///
+/// // Check each page's statistics
+/// match column_index {
+/// ColumnIndexMetaData::INT32(index) => {
+/// for (page_num, max_value) in
index.max_values_iter().enumerate() {
+/// // Page might contain matching rows if its max >= our min
+/// if let Some(max) = max_value {
+/// if *max >= min_value {
+/// relevant_pages.push(page_num);
+/// }
+/// }
+/// }
+/// }
+/// _ => {
+/// // Wrong column type - read all pages
+/// }
+/// }
+///
+/// relevant_pages
+/// }
+/// ```
+///
+/// [Page Index]:
https://github.com/apache/parquet-format/blob/master/PageIndex.md
+/// [`ColumnIndex`]: crate::file::page_index::column_index::ColumnIndexMetaData
+/// [`OffsetIndex`]: crate::file::page_index::offset_index::OffsetIndexMetaData
+#[derive(Debug, Clone, PartialEq)]
+pub struct PageIndex {
+ column_indexes: Option<Vec<Vec<Option<ColumnIndexMetaData>>>>,
+ offset_indexes: Option<Vec<Vec<Option<OffsetIndexMetaData>>>>,
+}
+
+impl PageIndex {
+ pub(crate) fn new(
+ column_indexes: Option<Vec<Vec<Option<ColumnIndexMetaData>>>>,
+ offset_indexes: Option<Vec<Vec<Option<OffsetIndexMetaData>>>>,
+ ) -> Self {
+ Self {
+ column_indexes,
+ offset_indexes,
+ }
+ }
+
+ /// Returns `true` if offset index structures are present
+ ///
+ /// This indicates whether [`OffsetIndexMetaData`] structures were loaded
or created.
+ /// Returns `true` even if some individual columns lack offset indexes.
+ ///
+ /// To check if a specific column has an offset index, use
[`Self::offset_index`].
+ pub fn has_offset_indexes(&self) -> bool {
+ self.offset_indexes.is_some()
+ }
+
+ /// Returns `true` if column index structures are present
+ ///
+ /// This indicates whether [`ColumnIndexMetaData`] structures were loaded
or created.
+ /// Returns `true` even if some individual columns lack column indexes.
+ ///
+ /// To check if a specific column has a column index, use
[`Self::column_index`].
+ pub fn has_column_indexes(&self) -> bool {
+ self.column_indexes.is_some()
+ }
+
+ /// Returns column indexes for all columns in the specified row group
Review Comment:
these accessors are (so) much easier to read in my mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
