alamb commented on code in PR #10852:
URL: https://github.com/apache/datafusion/pull/10852#discussion_r1640436779
##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -771,10 +885,205 @@ impl<'a> StatisticsConverter<'a> {
Ok(Arc::new(UInt64Array::from_iter(null_counts)))
}
+ /// Extract the minimum values from Data Page statistics.
+ ///
+ /// In Parquet files, in addition to the Column Chunk level statistics
+ /// (stored for each column for each row group) there are also
+ /// optional statistics stored for each data page, as part of
+ /// the [`ParquetColumnIndex`].
+ ///
+ /// Since a single Column Chunk is stored as one or more pages,
+ /// page level statistics can prune at a finer granularity.
+ ///
+ /// However since they are stored in a separate metadata
+ /// structure ([`Index`]) there is different code to extract them as
+ /// compared to arrow statistics.
+ ///
+ /// # Parameters:
+ ///
+ /// * `column_page_index`: The parquet column page indices, read from
+ /// `ParquetMetaData` column_index
+ ///
+ /// * `column_offset_index`: The parquet column offset indices, read from
+ /// `ParquetMetaData` offset_index
+ ///
+ /// * `row_group_indices`: The indices of the row groups, that are used to
+ /// extract the column page index and offset index on a per row group
+ /// per column basis.
+ ///
+ /// # Return Value
+ ///
+ /// The returned array contains 1 value for each `NativeIndex`
+ /// in the underlying `Index`es, in the same order as they appear
+ /// in `metadatas`.
+ ///
+ /// For example, if there are two `Index`es in `metadatas`:
+ /// 1. the first having `3` `PageIndex` entries
+ /// 2. the second having `2` `PageIndex` entries
+ ///
+ /// The returned array would have 5 rows.
+ ///
+ /// Each value is either:
+ /// * the minimum value for the page
+ /// * a null value, if the statistics can not be extracted
+ ///
+ /// Note that a null value does NOT mean the min value was actually
+ /// `null` it means it the requested statistic is unknown
+ ///
+ /// # Errors
+ ///
+ /// Reasons for not being able to extract the statistics include:
+ /// * the column is not present in the parquet file
+ /// * statistics for the pages are not present in the row group
+ /// * the stored statistic value can not be converted to the requested type
+ pub fn data_page_mins<I>(
+ &self,
+ column_page_index: &ParquetColumnIndex,
+ column_offset_index: &ParquetOffsetIndex,
+ row_group_indices: I,
+ ) -> Result<ArrayRef>
+ where
+ I: IntoIterator<Item = &'a usize>,
+ {
+ let data_type = self.arrow_field.data_type();
+
+ let Some(parquet_index) = self.parquet_index else {
+ return Ok(self.make_null_array(data_type, row_group_indices));
+ };
+
+ let iter = row_group_indices.into_iter().map(|rg_index| {
+ let column_page_index_per_row_group_per_column =
+ &column_page_index[*rg_index][parquet_index];
+ let num_data_pages =
&column_offset_index[*rg_index][parquet_index].len();
+
+ (*num_data_pages, column_page_index_per_row_group_per_column)
+ });
+
+ min_page_statistics(Some(data_type), iter)
+ }
+
+ /// Extract the maximum values from Data Page statistics.
+ ///
+ /// See docs on [`Self::data_page_mins`] for details.
+ pub fn data_page_maxes<I>(
+ &self,
+ column_page_index: &ParquetColumnIndex,
+ column_offset_index: &ParquetOffsetIndex,
+ row_group_indices: I,
+ ) -> Result<ArrayRef>
+ where
+ I: IntoIterator<Item = &'a usize>,
+ {
+ let data_type = self.arrow_field.data_type();
+
+ let Some(parquet_index) = self.parquet_index else {
+ return Ok(self.make_null_array(data_type, row_group_indices));
+ };
+
+ let iter = row_group_indices.into_iter().map(|rg_index| {
+ let column_page_index_per_row_group_per_column =
Review Comment:
wow those structures are hard to use 🤯 -- seems like having an accessor
would help a lot. Something to consider upstream maybe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
