tustvold commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603007745
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
To emphasise the point I made when this API was originally proposed, you
need more than just the ParquetStatistics in order to correctly interpret the
data. You need at least the FileMetadata to get the
https://docs.rs/parquet/latest/parquet/file/metadata/struct.FileMetaData.html#method.column_order
in order to be able to even interpret what the statistics mean for a given
column.
Additionally you need to actually have the parquet schema as the **arrow
datatype may not match what the parquet data is encoded as**. The parquet
schema is authoritative when reading parquet data, the arrow datatype is purely
what the data should be coerced to once read from parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
