alamb commented on a change in pull request #380: URL: https://github.com/apache/arrow-datafusion/pull/380#discussion_r638163739
########## File path: datafusion/src/physical_optimizer/pruning.rs ########## @@ -28,50 +28,75 @@ //! https://github.com/apache/arrow-datafusion/issues/363 it will //! be genericized. -use std::{collections::HashSet, sync::Arc}; +use std::{collections::HashSet, convert::TryInto, sync::Arc}; use arrow::{ - array::{ - make_array, new_null_array, ArrayData, ArrayRef, BooleanArray, - BooleanBufferBuilder, - }, - buffer::MutableBuffer, - datatypes::{DataType, Field, Schema}, + array::{ArrayRef, BooleanArray}, + datatypes::{Field, Schema, SchemaRef}, record_batch::RecordBatch, }; -use parquet::file::{ - metadata::RowGroupMetaData, statistics::Statistics as ParquetStatistics, -}; - use crate::{ error::{DataFusionError, Result}, execution::context::ExecutionContextState, logical_plan::{Expr, Operator}, optimizer::utils, physical_plan::{planner::DefaultPhysicalPlanner, ColumnarValue, PhysicalExpr}, + scalar::ScalarValue, }; +/// Interface to pass statistics information to [`PruningPredicates`] +pub trait PruningStatistics { + /// return the minimum value for the named column, if known + fn min_value(&self, column: &str) -> Option<ScalarValue>; Review comment: Ok, I can take a shot at trying to do this. @Dandandan are you thinking something like: ``` pub trait PruningStatistics { fn min_values(&self, column: &str) -> Option<ArrayRef>; fn max_values(&self, column: &str) -> Option<ArrayRef>; } ``` Which would be constrained to return Arrays the same number of rows? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
