rdettai commented on a change in pull request #965:
URL: https://github.com/apache/arrow-datafusion/pull/965#discussion_r706872869
##########
File path: datafusion/src/physical_plan/mod.rs
##########
@@ -89,6 +89,34 @@ impl Stream for EmptyRecordBatchStream {
/// Physical planner interface
pub use self::planner::PhysicalPlanner;
+/// Statistics for an physical plan node
+/// Fields are optional and can be inexact because the sources
+/// sometimes provide approximate estimates for performance reasons
+/// and the transformations output are not always predictable.
+#[derive(Debug, Clone, Default, PartialEq)]
+pub struct Statistics {
+ /// The number of table rows
+ pub num_rows: Option<usize>,
+ /// total byte of the table rows
+ pub total_byte_size: Option<usize>,
+ /// Statistics on a column level
+ pub column_statistics: Option<Vec<ColumnStatistics>>,
+ /// Some datasources or transformations might provide inexact estimates
+ pub is_exact: bool,
+}
+/// This table statistics are estimates about column
+#[derive(Clone, Debug, Default, PartialEq)]
+pub struct ColumnStatistics {
Review comment:
My assumption is that the statistics will depend on the query, that they
are not global to the dataset. The table format is usually able to get the
statistics at a partition
([hive](https://hive.apache.org/javadocs/r1.2.2/api/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.html#getPartitionColumnStatistics(java.lang.String,%20java.lang.String,%20java.util.List,%20java.util.List)))
or file level
([iceberg](https://iceberg.apache.org/javadoc/0.12.0/org/apache/iceberg/TableScan.html#planFiles())),
and it is much more interesting for us to do the optimizations using
statistics that come after the partition pruning, as they will be much more
accurate than global ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
