alamb commented on code in PR #25694:
URL: https://github.com/apache/datafusion/pull/25694#discussion_r4590175986


##########
datafusion/physical-plan/src/statistics.rs:
##########
@@ -709,6 +713,34 @@ impl StatisticsContext {
     }
 }
 
+/// Returns an error unless `statistics`, computed for `plan` by `provider` or,
+/// if `None`, by the operator itself, hold one [`ColumnStatistics`] per field
+/// of `plan`'s schema, as [`Statistics::column_statistics`] requires. 
Consumers
+/// index it by column, so a mismatch could make them go out of bounds or read
+/// the statistics of another column.
+///
+/// [`ColumnStatistics`]: datafusion_common::ColumnStatistics

Review Comment:
   ```suggestion
   /// Returns an error if the `statistics` mismatches the plan's schema
   ///
   /// [`ColumnStatistics`]: datafusion_common::ColumnStatistics
   ```



##########
datafusion/physical-plan/src/statistics.rs:
##########
@@ -709,6 +713,34 @@ impl StatisticsContext {
     }
 }
 
+/// Returns an error unless `statistics`, computed for `plan` by `provider` or,
+/// if `None`, by the operator itself, hold one [`ColumnStatistics`] per field
+/// of `plan`'s schema, as [`Statistics::column_statistics`] requires. 
Consumers
+/// index it by column, so a mismatch could make them go out of bounds or read
+/// the statistics of another column.
+///
+/// [`ColumnStatistics`]: datafusion_common::ColumnStatistics

Review Comment:
   this could be much clearer



##########
datafusion/physical-plan/src/execution_plan.rs:
##########
@@ -740,11 +740,15 @@ pub trait ExecutionPlan: Any + Debug + DisplayAs + Send + 
Sync {
     /// (the default), not an error.
     /// If `args.partition()` is `None`, it returns statistics for all 
partitions.
     ///
+    /// The result must hold one [`ColumnStatistics`] per field of
+    /// [`Self::schema`]; [`StatisticsContext::compute`] returns an error 
otherwise.

Review Comment:
   the detail of what returns an erorr is likely irrelevant
   
   >  [`StatisticsContext::compute`] returns an error otherwise.
   



##########
docs/source/library-user-guide/upgrading/56.0.0.md:
##########
@@ -885,6 +885,47 @@ switching to `pushdown_limit_helper_with_stats` with a 
context built from the
 statistics registry can change the result, because the providers are then
 consulted.
 
+### Statistics with the wrong number of column statistics are an error

Review Comment:
   this is way too detailed for a new error message. I don't think we need to 
document that the code detects some errors earlier



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to