kosiew commented on code in PR #25694:
URL: https://github.com/apache/datafusion/pull/25694#discussion_r4597805409
##########
docs/source/library-user-guide/upgrading/56.0.0.md:
##########
@@ -1014,6 +1014,20 @@ Two differences for operators that still implement
`supports_limit_pushdown`:
operator, as it does above `CoalescePartitionsExec`, and is copied below as
an early-stop fetch.
+### Statistics must have one `ColumnStatistics` per schema field
Review Comment:
Please move the new statistics-contract section to
`docs/source/library-user-guide/upgrading/57.0.0.md`, since the reviewed head
declares `57.0.0-dev` in `Cargo.toml`. Keeping it here incorrectly attributes
the new planning-time rejection to 56.0.0. Move only the newly added section
and leave the existing guidance intact.
##########
datafusion/physical-plan/src/statistics.rs:
##########
@@ -583,6 +586,7 @@ impl StatisticsContext {
provider.compute_statistics_with_args(plan, &child_extended,
args)?
{
let (statistics, extensions) = computed.into_parts();
+ validate_column_statistics(plan, &statistics,
Some(provider.as_ref()))?;
let extensions = if extensions.is_empty() {
Review Comment:
Optional, low priority: could we test a partition-aware provider that
returns valid overall statistics but malformed statistics for `Some(0)`? Call
`compute_arc` for both requests using the same context and plan, then assert
overall success and a provider/count error for partition 0. This would verify
that cached overall statistics do not mask invalid partition results. The
provider must handle partition requests explicitly rather than delegate them.
##########
datafusion/physical-plan/src/operator_statistics/mod.rs:
##########
@@ -3089,4 +3082,47 @@ mod tests {
assert_eq!(stats_b.base.num_rows, Precision::Inexact(50));
Ok(())
}
+
+ /// The walk rejects an operator or provider whose number of column
+ /// statistics differs from the number of fields.
+ #[test]
+ fn test_wrong_column_statistics_count_is_rejected() {
Review Comment:
Optional, low priority: could we add a positive test with an empty schema
and zero column-statistics entries, asserting successful computation and
preservation of a known row count? The validator correctly accepts `0 == 0`,
but this would protect legitimate zero-column plans against future regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]