This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-24027-24483dbeed7811a21480f307efd7becd87ce3111 in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 540bb67d16c320961b50f114f875ebf5ee54b092 Author: Bert Vermeiren <[email protected]> AuthorDate: Thu Aug 6 17:11:50 2026 +0200 fix: preserve total_byte_size in calculate_total_byte_size when num_r… (#24027) ## Which issue does this PR close? - Closes #24026 ## Rationale for this change `Statistics::calculate_total_byte_size `is meant to derive `total_byte_size` from `num_rows` and the schema's fixed-width columns. When all columns have a primitive width but `num_rows` is `Precision::Absent`, the old code computed `self.num_rows.multiply(&Precision::Exact(size))`, and `Precision::multiply` returns `Precision::Absent` whenever either operand is `Absent`. This silently overwrote any previously known `total_byte_size` (exact or inexact) with `Absent`, even though the non-primitive-width branch already handled this situation correctly by downgrading the existing value to inexact instead of discarding it. ## What changes are included in this PR? - In `Statistics::calculate_total_byte_size`, when the schema is all fixed-width but num_rows is` Precision::Absent,` keep the existing `total_byte_size` and downgrade it to inexact via `to_inexact(),` instead of overwriting it with `Absent`. - Updated the doc comment on `calculate_total_byte_size` to describe this behavior. - Added `test_calculate_total_byte_size` covering: an all-primitive schema with known row count (exact size), an all-primitive schema with unknown row count (preserved but downgraded to inexact), and a non-primitive schema (always downgraded to inexact regardless of row count). ## Are these changes tested? Yes — added `stats::tests::test_calculate_total_byte_size, exercising all three branches of the updated match. ## Are there any user-facing changes? No public API changes. Statistics propagation is more accurate (previously known total_byte_size estimates are no longer dropped to Absent when num_rows is unknown), which may result in slightly better cost-based planning decisions in some cases. Co-authored-by: Bert Vermeiren <[email protected]> --- datafusion/common/src/stats.rs | 39 +++++++++++++++++++++++++++++++++++---- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/datafusion/common/src/stats.rs b/datafusion/common/src/stats.rs index b7db556ee8..1a226c3698 100644 --- a/datafusion/common/src/stats.rs +++ b/datafusion/common/src/stats.rs @@ -427,7 +427,9 @@ impl Statistics { } /// Calculates `total_byte_size` based on the schema and `num_rows`. - /// If any of the columns has non-primitive width, `total_byte_size` is set to inexact. + /// If any of the columns has non-primitive width, or `num_rows` is unknown, + /// the previous `total_byte_size` is kept but downgraded to inexact rather + /// than discarded. pub fn calculate_total_byte_size(&mut self, schema: &Schema) { let mut row_size = Some(0); for field in schema.fields() { @@ -441,11 +443,11 @@ impl Statistics { } } } - match row_size { - None => { + match (row_size, &self.num_rows) { + (None, _) | (Some(_), Precision::Absent) => { self.total_byte_size = self.total_byte_size.to_inexact(); } - Some(size) => { + (Some(size), _) => { self.total_byte_size = self.num_rows.multiply(&Precision::Exact(size)); } } @@ -3345,4 +3347,33 @@ mod tests { precision_add_for_sum_in_place(&mut lhs, &Precision::Absent); assert_eq!(lhs, Precision::Absent); } + + #[test] + fn test_calculate_total_byte_size() { + let primitive_schema = Schema::new(vec![Field::new("a", DataType::Int32, false)]); + let non_primitive_schema = + Schema::new(vec![Field::new("a", DataType::Utf8, false)]); + + // All-primitive schema with a known row count computes an exact size. + let mut stats = Statistics::new_unknown(&primitive_schema); + stats.num_rows = Precision::Exact(10); + stats.calculate_total_byte_size(&primitive_schema); + assert_eq!(stats.total_byte_size, Precision::Exact(40)); + + // All-primitive schema with an unknown row count keeps a previously + // known `total_byte_size`, downgraded to inexact, instead of + // discarding it to `Absent`. + let mut stats = Statistics::new_unknown(&primitive_schema); + stats.total_byte_size = Precision::Exact(1234); + stats.calculate_total_byte_size(&primitive_schema); + assert_eq!(stats.total_byte_size, Precision::Inexact(1234)); + + // Non-primitive schema always downgrades any existing + // `total_byte_size` to inexact, regardless of `num_rows`. + let mut stats = Statistics::new_unknown(&non_primitive_schema); + stats.num_rows = Precision::Exact(10); + stats.total_byte_size = Precision::Exact(999); + stats.calculate_total_byte_size(&non_primitive_schema); + assert_eq!(stats.total_byte_size, Precision::Inexact(999)); + } } --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
