bert-beyondloops opened a new issue, #24026: URL: https://github.com/apache/datafusion/issues/24026
### Describe the bug
`Statistics::calculate_total_byte_size` (datafusion/common/src/stats.rs:433)
is meant to derive total_byte_size from num_rows and the schema's fixed-width
columns. When all columns have a primitive width, it currently does:
Some(size) => {
self.total_byte_size =
self.num_rows.multiply(&Precision::Exact(size));
}
Precision::multiply returns Precision::Absent whenever either operand is
Absent. So if self.num_rows is Precision::Absent at the time this is called,
total_byte_size is
overwritten with Precision::Absent — even if the statistics object already
carried a perfectly good (exact or inexact) total_byte_size estimate computed
some other way.
This silently throws away known size information any time row counts are
unknown, which can degrade the quality of downstream statistics-based
decisions.
The non-primitive-width branch already handles this correctly by
downgrading the existing value with to_inexact() instead of discarding it — the
primitive-width branch should do the same when num_rows is Absent.
### To Reproduce
```
use arrow::datatypes::{DataType, Field, Schema};
use datafusion_common::stats::{Precision, Statistics};
let schema = Schema::new(vec![Field::new("a", DataType::Int32, false)]);
let mut stats = Statistics::new_unknown(&schema);
stats.total_byte_size = Precision::Inexact(1234); // some previously known
estimate
// num_rows is left as Precision::Absent
stats.calculate_total_byte_size(&schema);
// total_byte_size is now Precision::Absent, even though the schema is all
// fixed-width and we had a prior estimate — expected
Precision::Inexact(1234).
assert_eq!(stats.total_byte_size, Precision::Absent);
```
### Expected behavior
When `num_rows` is `Precision::Absent `but the schema is entirely
fixed-width, `calculate_total_byte_size` should preserve the previously-set
`total_byte_size` (downgraded to inexact via `to_inexact()`), consistent with
how the non-primitive-width case is handled, instead of discarding it to
`Absent`.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
