liamzwbao commented on code in PR #17061:
URL: https://github.com/apache/datafusion/pull/17061#discussion_r2286543719
##########
datafusion/physical-plan/src/repartition/mod.rs:
##########
@@ -755,10 +756,40 @@ impl ExecutionPlan for RepartitionExec {
}
fn partition_statistics(&self, partition: Option<usize>) ->
Result<Statistics> {
- if partition.is_none() {
- self.input.partition_statistics(None)
+ if let Some(partition) = partition {
+ let partition_count = self.partitioning().partition_count();
+ if partition >= partition_count {
+ return internal_err!(
+ "RepartitionExec invalid partition {} (expected less than
{})",
+ partition,
+ self.partitioning().partition_count()
+ );
+ }
+
+ let mut stats = self.input.partition_statistics(None)?;
+
+ // Distribute statistics across partitions
+ stats.num_rows = stats
+ .num_rows
+ .get_value()
+ .map(|rows| Precision::Inexact(rows / partition_count))
+ .unwrap_or(Precision::Absent);
+ stats.total_byte_size = stats
+ .total_byte_size
+ .get_value()
+ .map(|bytes| Precision::Inexact(bytes / partition_count))
Review Comment:
Good callout, actually if partition_count is 0, the function will return an
internal error
[here](https://github.com/apache/datafusion/pull/17061/files/be5073a2e7714f495f839ddeb436c9e4f70a3b6f#diff-0886bee029ab49ae97b48450576c44717fd0300d8cff9edcde6308f1a40e0c3eR761)
and will not reach this line of code, added a test case for it. But it's worth
thinking that whether we should return an error or unknown statistics. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
