jayzhan211 commented on PR #26094:
URL: https://github.com/apache/datafusion/pull/26094#issuecomment-6091928822
@asolimando
Thanks for the `AggregateStatistics` test. A revert there now fails it (3 vs
42). The `LimitPushdown` half still has no test: with this line back to
`StatisticsContext::new()`, every test still passes. A provider `Exact(0)` here
removes a limit. The test below passes on this branch and fails under that
revert (1 row vs 3). The provider matches `AggregateExec` because a
`DataSourceExec` input gets the fetch pushed into the scan, so the limit never
reaches this rule.
```rs
/// `LimitPushdown` trusts an `Exact(0)` row count from a registered
/// provider: it removes a limit over that input
#[tokio::test]
async fn limit_pushdown_consults_statistics_providers() -> Result<()> {
let provider = ClosureStatisticsProvider::with_matches(
|plan| plan.is::<AggregateExec>(),
|plan, child_stats| {
let child_stats = child_stats
.iter()
.map(|c| Arc::clone(c.base_arc()))
.collect::<Vec<_>>();
let mut stats = Arc::unwrap_or_clone(
plan.statistics_from_inputs(&child_stats,
&StatisticsArgs::new())?,
);
stats.num_rows = Precision::Exact(0);
Ok(StatisticsResult::Computed(stats.into()))
},
);
let state = SessionStateBuilder::new()
.with_default_features()
.with_config(SessionConfig::new().with_target_partitions(1))
.with_statistics_registry(StatisticsRegistry::with_providers(vec![Arc::new(
provider,
)]))
.build();
let ctx = SessionContext::new_with_state(state);
ctx.sql("CREATE TABLE t AS VALUES (1), (2), (3)")
.await?
.collect()
.await?;
let batches = ctx
.sql("SELECT column1, count(*) FROM t GROUP BY column1 LIMIT 1")
.await?
.collect()
.await?;
assert_eq!(batches.iter().map(|b| b.num_rows()).sum::<usize>(), 3);
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
