englefly opened a new pull request, #69636:
URL: https://github.com/apache/doris/pull/69636
### What problem does this PR solve?
Problem Summary:
When only a subset of partitions is selected,
`StatsCalculator.computeOlapScan` scales `numNulls` of the column statistics by
`selectedPartitionsRowCount / tableRowCount`, because the table level
statistics cover the whole table.
With `enable_partition_analyze = true` the statistics of the selected
partitions are used instead, and their `numNulls` already only covers the
selected partitions. The scaling was applied to them anyway, so `numNulls` was
scaled a second time and became too small.
Reproduction: a table with 3 partitions of 4 rows each and 1 null in `val`
per partition. Pruning to one partition (4 of 12 rows) reports `numNulls =
0.3333` in `EXPLAIN MEMO PLAN` instead of `1`, and `select * from t where val
is null` is estimated to return `0.33` rows instead of `1`:
```
val#1 -> ndv=1.0000, min=2.000000(2), max=2.000000(2), count=4.0000,
numNulls=0.3333, ...
```
Fix:
- make the partition statistics helper return `null` instead of silently
falling back to the table level statistics, so that the caller knows which
scope the returned statistic has;
- scale `numNulls` only when the table level statistics are really used,
i.e. when `enable_partition_analyze` is off, or when the partition statistics
are unavailable and the scan falls back to the table level ones.
### Release note
None
### Check List (For Author)
- Test: Unit Test
-
`StatsCalculatorTest#testComputeOlapScanDoesNotScaleNumNullsOfPartitionStatistics`
reproduces the bug: without the fix it fails with `expected: <1.0> but was:
<0.3333333333333333>`.
-
`StatsCalculatorTest#testComputeOlapScanFallsBackToTableStatisticsWhenPartitionStatisticIsUnknown`
covers the fallback path, which still has to be scaled.
-
`StatsCalculatorTest#testComputeOlapScanScalesNumNullsForSelectedPartitions`
(existing case, table level statistics) still passes.
- `mvn -pl fe-core -am test -Dtest=StatsCalculatorTest` passes (19
tests), checkstyle passes.
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
