englefly opened a new pull request, #66898: URL: https://github.com/apache/doris/pull/66898
### What problem does this PR solve? Issue Number: N/A Problem Summary: `StatsCalculator.computeOlapScan` fetches column stats for all output slots of the olap scan, i.e. all columns of the table, but only operative slots' stats are needed by the query. This causes two problems: 1. The global column stats cache (`StatisticsCache.columnStatisticsCache`) is polluted by entries of columns never used by any query, evicting useful entries. 2. For wide tables, a large number of useless column stats are fetched, and each cache miss triggers a query against the internal statistics table. Fix: 1. `computeOlapScan` now fetches column stats only for operative slots (derived by `OperativeColumnDerive`). For other visible output slots it reads the stats cache only when the value is already loaded (`StatisticsCache.getColumnStatisticsIfPresent`), so no cache load is triggered for columns the query does not need: the cache is not polluted and no stats table query is issued, while plans stay identical to before when the stats are already cached (e.g. injected via `ALTER TABLE SET STATS`). When operative slots are not derived yet or full stats fidelity is required (`forbid_unknown_col_stats`), it falls back to all visible output slots. `checkNdvValidation` is limited the same way to avoid useless cache loads. 2. `OperativeColumnDerive` is additionally run right before the "init join" rewrite topic, so RBO rules that derive statistics (`InitJoinOrder`, `SkewJoin`, Eager aggregation, etc.) also see operative slots and no longer fetch stats of all table columns. A comment documents that RBO rules depending on statistics must be placed after `OperativeColumnDerive`. The derivation at the end of rewrite is kept, because rules after "init join" (e.g. stream scan normalization, variant virtual column push down) may rebuild scans or add virtual columns, and the final operative slots are recomputed there for CBO stats derivation and backend lazy materialization. ### Release note None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
