englefly opened a new pull request, #69209:
URL: https://github.com/apache/doris/pull/69209
### What problem does this PR solve?
Issue Number: None
Related PR: #52048
Problem Summary:
`LINEAR_ANALYZE_TEMPLATE` collected hot values in a third CTE: it grouped
the sampled column
by value, counted each group, and kept the top `hot_value_collect_count`
groups. That CTE
re-reads the whole sample, and it is the dominant cost of the template - the
one sample
analyze picks for a single distribution column / single unique key column.
The same template
also serves the full-scan fallback of a sample analyze (the requested sample
rows exceed the
table, or the picked tablets hold fewer rows than requested), where the
extra aggregation is
not bounded by the sample size at all.
This PR removes that CTE. `hot_value` keeps its position in the result set
and is reported as
NULL, exactly like `FULL_ANALYZE_WITHOUT_HOT_VALUE_TEMPLATE` and
`MERGE_PARTITION_TEMPLATE`
already do, and every reader already treats NULL as "not collected"
(`ColumnStatistic.hotValues == null`). Hot values are still collected by
`DUJ1_ANALYZE_TEMPLATE`, which derives them from the grouped sample it
already builds for the
NDV estimate, and by a full analyze that asks for them explicitly
(`ANALYZE TABLE ... WITH HOT VALUE`).
The params that only served the removed CTE (`subStringColName`, `rowCount2`
on the linear
path, `hotValueCollectCount`) are now filled for the DUJ1 algorithm only. As
a side effect,
the linear branch of `PluginDrivenSampleAnalysisTask` no longer renders an
unsubstituted
`${subStringColName}` into its SQL: it never set that param (the legacy
`HMSAnalysisTask` it
was ported from did not either), so a bucketed plain-hive table that took
that branch got
invalid SQL before this change.
Templates checked, and why they are left alone:
| Template | Hot value | Action |
|---|---|---|
| `LINEAR_ANALYZE_TEMPLATE` | collected via an extra GROUP BY over the
sample | **removed here** |
| `DUJ1_ANALYZE_TEMPLATE` | collected from the grouped sample it already
builds for NDV | kept |
| `FULL_ANALYZE_TEMPLATE` | only when the user writes `WITH HOT VALUE` |
kept (explicit opt-in, not a default cost) |
| `FULL_ANALYZE_WITHOUT_HOT_VALUE_TEMPLATE`, `MERGE_PARTITION_TEMPLATE` |
`null as hot_value` | unchanged |
| `PARTITION_ANALYZE_TEMPLATE`, `ANALYZE_PARTITION_COLUMN_TEMPLATE` | no hot
value | unchanged |
Behaviour changes:
- A sample analyze no longer collects hot values for columns estimated with
the linear
estimator: a single distribution column / single unique key column, and
any sample analyze
that falls back to a full table scan.
- `ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` was already rejected before
this change; its
message changes from "Sample analyze always collects hot value" to "Sample
analyze does not
support WITH HOT VALUE", because the old wording is no longer true.
### Release note
Sample analyze no longer collects hot values for columns that use the linear
NDV estimator
(single distribution / single key column, and the full-scan fallback of a
sample analyze),
which removes a second aggregation over the sampled data. Hot values are
still collected by
sample analyze for columns estimated with DUJ1, and by `ANALYZE TABLE ...
WITH HOT VALUE` in a
full analyze.
### Check List (For Author)
- Test
- [x] Unit Test
- `OlapAnalysisTaskTest` (25 tests, includes the linear-vs-DUJ1 hot
value split)
- `AnalyzeTableCommandTest` (5 tests, includes the `WITH HOT VALUE`
rejection message)
- [x] Regression test
- `test_hot_value` and `test_full_analyze_hot_value` updated for the
new behaviour;
they need a built FE + cluster, so they run in CI
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Sample analyze no longer collects hot values for columns
analyzed by the linear
estimator, and the `WITH HOT VALUE` rejection message for sample
analyze changed.
- Does this need documentation?
- [ ] No.
- [x] Yes. Any statement that sample analyze collects hot values needs
to be narrowed; no
docs PR yet.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
