This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a1a93fc4302 [improvement](statistics) Stop collecting hot value in the
linear sample estimator (#69209)
a1a93fc4302 is described below
commit a1a93fc43027331af371a172904342c056dbc81e
Author: minghong <[email protected]>
AuthorDate: Sat Oct 17 12:29:55 2026 +0800
[improvement](statistics) Stop collecting hot value in the linear sample
estimator (#69209)
### What problem does this PR solve?
Related PR: #52048
Problem Summary:
`LINEAR_ANALYZE_TEMPLATE` collected hot values in a third CTE: it
grouped the sampled column
by value, counted each group, and kept the top `hot_value_collect_count`
groups. That CTE
re-reads the whole sample, and it is the dominant cost of the template -
the one sample
analyze picks for a single distribution column / single unique key
column. The same template
also serves the full-scan fallback of a sample analyze (the requested
sample rows exceed the
table, or the picked tablets hold fewer rows than requested), where the
extra aggregation is
not bounded by the sample size at all.
This PR removes that CTE. `hot_value` keeps its position in the result
set and is reported as
NULL, exactly like `FULL_ANALYZE_WITHOUT_HOT_VALUE_TEMPLATE` and
`MERGE_PARTITION_TEMPLATE`
already do, and every reader already treats NULL as "not collected"
(`ColumnStatistic.hotValues == null`). Hot values are still collected by
`DUJ1_ANALYZE_TEMPLATE`, which derives them from the grouped sample it
already builds for the
NDV estimate, and by a full analyze that asks for them explicitly
(`ANALYZE TABLE ... WITH HOT VALUE`).
The params that only served the removed CTE (`subStringColName`,
`rowCount2` on the linear
path, `hotValueCollectCount`) are now filled for the DUJ1 algorithm
only. As a side effect,
the linear branch of `PluginDrivenSampleAnalysisTask` no longer renders
an unsubstituted
`${subStringColName}` into its SQL: it never set that param (the legacy
`HMSAnalysisTask` it
was ported from did not either), so a bucketed plain-hive table that
took that branch got
invalid SQL before this change.
Templates checked, and why they are left alone:
| Template | Hot value | Action |
|---|---|---|
| `LINEAR_ANALYZE_TEMPLATE` | collected via an extra GROUP BY over the
sample | **removed here** |
| `DUJ1_ANALYZE_TEMPLATE` | collected from the grouped sample it already
builds for NDV | kept |
| `FULL_ANALYZE_TEMPLATE` | only when the user writes `WITH HOT VALUE` |
kept (explicit opt-in, not a default cost) |
| `FULL_ANALYZE_WITHOUT_HOT_VALUE_TEMPLATE`, `MERGE_PARTITION_TEMPLATE`
| `null as hot_value` | unchanged |
| `PARTITION_ANALYZE_TEMPLATE`, `ANALYZE_PARTITION_COLUMN_TEMPLATE` | no
hot value | unchanged |
Behaviour changes:
- A sample analyze no longer collects hot values for columns estimated
with the linear
estimator: a single distribution column / single unique key column, and
any sample analyze
that falls back to a full table scan.
- Planner consequences of the missing hot value, stated explicitly
because they are the point of
the trade-off: `ChildrenPropertiesRegulator` (`shouldBanOnePhaseAgg` →
`skewOnShuffleExpr`
requires hot values and now bails out) no longer picks the skew-aware
aggregation shape for
those columns, `ShuffleKeyPruneUtils` skips the shuffle-key prune,
`SkewJoin` finds no hot
value, and `DecomposeRepeatWithPreAggregation` treats the missing hot
value as "balanced"
(it calls `hasSignificantHotValues` with `strict=false`). These change
plan *quality* only:
every consumer null-checks "not collected", and none of them produces a
wrong result. The
behaviour is the same one a full analyze without `WITH HOT VALUE`
already produces today.
- `ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` was already rejected
before this change; its
message changes from "Sample analyze always collects hot value" to
"Sample analyze does not
support WITH HOT VALUE", because the old wording is no longer true for
linear-estimated
columns.
- For a column in the linear class an automatic analyze can no longer
collect hot values at all:
`collectHotValue` is still forced to true for every sample job, but only
the DUJ1 estimator
publishes them, and the automatic job never runs a full analyze. Getting
them back requires a
manual `ANALYZE TABLE ... WITH HOT VALUE`.
- `ANALYZE ... WITH HOT VALUE` still does not publish hot values when
partition-level analyze is
enabled (`enable_partition_analyze=true` on a partitioned table),
because that path merges
per-partition statistics; pre-existing behaviour, noted here only so the
release note is not
read as unconditional.
- Regression suites that pinned the old behaviour are updated in this
PR: `test_hot_value`
(both sample-request-larger-than-table blocks now expect no hot value,
and the sampled linear
path is pinned as well), `test_full_analyze_hot_value` (test 5 documents
the fallback, test 7
compares a full analyze against a real DUJ1 sample on a uniform column),
`hot_value_analyze_sync` (single bucket plus an enlarged table, so the
analyze is guaranteed to
sample and its column keeps collecting hot value through DUJ1), and
`physical_agg_regulator`
(its two `analyze` statements now use `WITH HOT VALUE`, because its
expected plan shapes were
built on collected hot values and the suite tests the aggregation
strategy, not the analyze
flavour).
### Release note
Sample analyze no longer collects hot values for columns that use the
linear NDV estimator
(single distribution / single key column, and the full-scan fallback of
a sample analyze),
which removes a second aggregation over the sampled data. Hot values are
still collected by
sample analyze for columns estimated with DUJ1, and by `ANALYZE TABLE
... WITH HOT VALUE` in a
full analyze - except when partition-level analyze is enabled, where the
per-partition path has
never produced them. For a column in the linear class an automatic
analyze cannot collect hot
values any more, so a manual `ANALYZE TABLE ... WITH HOT VALUE` is the
way to get them back.
---
.../trees/plans/commands/AnalyzeCommand.java | 2 +-
.../statistics/analysis/BaseAnalysisTask.java | 31 +++++++-----
.../statistics/analysis/OlapAnalysisTask.java | 18 ++++---
.../analysis/PluginDrivenSampleAnalysisTask.java | 3 +-
.../doris/statistics/model/ColumnStatistic.java | 3 +-
.../plans/commands/AnalyzeTableCommandTest.java | 2 +-
.../statistics/analysis/OlapAnalysisTaskTest.java | 57 +++++++++++++---------
.../agg_strategy/physical_agg_regulator.groovy | 8 ++-
.../stats/col_stats/hot_value_analyze_sync.groovy | 12 +++--
.../statistics/test_full_analyze_hot_value.groovy | 54 ++++++++++++--------
.../suites/statistics/test_hot_value.groovy | 26 +++++-----
11 files changed, 129 insertions(+), 87 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java
index 917511885e6..2b612683458 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeCommand.java
@@ -146,7 +146,7 @@ public abstract class AnalyzeCommand extends Command
implements ForwardWithSync
analyzeProperties.check();
}
if (analyzeProperties.hasCollectHotValue() && getAnalysisMethod() ==
AnalysisInfo.AnalysisMethod.SAMPLE) {
- throw new AnalysisException("Sample analyze always collects hot
value");
+ throw new AnalysisException("Sample analyze does not support WITH
HOT VALUE");
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java
index 5bbb96e1134..b5cc7b53aa7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/BaseAnalysisTask.java
@@ -142,6 +142,14 @@ public abstract class BaseAnalysisTask {
+ "FROM (SELECT ${colName}${lengthAssert} "
+ "FROM ${catalogName}.${dbName}.${tblName} ${index}) __lc_t";
+ // Hot values are intentionally NOT collected by the linear estimator. The
extra
+ // GROUP BY over the sampled column re-reads the whole sample and is the
dominant cost
+ // of this template, which is used for every single distribution / single
key column
+ // sample analyze. `hot_value` keeps its position in the result set and is
reported as
+ // NULL, which every reader treats as "not collected" (see
ColumnStatistic#hotValues),
+ // so the row layout stays identical to that of the other templates. Hot
values are
+ // still collected by DUJ1_ANALYZE_TEMPLATE, which derives them from the
grouped sample
+ // it already builds for the NDV estimate.
protected static final String LINEAR_ANALYZE_TEMPLATE = "WITH cte1 AS ("
+ "SELECT ${colName}${lengthAssert} "
+ "FROM ${catalogName}.${dbName}.${tblName} ${index}
${sampleHints} ${limit} ${preAggHint}), "
@@ -159,19 +167,10 @@ public abstract class BaseAnalysisTask {
+ "SUBSTRING(CAST(${min} AS STRING), 1, 1024) AS `min`, "
+ "SUBSTRING(CAST(${max} AS STRING), 1, 1024) AS `max`, "
+ "${dataSizeFunction} * ${scaleFactor} AS `data_size`, "
- + "NOW() FROM cte1), "
- + "cte3 AS ("
- + "SELECT IFNULL(GROUP_CONCAT(CONCAT("
- + "REPLACE(REPLACE(t.`column_key`, \":\", \"\\\\:\"),
\";\", \"\\\\;\"), "
- + "\" :\", ROUND(t.`count` / ${rowCount2}, 2)), \" ;\"),
'') "
- + "as `hot_value` "
- + "FROM ("
- + "SELECT ${subStringColName} as `hash_value`, "
- + "MAX(${colName}) as `column_key`, "
- + "COUNT(1) AS `count` "
- + "FROM cte1 WHERE ${colName} IS NOT NULL "
- + "GROUP BY `hash_value` ORDER BY `count` DESC LIMIT
${hotValueCollectCount}) t) "
- + "SELECT * FROM cte2 CROSS JOIN cte3";
+ + "NOW() AS `update_time`, "
+ + "null as `hot_value` "
+ + "FROM cte1) "
+ + "SELECT * FROM cte2";
protected static final String DUJ1_ANALYZE_TEMPLATE = "WITH cte1 AS ("
+ "SELECT MAX(t0.`col_value`) as `col_value`, COUNT(1) as `count`,
SUM(`len`) as `column_length` "
@@ -621,6 +620,12 @@ public abstract class BaseAnalysisTask {
protected abstract void deleteNotExistPartitionStats(AnalysisInfo jobInfo)
throws DdlException;
+ /**
+ * Whether this task may collect hot values. It only decides which
template a full analyze uses
+ * (with or without the hot value CTE). A sample analyze collects hot
values only when the
+ * estimator it picked provides them, i.e. {@code DUJ1_ANALYZE_TEMPLATE};
the linear estimator
+ * never does, even though a sample job sets {@code collectHotValue} to
true.
+ */
protected boolean shouldCollectHotValue() {
return Boolean.TRUE.equals(info.collectHotValue);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java
index 606a2424d10..b1c3c63b675 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java
@@ -124,19 +124,20 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
Map<String, String> params = buildSqlParams();
params.put("min", StatisticsUtil.quote(min));
params.put("max", StatisticsUtil.quote(max));
- params.put("hotValueCollectCount",
String.valueOf(SessionVariable.getHotValueCollectCount()));
long tableRowCount = info.indexId == -1
? tbl.getRowCount()
: ((OlapTable) tbl).getRowCountForIndex(info.indexId, false);
SampleCollectInfo collectInfo = getSampleCollectInfo(tableRowCount);
getSampleParams(params, tableRowCount, collectInfo);
- StringSubstitutor stringSubstitutor = new StringSubstitutor(params);
- String sql;
+ String template = LINEAR_ANALYZE_TEMPLATE;
if (collectInfo.algorithm == AnalyzeSampleAlgorithm.DUJ1) {
- sql = stringSubstitutor.replace(DUJ1_ANALYZE_TEMPLATE);
- } else {
- sql = stringSubstitutor.replace(LINEAR_ANALYZE_TEMPLATE);
+ // DUJ1 is the only sample algorithm that collects hot values, and
it derives them
+ // from the grouped sample it already builds for the NDV estimate,
so only it needs
+ // the top-N count.
+ params.put("hotValueCollectCount",
String.valueOf(SessionVariable.getHotValueCollectCount()));
+ template = DUJ1_ANALYZE_TEMPLATE;
}
+ String sql = new StringSubstitutor(params).replace(template);
LOG.info("Analyze param: algorithm {}, partitionColumnTooMany {},
keyColumnTooMany {}",
collectInfo.algorithm, partitionColumnSampleTooManyRows,
keyColumnSampleTooManyRows);
LOG.debug(sql);
@@ -312,7 +313,6 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
params.put("rowCount", String.valueOf(tableRowCount));
params.put("type", col.getType().toString());
params.put("limit", "");
- params.put("subStringColName", getStringTypeColName(col));
// For agg table and mor unique table, set PREAGGOPEN preAggHint.
if (((OlapTable) tbl).getKeysType().equals(KeysType.AGG_KEYS)
@@ -362,11 +362,13 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
protected void setSampleParamsByAlgorithm(Map<String, String> params, long
tableRowCount,
AnalyzeSampleAlgorithm algorithm) {
if (algorithm == AnalyzeSampleAlgorithm.DUJ1) {
+ // DUJ1 is the only sample algorithm that collects hot values: it
derives them from
+ // the grouped sample it already builds for the NDV estimate below.
+ params.put("subStringColName", getStringTypeColName(col));
params.put("ndvFunction",
getNdvFunction(String.valueOf(tableRowCount)));
params.put("dataSizeFunction", getDataSizeFunction(col, true));
params.put("rowCount2", "(SELECT SUM(`count`) FROM cte1 WHERE
`col_value` IS NOT NULL)");
} else {
- params.put("rowCount2", "(SELECT COUNT(1) FROM cte1 WHERE
${colName} IS NOT NULL)");
// For single unique key, use count as ndv.
if (isSingleUniqueKey()) {
params.put("ndvFunction", String.valueOf(tableRowCount));
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java
index 91690df1eb0..135edbfcc5a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/PluginDrivenSampleAnalysisTask.java
@@ -63,7 +63,6 @@ public class PluginDrivenSampleAnalysisTask extends
ExternalAnalysisTask {
params.put("dataSizeFunction", getDataSizeFunction(col, false));
Pair<Double, Long> sampleInfo = getSampleInfo();
params.put("scaleFactor", String.valueOf(sampleInfo.first));
- params.put("hotValueCollectCount",
String.valueOf(SessionVariable.getHotValueCollectCount()));
if (LOG.isDebugEnabled()) {
LOG.debug("Will do sample collection for column {}",
col.getName());
}
@@ -91,10 +90,10 @@ public class PluginDrivenSampleAnalysisTask extends
ExternalAnalysisTask {
sb.append(LINEAR_ANALYZE_TEMPLATE);
params.put("ndvFunction", "ROUND(NDV(${colName}) *
${scaleFactor})");
params.put("rowCount", "ROUND(COUNT(1) * ${scaleFactor})");
- params.put("rowCount2", "(SELECT COUNT(1) FROM cte1 WHERE
${colName} IS NOT NULL)");
} else {
sb.append(DUJ1_ANALYZE_TEMPLATE);
params.put("subStringColName", getStringTypeColName(col));
+ params.put("hotValueCollectCount",
String.valueOf(SessionVariable.getHotValueCollectCount()));
params.put("dataSizeFunction", getDataSizeFunction(col, true));
params.put("ndvFunction", getNdvFunction("ROUND(SUM(t1.count) *
${scaleFactor})"));
params.put("rowCount", "ROUND(SUM(t1.count) * ${scaleFactor})");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java
index 1af96763d85..66214b514b3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/model/ColumnStatistic.java
@@ -101,7 +101,8 @@ public class ColumnStatistic {
public final String updatedTime;
/**
- * hotValues == null: not collected (e.g. full analyze). Do NOT use for
optimization.
+ * hotValues == null: not collected (e.g. full analyze, or a sample
analyze that used
+ * the linear estimator). Do NOT use for optimization.
* hotValues != null && hotValues.isEmpty(): collected but no hot values
(not skewed).
* hotValues != null && !hotValues.isEmpty(): collected with hot values.
*/
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java
index ca1c3243ce9..369644bf193 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommandTest.java
@@ -157,7 +157,7 @@ public class AnalyzeTableCommandTest {
"ANALYZE TABLE test_db.test_tbl WITH SAMPLE ROWS 100 WITH HOT
VALUE");
AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
() -> command.validate(ctx));
- Assertions.assertEquals("Sample analyze always collects hot value",
exception.getDetailMessage());
+ Assertions.assertEquals("Sample analyze does not support WITH HOT
VALUE", exception.getDetailMessage());
}
private Map<String, String> defaultAnalyzeProperties() {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java
b/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java
index b6976a5d41f..a8e5f74ea5a 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/statistics/analysis/OlapAnalysisTaskTest.java
@@ -162,13 +162,8 @@ public class OlapAnalysisTaskTest {
+ "ROUND(SUM(CASE WHEN `null` IS NULL THEN 1 ELSE 0 END) *
${scaleFactor}) AS `null_count`, "
+ "SUBSTRING(CAST('1' AS STRING), 1, 1024) AS `min`, "
+ "SUBSTRING(CAST('2' AS STRING), 1, 1024) AS `max`, "
- + "COUNT(1) * 4 * ${scaleFactor} AS `data_size`, NOW()
FROM cte1), "
- + "cte3 AS (SELECT
IFNULL(GROUP_CONCAT(CONCAT(REPLACE(REPLACE(t.`column_key`, "
- + "\":\", \"\\\\:\"), \";\", \"\\\\;\"), \" :\",
ROUND(t.`count` / ${rowCount2}, 2)), "
- + "\" ;\"), '') as `hot_value` FROM (SELECT
${subStringColName} as `hash_value`, "
- + "MAX(`null`) as `column_key`, COUNT(1) AS `count` FROM
cte1 "
- + "WHERE `null` IS NOT NULL GROUP BY `hash_value` ORDER BY
`count` DESC LIMIT 10) t) "
- + "SELECT * FROM cte2 CROSS JOIN cte3", sql);
+ + "COUNT(1) * 4 * ${scaleFactor} AS `data_size`, NOW() AS
`update_time`, "
+ + "null as `hot_value` FROM cte1) SELECT * FROM cte2",
sql);
return null;
}).when(olapAnalysisTask).runQuery(ArgumentMatchers.anyString());
@@ -322,8 +317,9 @@ public class OlapAnalysisTaskTest {
new
OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.FULL, null));
Assertions.assertEquals("1", params.get("scaleFactor"));
Assertions.assertEquals("", params.get("sampleHints"));
- Assertions.assertEquals("(SELECT COUNT(1) FROM cte1 WHERE ${colName}
IS NOT NULL)",
- params.get("rowCount2"));
+ // The linear estimator does not collect hot values, so it fills no
hot value param.
+ Assertions.assertNull(params.get("rowCount2"));
+ Assertions.assertNull(params.get("subStringColName"));
Assertions.assertEquals("ROUND(NDV(${colName}) * ${scaleFactor})",
params.get("ndvFunction"));
Assertions.assertNull(params.get("preAggHint"));
Assertions.assertEquals("COUNT(1)", params.get("rowCount"));
@@ -334,8 +330,8 @@ public class OlapAnalysisTaskTest {
new
OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.LINEAR,
Pair.of(Lists.newArrayList(1L, 2L), 100L)));
Assertions.assertEquals("TABLET(1, 2)", params.get("sampleHints"));
- Assertions.assertEquals("(SELECT COUNT(1) FROM cte1 WHERE ${colName}
IS NOT NULL)",
- params.get("rowCount2"));
+ Assertions.assertNull(params.get("rowCount2"));
+ Assertions.assertNull(params.get("subStringColName"));
Assertions.assertEquals("ROUND(NDV(${colName}) * ${scaleFactor})",
params.get("ndvFunction"));
params.clear();
@@ -348,6 +344,7 @@ public class OlapAnalysisTaskTest {
Assertions.assertEquals("TABLET(1, 2)", params.get("sampleHints"));
Assertions.assertEquals("(SELECT SUM(`count`) FROM cte1 WHERE
`col_value` IS NOT NULL)",
params.get("rowCount2"));
+ Assertions.assertEquals("${colName}", params.get("subStringColName"));
Assertions.assertTrue(params.get("ndvFunction").contains("`t1`.`col_value`"),
"DUJ1 ndvFunction must reference cte1 output, got: " +
params.get("ndvFunction"));
params.clear();
@@ -983,7 +980,7 @@ public class OlapAnalysisTaskTest {
}
@Test
- public void testDoSampleAlwaysCollectsHotValue() throws Exception {
+ public void testDoSampleHotValueByAlgorithm() throws Exception {
CatalogIf catalogIf = Mockito.mock(CatalogIf.class);
DatabaseIf databaseIf = Mockito.mock(DatabaseIf.class);
OlapTable tableIf = Mockito.mock(OlapTable.class);
@@ -999,16 +996,6 @@ public class OlapAnalysisTaskTest {
Mockito.doReturn(new ResultRow(Lists.newArrayList("1",
"2"))).when(task).collectMinMax();
Mockito.doNothing().when(task).getSampleParams(ArgumentMatchers.any(),
ArgumentMatchers.anyLong(),
ArgumentMatchers.any());
- Mockito.doAnswer(invocation -> {
- String sql = invocation.getArgument(0);
- Assertions.assertTrue(sql.contains("as `hot_value`"), sql);
- Assertions.assertTrue(sql.contains("cte3"), sql);
- Assertions.assertTrue(sql.contains("CROSS JOIN cte3"), sql);
- Assertions.assertTrue(sql.contains("LIMIT 10"), sql);
- Assertions.assertFalse(sql.contains("null as `hot_value`"), sql);
- return null;
- }).when(task).runQuery(Mockito.anyString());
-
task.col = new Column("testCol",
Type.fromPrimitiveType(PrimitiveType.INT),
true, null, null, null);
task.tbl = tableIf;
@@ -1021,10 +1008,36 @@ public class OlapAnalysisTaskTest {
task.db = databaseIf;
task.tableSample = new TableSample(false, 100L);
+ // The linear estimator does not collect hot values: the extra GROUP
BY over the sampled
+ // column is the dominant cost of LINEAR_ANALYZE_TEMPLATE.
+ Mockito.doAnswer(invocation -> {
+ String sql = invocation.getArgument(0);
+ Assertions.assertTrue(sql.contains("null as `hot_value`"), sql);
+ Assertions.assertFalse(sql.contains("cte3"), sql);
+ Assertions.assertFalse(sql.contains("CROSS JOIN cte3"), sql);
+ Assertions.assertFalse(sql.contains("GROUP BY `hash_value`"), sql);
+ // The linear template carries no hot value placeholder at all.
Which params each
+ // algorithm fills is pinned by testGetSampleParams.
+ Assertions.assertFalse(sql.contains("${hotValueCollectCount}"),
sql);
+ Assertions.assertFalse(sql.contains("${subStringColName}"), sql);
+ Assertions.assertFalse(sql.contains("${rowCount2}"), sql);
+ return null;
+ }).when(task).runQuery(Mockito.anyString());
Mockito.doReturn(new
OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.LINEAR,
Pair.of(Lists.newArrayList(1L, 2L), 100L)))
.when(task).getSampleCollectInfo(Mockito.anyLong());
task.doSample();
+
+ // DUJ1 still collects hot values: it derives them from the grouped
sample it already
+ // builds for the NDV estimate.
+ Mockito.doAnswer(invocation -> {
+ String sql = invocation.getArgument(0);
+ Assertions.assertTrue(sql.contains("as `hot_value`"), sql);
+ Assertions.assertTrue(sql.contains("CROSS JOIN cte3"), sql);
+ Assertions.assertTrue(sql.contains("LIMIT 10"), sql);
+ Assertions.assertFalse(sql.contains("null as `hot_value`"), sql);
+ return null;
+ }).when(task).runQuery(Mockito.anyString());
Mockito.doReturn(new
OlapAnalysisTask.SampleCollectInfo(AnalyzeSampleAlgorithm.DUJ1,
Pair.of(Lists.newArrayList(1L, 2L), 100L)))
.when(task).getSampleCollectInfo(Mockito.anyLong());
diff --git
a/regression-test/suites/nereids_rules_p0/agg_strategy/physical_agg_regulator.groovy
b/regression-test/suites/nereids_rules_p0/agg_strategy/physical_agg_regulator.groovy
index 4620e7c931e..51d0cb67471 100644
---
a/regression-test/suites/nereids_rules_p0/agg_strategy/physical_agg_regulator.groovy
+++
b/regression-test/suites/nereids_rules_p0/agg_strategy/physical_agg_regulator.groovy
@@ -23,8 +23,11 @@ suite("physical_agg_regulator") {
set enable_parallel_result_sink=false;
set be_number_for_test=1;
"""
+ // A sample request larger than the table falls back to a full scan with
the linear estimator,
+ // which does not collect hot value. This case needs hot value to exercise
the skew-aware
+ // aggregation shape, so ask for it explicitly instead of relying on the
sample path.
multi_sql """
- analyze table t1025_skew5000 with sample rows 7000 with sync;
+ analyze table t1025_skew5000 with sync with hot value;
--drop cached stats t1025;
"""
// 这个预期应该使用cte之后any
@@ -32,8 +35,9 @@ suite("physical_agg_regulator") {
select count(distinct a_1), count(distinct b_5) from t1025_skew5000 group
by d_1025;"""
+ // Same reason as above: this table's plan expectations were built on
collected hot values.
multi_sql """
- analyze table t1025 with sample rows 4000 with sync;
+ analyze table t1025 with sync with hot value;
--drop cached stats t1025;
"""
qt_not_skew """explain shape plan
diff --git
a/regression-test/suites/query_p0/stats/col_stats/hot_value_analyze_sync.groovy
b/regression-test/suites/query_p0/stats/col_stats/hot_value_analyze_sync.groovy
index 39c43134ab7..8ebc437890e 100644
---
a/regression-test/suites/query_p0/stats/col_stats/hot_value_analyze_sync.groovy
+++
b/regression-test/suites/query_p0/stats/col_stats/hot_value_analyze_sync.groovy
@@ -49,10 +49,16 @@ suite('hot_value_analyze_sync') {
String realDb = context.config.getDbNameByFile(context.file)
sql """
drop table if exists t1025;
- create table t1025(a_1 int, b_5 int, c_10 int, d_1025 int) distributed by
hash(c_10) properties('replication_num'='1');
- insert into t1025 select 1, number%5 , number%10, number from
numbers("number"="100");
+ create table t1025(a_1 int, b_5 int, c_10 int, d_1025 int) distributed by
hash(c_10) buckets 1 properties('replication_num'='1');
+ insert into t1025 select 1, number%5 , number%10, number from
numbers("number"="10000");
"""
- wait_row_count_reported(realDb, "t1025", 0, 4, "100")
+ wait_row_count_reported(realDb, "t1025", 0, 4, "10000")
+ // One bucket, so sample analyze is guaranteed to sample instead of
falling back to a full scan:
+ // getSampleCollectInfo falls back when the sample request exceeds the
table, or when the picked
+ // tablets hold fewer rows than requested, and both fallbacks use the
linear estimator for every
+ // column, which does not collect hot value (`null as hot_value`). d_1025
is not the distribution
+ // column, so a real sample analyzes it with the DUJ1 estimator, which
does collect hot value -
+ // the property this suite exists to check.
sql "analyze table t1025 with sample rows 4000 with sync;"
def result = sql "show column cached stats t1025"
log.info(result.toString())
diff --git
a/regression-test/suites/statistics/test_full_analyze_hot_value.groovy
b/regression-test/suites/statistics/test_full_analyze_hot_value.groovy
index ad7df099e8d..c709f20ed42 100644
--- a/regression-test/suites/statistics/test_full_analyze_hot_value.groovy
+++ b/regression-test/suites/statistics/test_full_analyze_hot_value.groovy
@@ -133,42 +133,54 @@ suite("test_full_analyze_hot_value") {
// All 100 rows have the same value " : ;a", so it should appear with
ratio 1.0
assertEquals("' : ;a':1.0", result[0][17])
- // Test 5: Sample analyze collects hot_value by default.
+ // Test 5: A sample analyze whose sample request exceeds the table row
count falls back to a
+ // full scan and uses the linear estimator, which does not collect
hot_value.
sql """drop stats full_hot_skew"""
sql """analyze table full_hot_skew with sample rows 400 with sync"""
result = sql """show column stats full_hot_skew(value1)"""
logger.info("Test5 result: " + result)
assertEquals(1, result.size())
- assertTrue(result[0][17].contains(":"), "Sample analyze should also
collect hot_value")
- hotValues = result[0][17].split(";")
- assertEquals(2, hotValues.length)
- assertTrue(hotValues[0].trim() == "'1':0.5" || hotValues[0].trim() ==
"'0':0.5")
- assertTrue(hotValues[1].trim() == "'1':0.5" || hotValues[1].trim() ==
"'0':0.5")
+ assertEquals("null", String.valueOf(result[0][17]))
- // Test 6: Sample analyze rejects WITH HOT VALUE because it always
collects hot_value.
+ // Test 6: Sample analyze rejects WITH HOT VALUE because it cannot honor
the flag: inside a
+ // sample analyze hot values only come from the DUJ1 estimator.
test {
sql """analyze table full_hot_skew with sample rows 400 with sync with
hot value"""
- exception "Sample analyze always collects hot value"
+ exception "Sample analyze does not support WITH HOT VALUE"
}
- // Test 7: Explicit full analyze produces same hot_value as sample analyze
for same data.
- sql """drop stats full_hot_skew"""
- sql """analyze table full_hot_skew with sync with hot value"""
- def fullResult = sql """show column stats full_hot_skew(value1)"""
+ // Test 7: Explicit full analyze produces the same hot_value as a sample
analyze that really
+ // samples. The column value is uniform, so the sampled rows always carry
the same
+ // distribution as the whole table, and the table is larger than the
requested sample rows,
+ // so sample analyze uses the DUJ1 estimator instead of falling back to a
full scan.
+ sql """drop table if exists full_hot_sample_agree"""
+ sql """CREATE TABLE full_hot_sample_agree (
+ key1 int NULL,
+ value1 varchar(50) NULL
+ )ENGINE=OLAP
+ DUPLICATE KEY(`key1`)
+ COMMENT "OLAP"
+ DISTRIBUTED BY HASH(`key1`) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1"
+ )
+ """
+ sql """insert into full_hot_sample_agree select number, " : ;a" from
numbers("number"="10000")"""
+ wait_row_count_reported("test_full_analyze_hot_value",
"full_hot_sample_agree", 0, 4, "10000")
+
+ sql """analyze table full_hot_sample_agree with sync with hot value"""
+ def fullResult = sql """show column stats full_hot_sample_agree(value1)"""
logger.info("Test7 full result: " + fullResult)
assertEquals(1, fullResult.size())
- assertTrue(fullResult[0][17].contains(":"))
- def fullParts = fullResult[0][17].split(";").collect { it.trim() }.sort()
+ assertEquals("' : ;a':1.0", fullResult[0][17])
- sql """drop stats full_hot_skew"""
- sql """analyze table full_hot_skew with sample rows 40000 with sync"""
- def sampleResult = sql """show column stats full_hot_skew(value1)"""
+ sql """drop stats full_hot_sample_agree"""
+ sql """analyze table full_hot_sample_agree with sample rows 400 with
sync"""
+ def sampleResult = sql """show column stats
full_hot_sample_agree(value1)"""
logger.info("Test7 sample result: " + sampleResult)
assertEquals(1, sampleResult.size())
- assertTrue(sampleResult[0][17].contains(":"))
- // Both full and sample should produce the same hot_value entries (order
may differ)
- def sampleParts = sampleResult[0][17].split(";").collect { it.trim()
}.sort()
- assertEquals(fullParts, sampleParts)
+ // Both full and sample should produce the same hot_value entry.
+ assertEquals(fullResult[0][17], sampleResult[0][17])
// Test 8: Explicit full analyze on empty table should produce an empty
hot_value string.
sql """drop table if exists full_hot_empty"""
diff --git a/regression-test/suites/statistics/test_hot_value.groovy
b/regression-test/suites/statistics/test_hot_value.groovy
index 9759adb8ec1..ad95ed31ac7 100644
--- a/regression-test/suites/statistics/test_hot_value.groovy
+++ b/regression-test/suites/statistics/test_hot_value.groovy
@@ -117,6 +117,9 @@ suite("test_hot_value") {
logger.info("result " + result)
assertEquals(1, result.size())
assertEquals("10000.0", result[0][2])
+ // key1 is the single distribution column, so a real sample analyzes it
with the linear
+ // estimator, which does not collect hot_value.
+ assertEquals("null", String.valueOf(result[0][17]))
result = sql """show column stats test1(value1)"""
logger.info("1. result " + result)
assertEquals(1, result.size())
@@ -136,29 +139,24 @@ suite("test_hot_value") {
assertTrue(hotValues[1] == "'1':0.5" || hotValues[1] == "'0':0.5")
sql """drop stats test1"""
+ // The requested sample rows are larger than the table, so sample analyze
falls back to a
+ // full scan with the linear estimator, which does not collect hot_value.
sql """analyze table test1 with sample rows 40000 with sync"""
result = sql """show column stats test1(key1)"""
logger.info("result " + result)
assertEquals(1, result.size())
assertEquals("10000.0", result[0][2])
+ assertEquals("null", String.valueOf(result[0][17]))
result = sql """show column stats test1(value1)"""
logger.info("2. result " + result)
- logger.info("result.size()" + result.size())
assertEquals(1, result.size())
- logger.info("result[0][2] " + result[0][2])
assertEquals("10000.0", result[0][2])
- hotValues = result[0][17].split(";")
- assertEquals(2, hotValues.length)
- assertTrue(hotValues[0] == "'1':0.5" || hotValues[0] == "'0':0.5")
- assertTrue(hotValues[1] == "'1':0.5" || hotValues[1] == "'0':0.5")
+ assertEquals("null", String.valueOf(result[0][17]))
result = sql """show column cached stats test1(value1)"""
logger.info("result " + result)
assertEquals(1, result.size())
assertEquals("10000.0", result[0][2])
- hotValues = result[0][17].split(";")
- assertEquals(2, hotValues.length)
- assertTrue(hotValues[0] == "'1':0.5" || hotValues[0] == "'0':0.5")
- assertTrue(hotValues[1] == "'1':0.5" || hotValues[1] == "'0':0.5")
+ assertEquals("null", String.valueOf(result[0][17]))
sql """alter table test1 modify column value1 set stats
('row_count'='100.0', 'ndv'='5.0', 'num_nulls'='0.0', 'data_size'='34.0',
'min_value'='AFRICA', 'max_value'='MIDDLE EAST', 'hot_values'='aaa :0.8');"""
result = sql """show column stats test1(value1)"""
@@ -203,13 +201,15 @@ suite("test_hot_value") {
assertEquals("' : ;a':1.0", result[0][17])
sql """drop stats test2"""
+ // As for test1 above, the requested sample rows are larger than the
table, so sample analyze
+ // falls back to a full scan with the linear estimator, which does not
collect hot_value.
sql """analyze table test2 with sample rows 4000000 with sync"""
result = sql """show column stats test2(value1)"""
assertEquals(1, result.size())
- assertEquals("' : ;a':1.0", result[0][17])
- result = sql """show column cached stats test2(value1)"""
+ assertEquals("null", String.valueOf(result[0][17]))
+ result = sql """show column cached stats test2(value1)"""
assertEquals(1, result.size())
- assertEquals("' : ;a':1.0", result[0][17])
+ assertEquals("null", String.valueOf(result[0][17]))
sql """set enable_dphyp_optimizer=true"""
sql """
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
