This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 326c1363497 [fix](cache) Share cache eligibility for nondeterministic
expressions (#68672)
326c1363497 is described below
commit 326c13634971d8cf641f0be6f56928c327d3f7d2
Author: HappenLee <[email protected]>
AuthorDate: Sat Oct 10 14:46:57 2026 +0800
[fix](cache) Share cache eligibility for nondeterministic expressions
(#68672)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Condition cache can incorrectly reuse false ranges for expressions whose
results vary across executions, and for virtual-column predicates whose
definitions are omitted from the digest. For example, running `abs(k +
1) + abs(k + 1) = 0` before `abs(k - 9999) + abs(k - 9999) = 0` can
incorrectly omit `k = 9999`.
Share expression eligibility with query cache, inspect scan predicates,
virtual columns and runtime-filter probes (including implicit casts),
and pass scan eligibility to BE before computing the cache digest.
Recognize scalar and ARRAY/MAP/STRUCT TIME-to-date conversions while
preserving supported NULL folding. VirtualSlotRef returns the
zero-digest sentinel to disable reuse when its definition is absent from
the inherited digest.
Add per-query ConditionCacheLookup counters for hits and misses.
Regression coverage identifies individual execution profiles, checks
that unsafe scans perform zero lookups, and retains positive checks for
deterministic cache hits.
Rebased onto master `a53b182f810be2b3731c7dfdf6f40ff5a0ad63bd`. Preserve
master's #68438 shuffle identity and constant-seed validation while
allowing NULL-only folding, and retain master's grouped non-blocking
runtime-filter test alongside cache-eligibility coverage. All five
original commits are retained.
### Release note
Prevent incorrect cached results for volatile predicates,
query-date-dependent TIME casts, and virtual-column predicates. Preserve
supported NULL folding for TIME casts and shuffle calls. Expose
ConditionCacheLookup in OLAP scan profiles.
Rolling upgrade: upgrade BE before FE. The optional plan flag defaults
false, so a new BE disables condition cache for old FE plans; an old BE
cannot enforce the new eligibility flag.
### Check List (For Author)
- Test:
- [x] Unit Test: 39 targeted FE tests passed after rebase using
`FE_UT_PARALLEL=48 bash run-fe-ut.sh --run
'org.apache.doris.nereids.util.CacheExpressionTest,org.apache.doris.planner.ConditionCacheEligibilityTest,org.apache.doris.nereids.glue.translator.RuntimeFilterTranslatorBucketPruneTest,org.apache.doris.nereids.trees.expressions.functions.scalar.UniqueFunctionTest'`
(8 + 6 + 7 + 18).
- [x] Manual test: FE Checkstyle passed with zero violations; repository
check-format.sh with clang-format 16.0.6 and BE build hygiene passed.
Original patch comparison confirms identical added/removed lines for 23
files; ArrayShuffle adapts to the equivalent functionality now in
master.
- [ ] Regression test: BE tests and regression suites were not rerun
locally for this rebase; existing coverage is retained and run buildall
will verify integration.
- Behavior changed:
- [x] Yes. Unsafe scans bypass condition cache; deterministic scans
retain eligibility. No additional behavior change introduced by rebase.
- Does this need documentation?
- [x] No. Correctness fix using existing cache switches; upgrade
considerations are stated above.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
be/src/exec/operator/olap_scan_operator.cpp | 2 +
be/src/exec/operator/olap_scan_operator.h | 1 +
be/src/exec/operator/scan_operator.cpp | 4 +-
be/src/exec/operator/scan_operator.h | 1 +
be/src/exec/scan/olap_scanner.cpp | 2 +
be/src/exprs/virtual_slot_ref.h | 2 +
be/src/storage/olap_common.h | 2 +
be/src/storage/segment/segment_iterator.cpp | 1 +
be/test/exprs/virtual_slot_ref_test.cpp | 23 +++
.../segment_iterator_candidate_pushdown_test.cpp | 44 +++++
.../glue/translator/PhysicalPlanTranslator.java | 43 ++---
.../glue/translator/RuntimeFilterTranslator.java | 16 ++
.../rules/analysis/FoldConstantForSqlCache.java | 15 +-
.../doris/nereids/trees/expressions/Cast.java | 42 +++++
.../expressions/functions/scalar/ArrayShuffle.java | 8 +
.../apache/doris/nereids/util/ExpressionUtils.java | 10 ++
.../java/org/apache/doris/planner/PlanNode.java | 10 ++
.../RuntimeFilterTranslatorBucketPruneTest.java | 38 ++++
.../doris/nereids/util/CacheExpressionTest.java | 199 +++++++++++++++++++++
.../planner/ConditionCacheEligibilityTest.java | 173 ++++++++++++++++++
gensrc/thrift/PlanNodes.thrift | 3 +
.../query_p0/cache/condition_cache_eligibility.out | 16 ++
.../correctness/test_cast_time_to_datetime.groovy | 9 +-
.../cache/condition_cache_eligibility.groovy | 176 ++++++++++++++++++
24 files changed, 801 insertions(+), 39 deletions(-)
diff --git a/be/src/exec/operator/olap_scan_operator.cpp
b/be/src/exec/operator/olap_scan_operator.cpp
index 768eedf154c..c84909ff8ea 100644
--- a/be/src/exec/operator/olap_scan_operator.cpp
+++ b/be/src/exec/operator/olap_scan_operator.cpp
@@ -126,6 +126,8 @@ PushDownType
OlapScanLocalState::_should_push_down_binary_predicate(
Status OlapScanLocalState::_init_profile() {
RETURN_IF_ERROR(ScanLocalState<OlapScanLocalState>::_init_profile());
+ _condition_cache_lookup_counter =
+ ADD_COUNTER(_scanner_profile, "ConditionCacheLookup", TUnit::UNIT);
// Rows read from storage.
// Include the rows read from doris page cache.
_scan_rows = ADD_COUNTER(custom_profile(), "ScanRows", TUnit::UNIT);
diff --git a/be/src/exec/operator/olap_scan_operator.h
b/be/src/exec/operator/olap_scan_operator.h
index a76783386eb..b7630cb5545 100644
--- a/be/src/exec/operator/olap_scan_operator.h
+++ b/be/src/exec/operator/olap_scan_operator.h
@@ -162,6 +162,7 @@ private:
snii::SniiPrxRuntimeProfileCounters _snii_prx_profile_counters;
snii::SniiPhraseRuntimeProfileCounters _snii_phrase_profile_counters;
+ RuntimeProfile::Counter* _condition_cache_lookup_counter = nullptr;
RuntimeProfile::Counter* _tablet_counter = nullptr;
RuntimeProfile::Counter* _buckets_pruned_by_rf_counter = nullptr;
RuntimeProfile::Counter* _key_range_counter = nullptr;
diff --git a/be/src/exec/operator/scan_operator.cpp
b/be/src/exec/operator/scan_operator.cpp
index b2f960ec5ff..0ed2d762953 100644
--- a/be/src/exec/operator/scan_operator.cpp
+++ b/be/src/exec/operator/scan_operator.cpp
@@ -238,7 +238,8 @@ Status ScanLocalState<Derived>::open(RuntimeState* state) {
}
// Disable condition cache in topn filter valid. TODO:: Try to support the
topn filter in condition cache
- if (state->query_options().condition_cache_digest &&
p._topn_filter_source_node_ids.empty()) {
+ if (p._enable_condition_cache &&
state->query_options().condition_cache_digest &&
+ p._topn_filter_source_node_ids.empty()) {
_condition_cache_digest =
state->query_options().condition_cache_digest;
for (auto& conjunct : _conjuncts) {
_condition_cache_digest =
conjunct->get_digest(_condition_cache_digest);
@@ -1209,6 +1210,7 @@ ScanOperatorX<LocalStateType>::ScanOperatorX(ObjectPool*
pool, const TPlanNode&
int operator_id, const
DescriptorTbl& descs,
int parallel_tasks)
: OperatorX<LocalStateType>(pool, tnode, operator_id, descs),
+ _enable_condition_cache(tnode.enable_condition_cache),
_runtime_filter_descs(tnode.runtime_filters),
_parallel_tasks(parallel_tasks) {
if (tnode.__isset.push_down_count) {
diff --git a/be/src/exec/operator/scan_operator.h
b/be/src/exec/operator/scan_operator.h
index bf112db7bfc..7c5e5eb11b9 100644
--- a/be/src/exec/operator/scan_operator.h
+++ b/be/src/exec/operator/scan_operator.h
@@ -445,6 +445,7 @@ protected:
// If the query like select * from table limit 10; then the query should
run in
// single scanner to avoid too many scanners which will cause lots of
useless read.
bool _should_run_serial = false;
+ bool _enable_condition_cache = false;
VExprContextSPtrs _common_expr_ctxs_push_down;
diff --git a/be/src/exec/scan/olap_scanner.cpp
b/be/src/exec/scan/olap_scanner.cpp
index 9955ae58147..1762ee1c154 100644
--- a/be/src/exec/scan/olap_scanner.cpp
+++ b/be/src/exec/scan/olap_scanner.cpp
@@ -923,6 +923,8 @@ void OlapScanner::_collect_profile_before_close() {
stats.output_index_result_column_timer);
COUNTER_UPDATE(local_state->_filtered_segment_counter,
stats.filtered_segment_number);
COUNTER_UPDATE(local_state->_total_segment_counter,
stats.total_segment_number);
+ COUNTER_UPDATE(local_state->_condition_cache_lookup_counter,
+ stats.condition_cache_lookup_count);
COUNTER_UPDATE(local_state->_condition_cache_hit_counter,
stats.condition_cache_hit_seg_nums);
COUNTER_UPDATE(local_state->_condition_cache_filtered_rows_counter,
stats.condition_cache_filtered_rows);
diff --git a/be/src/exprs/virtual_slot_ref.h b/be/src/exprs/virtual_slot_ref.h
index 5994c6d9862..9e9669575d8 100644
--- a/be/src/exprs/virtual_slot_ref.h
+++ b/be/src/exprs/virtual_slot_ref.h
@@ -36,6 +36,8 @@ public:
std::string expr_label() override;
std::string debug_string() const override;
bool is_constant() const override { return false; }
+ // The virtual column definition is stored outside _children and is not
part of the digest.
+ uint64_t get_digest(uint64_t seed) const override { return 0; }
int column_id() const { return _column_id; }
int slot_id() const { return _slot_id; }
bool equals(const VExpr& other) override;
diff --git a/be/src/storage/olap_common.h b/be/src/storage/olap_common.h
index 30899ff271b..bf9908117d2 100644
--- a/be/src/storage/olap_common.h
+++ b/be/src/storage/olap_common.h
@@ -379,6 +379,8 @@ struct OlapReaderStatistics {
int64_t output_index_result_column_timer = 0;
// number of segment filtered by column stat when creating seg iterator
int64_t filtered_segment_number = 0;
+ // number of condition cache lookups, including misses
+ int64_t condition_cache_lookup_count = 0;
// number of segment with condition cache hit
int64_t condition_cache_hit_seg_nums = 0;
// number of rows filtered by condition cache hit
diff --git a/be/src/storage/segment/segment_iterator.cpp
b/be/src/storage/segment/segment_iterator.cpp
index dabab3bfdaf..488677badef 100644
--- a/be/src/storage/segment/segment_iterator.cpp
+++ b/be/src/storage/segment/segment_iterator.cpp
@@ -163,6 +163,7 @@ void SegmentIterator::_init_row_bitmap_by_condition_cache()
{
// Increment search count when digest != 0
DorisMetrics::instance()->condition_cache_search_count->increment(1);
+ _opts.stats->condition_cache_lookup_count++;
ConditionCacheHandle handle;
_find_condition_cache = condition_cache->lookup(cache_key,
&handle);
diff --git a/be/test/exprs/virtual_slot_ref_test.cpp
b/be/test/exprs/virtual_slot_ref_test.cpp
index ce1961ba518..c7764faeb82 100644
--- a/be/test/exprs/virtual_slot_ref_test.cpp
+++ b/be/test/exprs/virtual_slot_ref_test.cpp
@@ -26,6 +26,8 @@
#include "common/object_pool.h"
#include "core/data_type/data_type_string.h"
+#include "exec/scan/scanner.h"
+#include "exprs/vcast_expr.h"
#include "exprs/vexpr_context.h"
#include "runtime/descriptors.h"
#include "testutil/mock/mock_runtime_state.h"
@@ -107,6 +109,27 @@ TEST_F(VirtualSlotRefTest, ConstructorWithSlotDescriptor) {
EXPECT_EQ(virtual_slot_ref.column_id(), -1); // Should be -1 initially
}
+TEST_F(VirtualSlotRefTest, ConditionCacheDigestIsDisabled) {
+ const VExprSPtr ref =
+ VirtualSlotRef::create_shared(create_virtual_slot_ref_node(1,
"virtual_column"));
+ EXPECT_EQ(ref->get_digest(12345), 0);
+
+ const VirtualSlotRef descriptor_ref(create_slot_descriptor(2,
"virtual_column"));
+ EXPECT_EQ(descriptor_ref.get_digest(12345), 0);
+}
+
+TEST_F(VirtualSlotRefTest,
ConditionCacheDigestPropagatesThroughParentAndConjuncts) {
+ auto cast_node = create_virtual_slot_ref_node(1);
+ cast_node.__set_node_type(TExprNodeType::CAST_EXPR);
+ cast_node.__set_num_children(1);
+ auto cast_expr = VCastExpr::create_shared(cast_node);
+
cast_expr->add_child(VirtualSlotRef::create_shared(create_virtual_slot_ref_node(1)));
+ const auto conjunct = VExprContext::create_shared(cast_expr);
+
+ EXPECT_EQ(conjunct->get_digest(12345), 0);
+ EXPECT_EQ(Scanner::TEST_build_condition_cache_digest(12345, {conjunct}),
0);
+}
+
TEST_F(VirtualSlotRefTest, EqualsFunction_SameObjects) {
// Test equals with same objects
TExprNode node = create_virtual_slot_ref_node(1, "test_label");
diff --git
a/be/test/storage/segment/segment_iterator_candidate_pushdown_test.cpp
b/be/test/storage/segment/segment_iterator_candidate_pushdown_test.cpp
index d03293abb6f..8b87b392e7e 100644
--- a/be/test/storage/segment/segment_iterator_candidate_pushdown_test.cpp
+++ b/be/test/storage/segment/segment_iterator_candidate_pushdown_test.cpp
@@ -468,6 +468,50 @@ TEST_F(SegmentIteratorCandidatePushdownTest,
versioned_deletes_do_not_publish_co
EXPECT_FALSE(older_reader->_find_condition_cache);
EXPECT_EQ(older_reader->_row_bitmap.cardinality(), 100);
EXPECT_EQ(_iter->_opts.condition_cache_digest, 0);
+ EXPECT_EQ(_stats.condition_cache_lookup_count, 0);
+ EXPECT_EQ(older_stats.condition_cache_lookup_count, 1);
+}
+
+TEST_F(SegmentIteratorCandidatePushdownTest,
+ condition_cache_lookups_count_misses_and_hits_per_reader) {
+ ScopedConditionCache cache;
+ constexpr uint64_t digest = 12345;
+ _iter->_opts.condition_cache_digest = digest;
+ _iter->_row_bitmap.addRange(0, 100);
+ _iter->_init_row_bitmap_by_condition_cache();
+ EXPECT_FALSE(_iter->_find_condition_cache);
+ EXPECT_EQ(_stats.condition_cache_lookup_count, 1);
+ EXPECT_EQ(_stats.condition_cache_hit_seg_nums, 0);
+
+ ConditionCache::CacheKey key(_iter->_opts.rowset_id, _iter->segment_id(),
digest);
+ cache.get()->insert(key, std::move(_iter->_condition_cache));
+
+ OlapReaderStatistics second_stats;
+ auto second_reader = std::make_unique<SegmentIterator>(_segment,
_read_schema);
+ second_reader->_opts.stats = &second_stats;
+ second_reader->_opts.condition_cache_digest = digest;
+ second_reader->_common_expr_ctxs_push_down =
_iter->_common_expr_ctxs_push_down;
+ second_reader->_row_bitmap.addRange(0, 100);
+ second_reader->_init_row_bitmap_by_condition_cache();
+ EXPECT_TRUE(second_reader->_find_condition_cache);
+ EXPECT_EQ(second_stats.condition_cache_lookup_count, 1);
+ EXPECT_EQ(second_stats.condition_cache_hit_seg_nums, 1);
+ EXPECT_TRUE(second_reader->_row_bitmap.isEmpty());
+ EXPECT_EQ(_stats.condition_cache_lookup_count, 1);
+ EXPECT_EQ(_stats.condition_cache_hit_seg_nums, 0);
+}
+
+TEST_F(SegmentIteratorCandidatePushdownTest,
condition_cache_bypass_does_not_count_lookups) {
+ ScopedConditionCache cache;
+ _iter->_opts.condition_cache_digest = 0;
+ _iter->_init_row_bitmap_by_condition_cache();
+ EXPECT_EQ(_stats.condition_cache_lookup_count, 0);
+
+ _iter->_opts.condition_cache_digest = 12345;
+ _iter->_common_expr_ctxs_push_down.clear();
+ _iter->_init_row_bitmap_by_condition_cache();
+ EXPECT_EQ(_stats.condition_cache_lookup_count, 0);
+ EXPECT_EQ(_iter->_opts.condition_cache_digest, 0);
}
TEST_F(SegmentIteratorCandidatePushdownTest,
condition_ranges_engage_candidate_before_expr) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 39eb500b6db..970d640c436 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -33,7 +33,6 @@ import org.apache.doris.analysis.SortInfo;
import org.apache.doris.analysis.TableSample;
import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.analysis.TupleId;
-import org.apache.doris.catalog.AliasFunction;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.KeysType;
@@ -95,12 +94,10 @@ import
org.apache.doris.nereids.trees.expressions.SessionVarGuardExpr;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.WindowFrame;
-import org.apache.doris.nereids.trees.expressions.functions.Udf;
import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateParam;
import org.apache.doris.nereids.trees.expressions.functions.agg.AggregatePhase;
import
org.apache.doris.nereids.trees.expressions.functions.scalar.GroupingScalarFunction;
-import
org.apache.doris.nereids.trees.expressions.functions.scalar.UniqueFunction;
import org.apache.doris.nereids.trees.plans.AbstractPlan;
import org.apache.doris.nereids.trees.plans.AggMode;
import org.apache.doris.nereids.trees.plans.AggPhase;
@@ -885,6 +882,7 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
@NotNull
private PlanFragment getPlanFragmentForPhysicalFileScan(PhysicalFileScan
fileScan, PlanTranslatorContext context,
ScanNode scanNode) {
+ scanNode.setEnableConditionCache(true);
scanNode.setNereidsId(fileScan.getId());
context.getNereidsIdToPlanNodeIdMap().put(fileScan.getId(),
scanNode.getId());
scanNode.setPushDownAggNoGrouping(context.getRelationPushAggOp(fileScan.getRelationId()));
@@ -942,6 +940,8 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
OlapScanNode olapScanNode = new OlapScanNode(context.nextPlanNodeId(),
tupleDescriptor, "OlapScanNode",
context.getScanContext());
+ olapScanNode.setEnableConditionCache(
+
!ExpressionUtils.containsNonCacheableExpression(olapScan.getVirtualColumns()));
olapScanNode.setNereidsId(olapScan.getId());
context.getNereidsIdToPlanNodeIdMap().put(olapScan.getId(),
olapScanNode.getId());
olapScanNode.setDistributeExprLists(getDistributeExpr(olapScan));
@@ -1148,6 +1148,7 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
TableValuedFunctionIf catalogFunction =
tvfRelation.getFunction().getCatalogFunction();
SessionVariable sv = ConnectContext.get().getSessionVariable();
ScanNode scanNode =
catalogFunction.getScanNode(context.nextPlanNodeId(), tupleDescriptor, sv);
+ scanNode.setEnableConditionCache(true);
scanNode.setDistributeExprLists(getDistributeExpr(tvfRelation));
scanNode.setNereidsId(tvfRelation.getId());
context.getNereidsIdToPlanNodeIdMap().put(tvfRelation.getId(),
scanNode.getId());
@@ -2864,6 +2865,7 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
TableValuedFunctionIf catalogFunction =
tvfRelation.getFunction().getCatalogFunction();
SessionVariable sv = ConnectContext.get().getSessionVariable();
ScanNode scanNode =
catalogFunction.getScanNode(context.nextPlanNodeId(), tupleDescriptor, sv);
+ scanNode.setEnableConditionCache(true);
scanNode.setDistributeExprLists(getDistributeExpr(tvfRelation));
scanNode.setNereidsId(tvfRelation.getId());
context.getNereidsIdToPlanNodeIdMap().put(tvfRelation.getId(),
scanNode.getId());
@@ -3287,6 +3289,10 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
private void addConjunctsToPlanNode(PhysicalFilter<? extends Plan> filter,
PlanNode planNode,
PlanTranslatorContext context) {
+ if (planNode instanceof ScanNode
+ &&
ExpressionUtils.containsNonCacheableExpression(filter.getConjuncts())) {
+ planNode.setEnableConditionCache(false);
+ }
for (Expression conjunct : filter.getConjuncts()) {
for (Expression singleConjunct :
ExpressionUtils.extractConjunctionToSet(conjunct)) {
planNode.addConjunct(ExpressionTranslator.translate(singleConjunct, context));
@@ -4037,28 +4043,28 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
boolean result;
if (plan instanceof PhysicalHashAggregate) {
PhysicalHashAggregate<? extends Plan> aggregate =
(PhysicalHashAggregate<? extends Plan>) plan;
- if (hasUndeterministicExpression(aggregate.getGroupByExpressions())
- ||
hasUndeterministicExpression(aggregate.getOutputExpressions())) {
+ if
(ExpressionUtils.containsNonCacheableExpression(aggregate.getGroupByExpressions())
+ ||
ExpressionUtils.containsNonCacheableExpression(aggregate.getOutputExpressions()))
{
result = true;
} else {
result = hasUndeterministicExpression(aggregate.child());
}
} else if (plan instanceof PhysicalFilter) {
PhysicalFilter<? extends Plan> filter = (PhysicalFilter<? extends
Plan>) plan;
- if (hasUndeterministicExpression(filter.getExpressions())) {
+ if
(ExpressionUtils.containsNonCacheableExpression(filter.getExpressions())) {
result = true;
} else {
result = hasUndeterministicExpression(filter.child());
}
} else if (plan instanceof PhysicalProject) {
PhysicalProject<? extends Plan> project = (PhysicalProject<?
extends Plan>) plan;
- if (hasUndeterministicExpression(project.getProjects())) {
+ if
(ExpressionUtils.containsNonCacheableExpression(project.getProjects())) {
result = true;
} else {
result = hasUndeterministicExpression(project.child());
}
} else if (plan instanceof PhysicalOlapScan) {
- result = false;
+ result =
ExpressionUtils.containsNonCacheableExpression(((PhysicalOlapScan)
plan).getVirtualColumns());
} else {
// unsupported for query cache
result = true;
@@ -4066,25 +4072,4 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor<PlanFragment, Pla
plan.setMutableState(cacheKey, result);
return result;
}
-
- private boolean hasUndeterministicExpression(Collection<? extends
Expression> expressions) {
- for (Expression groupByExpression : expressions) {
- if (groupByExpression.containsType(AliasFunction.class, Udf.class,
UniqueFunction.class)) {
- return true;
- }
-
- boolean nonDeterministic = groupByExpression.anyMatch(e -> {
- if (e instanceof Expression) {
- if (!((Expression) e).isDeterministic()) {
- return true;
- }
- }
- return false;
- });
- if (nonDeterministic) {
- return true;
- }
- }
- return false;
- }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
index f1ef04201d1..47c90bd413f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
@@ -31,6 +31,7 @@ import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.plans.physical.RuntimeFilter;
import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.planner.CTEScanNode;
import org.apache.doris.planner.DataStreamSink;
import org.apache.doris.planner.DistributionMode;
@@ -232,6 +233,7 @@ public class RuntimeFilterTranslator {
Expr targetExpr = targetExprList.get(i);
origFilter.addTarget(new RuntimeFilterTarget(
scanNode, targetExpr, true, isLocalTarget));
+ updateConditionCacheEligibility(scanNode, group.get(i));
// TRuntimeFilterDesc keys target expressions and pruning
metadata by scan node ID.
// If one grouped RF has different targets on the same
scan, BE cannot match
// metadata back to a specific target expression, so skip
pruning metadata.
@@ -335,6 +337,7 @@ public class RuntimeFilterTranslator {
Expr targetExpr = targetExprList.get(i);
origFilter.addTarget(new RuntimeFilterTarget(
scanNode, targetExpr, true, isLocalTarget));
+ updateConditionCacheEligibility(scanNode, filter);
setPruningMetadata(origFilter, scanNode, filter);
}
origFilter.setBloomFilterSizeCalculatedByNdv(filter.isBloomFilterSizeCalculatedByNdv());
@@ -365,6 +368,19 @@ public class RuntimeFilterTranslator {
return origFilter;
}
+ private void updateConditionCacheEligibility(ScanNode scanNode,
RuntimeFilter filter) {
+ Expression target = filter.getTargetExpression();
+ DataType sourceType = filter.getSrcExpr().getDataType();
+ // Include the implicit cast inserted by
castTargetToSourceTypeIfNeeded.
+ if (!target.getDataType().equals(sourceType)) {
+ target = new Cast(target, sourceType);
+ }
+ // RF probes become scan predicates even when no PhysicalFilter is
attached to the scan.
+ if
(ExpressionUtils.containsNonCacheableExpression(ImmutableList.of(target))) {
+ scanNode.setEnableConditionCache(false);
+ }
+ }
+
private void setPruningMetadata(org.apache.doris.planner.RuntimeFilter
runtimeFilter,
ScanNode scanNode, RuntimeFilter nereidsFilter) {
if (nereidsFilter.canPrunePartitions()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FoldConstantForSqlCache.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FoldConstantForSqlCache.java
index 4a3a6c0dc8d..8af11374f45 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FoldConstantForSqlCache.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FoldConstantForSqlCache.java
@@ -56,9 +56,6 @@ public class FoldConstantForSqlCache implements
CustomRewriter {
BoundFunction boundFunction,
ExpressionRewriteContext context) {
Expression fold =
super.visitBoundFunction(boundFunction, context);
boolean unfold = !fold.isDeterministic();
- if (unfold) {
-
sqlCacheContext.setCannotProcessExpression(true);
- }
if (!boundFunction.isDeterministic() &&
!unfold) {
sqlCacheContext.addFoldNondeterministicPair(boundFunction, fold);
}
@@ -66,6 +63,10 @@ public class FoldConstantForSqlCache implements
CustomRewriter {
}
}.rewrite(root, ctx);
+ // Casts can depend on the query clock too, without
containing a BoundFunction.
+ if (foldNondeterministic.containsNondeterministic()) {
+ sqlCacheContext.setCannotProcessExpression(true);
+ }
if (foldNondeterministic != root) {
sqlCacheContext.addFoldFullNondeterministicPair(root, foldNondeterministic);
return foldNondeterministic;
@@ -86,12 +87,16 @@ public class FoldConstantForSqlCache implements
CustomRewriter {
cascadesContext.getConnectContext().getSessionVariable());
StatementContext statementContext =
cascadesContext.getStatementContext();
SqlCacheContext sqlCacheContext =
statementContext.getSqlCacheContext().orElse(null);
- if (!wantToUseSqlCache || !statementContext.hasNondeterministic()
- || sqlCacheContext == null
+ if (!wantToUseSqlCache || sqlCacheContext == null
|| !sqlCacheContext.supportSqlCache()) {
return plan;
}
+ // The binder tracks function calls; implicit casts can be introduced
after binding them.
+ if (!statementContext.hasNondeterministic() && !plan.anyMatch(node ->
+ ((Plan)
node).getExpressions().stream().anyMatch(Expression::containsNondeterministic)))
{
+ return plan;
+ }
rewriteJob.execute(jobContext);
return cascadesContext.getRewritePlan();
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
index 2c9795e1b44..578ecb09ae1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
@@ -80,6 +80,48 @@ public class Cast extends Expression implements
UnaryExpression, Monotonic {
this.isStrict = isStrict;
}
+ @Override
+ public boolean isDeterministic() {
+ // TIME has no date. These casts supply the date from the current
query's clock.
+ return child().isNullLiteral() ||
!containsTimeToDateCast(child().getDataType(), targetType);
+ }
+
+ private static boolean containsTimeToDateCast(DataType sourceType,
DataType targetType) {
+ if (sourceType.equals(targetType)) {
+ return false;
+ }
+ if (sourceType instanceof ArrayType && targetType instanceof
ArrayType) {
+ return containsTimeToDateCast(((ArrayType)
sourceType).getItemType(),
+ ((ArrayType) targetType).getItemType());
+ }
+ if (sourceType instanceof MapType && targetType instanceof MapType) {
+ MapType sourceMap = (MapType) sourceType;
+ MapType targetMap = (MapType) targetType;
+ return containsTimeToDateCast(sourceMap.getKeyType(),
targetMap.getKeyType())
+ || containsTimeToDateCast(sourceMap.getValueType(),
targetMap.getValueType());
+ }
+ if (sourceType instanceof StructType && targetType instanceof
StructType) {
+ List<StructField> sourceFields = ((StructType)
sourceType).getFields();
+ List<StructField> targetFields = ((StructType)
targetType).getFields();
+ // foldable() is called before CheckCast rejects incompatible
struct shapes.
+ if (sourceFields.size() != targetFields.size()) {
+ return false;
+ }
+ for (int i = 0; i < sourceFields.size(); i++) {
+ if (containsTimeToDateCast(sourceFields.get(i).getDataType(),
targetFields.get(i).getDataType())) {
+ return true;
+ }
+ }
+ return false;
+ }
+ return sourceType.isTimeType() && targetType.isDateLikeType();
+ }
+
+ @Override
+ public boolean foldable() {
+ return isDeterministic();
+ }
+
public boolean isExplicitType() {
return isExplicitType;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayShuffle.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayShuffle.java
index f9039ace214..303fb924a89 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayShuffle.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayShuffle.java
@@ -28,6 +28,7 @@ import
org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
import org.apache.doris.nereids.types.BigIntType;
import org.apache.doris.nereids.types.coercion.AnyDataType;
+import org.apache.doris.nereids.util.ExpressionUtils;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableList;
@@ -62,6 +63,7 @@ public class ArrayShuffle extends UniqueFunction
}
private ArrayShuffle(VolatileIdentity volatileIdentity, List<Expression>
children) {
+ // A seed fixes the sequence within a block, not the permutation for
an individual row.
super("array_shuffle", volatileIdentity, children);
}
@@ -79,6 +81,12 @@ public class ArrayShuffle extends UniqueFunction
return new ArrayShuffle(getFunctionParams(children));
}
+ @Override
+ public boolean foldable() {
+ // Only NULL propagation is safe; non-NULL permutations depend on the
input block.
+ return ExpressionUtils.hasNullLiteral(getArguments());
+ }
+
@Override
public ArrayShuffle withIgnoreUniqueId(boolean ignoreUniqueId) {
return new
ArrayShuffle(volatileIdentity.withIgnoreUniqueId(ignoreUniqueId), children);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
index 8aa62473e39..918856408e0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.util;
+import org.apache.doris.catalog.AliasFunction;
import org.apache.doris.common.Config;
import org.apache.doris.common.MaterializedViewException;
import org.apache.doris.common.NereidsException;
@@ -55,6 +56,7 @@ import org.apache.doris.nereids.trees.expressions.WhenClause;
import org.apache.doris.nereids.trees.expressions.WindowExpression;
import org.apache.doris.nereids.trees.expressions.functions.BoundFunction;
import
org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.functions.Udf;
import org.apache.doris.nereids.trees.expressions.functions.agg.Avg;
import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
import org.apache.doris.nereids.trees.expressions.functions.agg.Max;
@@ -76,6 +78,7 @@ import
org.apache.doris.nereids.trees.expressions.functions.scalar.NonNullable;
import org.apache.doris.nereids.trees.expressions.functions.scalar.NullIf;
import org.apache.doris.nereids.trees.expressions.functions.scalar.Nullable;
import org.apache.doris.nereids.trees.expressions.functions.scalar.Nvl;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.UniqueFunction;
import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
@@ -136,6 +139,13 @@ public class ExpressionUtils {
// folding the conjunct). We restrict MAX_MARK_SLOT_COUNT to 4 to bound
this cost.
private static final int MAX_MARK_SLOT_COUNT = 4;
+ /** Whether evaluating these expressions can produce results unsuitable
for reuse across queries. */
+ public static boolean containsNonCacheableExpression(Collection<? extends
Expression> expressions) {
+ return expressions.stream().anyMatch(expression ->
+ expression.containsType(AliasFunction.class, Udf.class,
UniqueFunction.class)
+ || expression.containsNondeterministic());
+ }
+
public static List<Expression> extractConjunction(Expression expr) {
return extract(And.class, expr);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
index 580dc9a4eaa..d680afac953 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/PlanNode.java
@@ -95,6 +95,8 @@ public abstract class PlanNode extends TreeNode<PlanNode> {
protected ArrayList<TupleId> tupleIds;
protected List<Expr> conjuncts = Lists.newArrayList();
+ // Enabled only after Nereids has checked the scan's expression
dependencies.
+ private boolean enableConditionCache = false;
// Conjuncts used to filter the original load file.
// In the load execution plan, the difference between "preFilterConjuncts"
and "conjuncts" is that
@@ -182,6 +184,7 @@ public abstract class PlanNode extends TreeNode<PlanNode> {
this.offset = node.offset;
this.tupleIds = Lists.newArrayList(node.tupleIds);
this.conjuncts = Expr.cloneList(node.conjuncts, null);
+ this.enableConditionCache = node.enableConditionCache;
this.cardinality = -1;
this.planNodeName = "V" + planNodeName;
@@ -313,6 +316,10 @@ public abstract class PlanNode extends TreeNode<PlanNode> {
return targetConjuncts.get(0);
}
+ public void setEnableConditionCache(boolean enableConditionCache) {
+ this.enableConditionCache = enableConditionCache;
+ }
+
public void addConjuncts(List<Expr> conjuncts) {
if (conjuncts == null) {
return;
@@ -476,6 +483,9 @@ public abstract class PlanNode extends TreeNode<PlanNode> {
TPlanNode msg = new TPlanNode();
msg.node_id = id.asInt();
msg.setNereidsId(nereidsId);
+ if (this instanceof ScanNode) {
+ msg.setEnableConditionCache(enableConditionCache);
+ }
msg.setIsSerialOperator(isSerialOperatorOnBe(ConnectContext.get()));
msg.num_children = children.size();
msg.limit = limit;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslatorBucketPruneTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslatorBucketPruneTest.java
index c83c732ebe3..00faf92ddeb 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslatorBucketPruneTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslatorBucketPruneTest.java
@@ -29,6 +29,7 @@ import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.IdGenerator;
import org.apache.doris.nereids.processor.post.RuntimeFilterContext;
import org.apache.doris.nereids.trees.expressions.Add;
+import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.SlotReference;
@@ -37,7 +38,10 @@ import
org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
import org.apache.doris.nereids.trees.plans.physical.RuntimeFilter;
import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.StringType;
+import org.apache.doris.nereids.types.TimeV2Type;
import org.apache.doris.planner.OlapScanNode;
import org.apache.doris.planner.PlanFragment;
import org.apache.doris.planner.PlanFragmentId;
@@ -180,6 +184,40 @@ class RuntimeFilterTranslatorBucketPruneTest {
: ImmutableList.of(blocking, nonBlocking));
}
+ @Test
+ void testVolatileProbeDisablesCacheForSingleAndGroupedFilters() {
+ for (boolean grouped : ImmutableList.of(false, true)) {
+ for (boolean implicitCast : ImmutableList.of(false, true)) {
+ TranslatorHarness harness = new
TranslatorHarness(PrimitiveType.DATETIMEV2);
+ SlotReference target = harness.addTargetSlot("s", new
Column("s", PrimitiveType.STRING),
+ StringType.INSTANCE);
+ Expression time = new Cast(target, TimeV2Type.SYSTEM_DEFAULT);
+ Expression probe = implicitCast ? time : new Cast(time,
DateTimeV2Type.SYSTEM_DEFAULT);
+ RuntimeFilter filter = harness.newFilter(target, probe);
+ List<RuntimeFilter> filters = grouped
+ ? ImmutableList.of(harness.newFilter(target, new
Cast(target, DateTimeV2Type.SYSTEM_DEFAULT)),
+ filter)
+ : ImmutableList.of(filter);
+ TRuntimeFilterDesc desc = harness.translate(filters);
+
Assertions.assertTrue(desc.planId_to_target_expr.containsKey(SCAN_NODE_ID));
+
Mockito.verify(harness.scanNode).setEnableConditionCache(false);
+ Mockito.verify(harness.scanNode,
Mockito.never()).setEnableConditionCache(true);
+ }
+ }
+ }
+
+ @Test
+ void testDeterministicProbeDoesNotChangeCacheEligibility() {
+ for (boolean grouped : ImmutableList.of(false, true)) {
+ TranslatorHarness harness = new
TranslatorHarness(PrimitiveType.BIGINT);
+ SlotReference target = harness.addTargetSlot("k",
harness.distributionColumn, IntegerType.INSTANCE);
+ RuntimeFilter filter = harness.newFilter(target, new Add(target,
new IntegerLiteral(1)));
+ harness.translate(grouped ? ImmutableList.of(filter,
harness.newFilter(target, target))
+ : ImmutableList.of(filter));
+ Mockito.verify(harness.scanNode,
Mockito.never()).setEnableConditionCache(Mockito.anyBoolean());
+ }
+ }
+
private static int firstLegacySlotId(TranslatorHarness harness,
SlotReference target) {
return
harness.translatorContext.findSlotRef(target.getExprId()).getSlotId().asInt();
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/CacheExpressionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/CacheExpressionTest.java
new file mode 100644
index 00000000000..11aeff8bad3
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/CacheExpressionTest.java
@@ -0,0 +1,199 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.util;
+
+import org.apache.doris.catalog.Function;
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.catalog.FunctionVolatility;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRuleOnFE;
+import org.apache.doris.nereids.trees.expressions.Add;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.VolatileIdentity;
+import org.apache.doris.nereids.trees.expressions.functions.Udf;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayShuffle;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Now;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Random;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Uuid;
+import org.apache.doris.nereids.trees.expressions.functions.udf.JavaUdf;
+import org.apache.doris.nereids.trees.expressions.functions.udf.PythonUdf;
+import org.apache.doris.nereids.trees.expressions.literal.BigIntLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DateTimeType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
+import org.apache.doris.nereids.types.DateType;
+import org.apache.doris.nereids.types.DateV2Type;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.MapType;
+import org.apache.doris.nereids.types.StringType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+import org.apache.doris.nereids.types.TimeStampTzType;
+import org.apache.doris.nereids.types.TimeV2Type;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class CacheExpressionTest {
+ @Test
+ void testNestedExpressionsAndConjunctOrder() {
+ Expression stable = new Add(new SlotReference("k",
IntegerType.INSTANCE), new IntegerLiteral(1));
+ Assertions.assertFalse(uncacheable(stable));
+ for (Expression unstable : ImmutableList.of(new Random(), new
Random(new BigIntLiteral(1)),
+ new Uuid(), new Now())) {
+ Assertions.assertTrue(uncacheable(new Cast(unstable,
StringType.INSTANCE)));
+
Assertions.assertTrue(ExpressionUtils.containsNonCacheableExpression(ImmutableList.of(stable,
unstable)));
+
Assertions.assertTrue(ExpressionUtils.containsNonCacheableExpression(ImmutableList.of(unstable,
stable)));
+ }
+ }
+
+ @Test
+ void testAllUdfVolatilitiesRemainExcluded() {
+ Expression slot = new SlotReference("k", IntegerType.INSTANCE);
+ FunctionSignature signature =
FunctionSignature.ret(IntegerType.INSTANCE).args(IntegerType.INSTANCE);
+ for (FunctionVolatility volatility : FunctionVolatility.values()) {
+ VolatileIdentity identity = Udf.createVolatileIdentity(volatility);
+ Expression javaUdf = new JavaUdf("f", 1, "db",
Function.BinaryType.JAVA_UDF, signature,
+ Function.NullableMode.DEPEND_ON_ARGUMENT, volatility,
identity,
+ "file:///udf.jar", "F", "", "", "", false, 360, slot);
+ Expression pythonUdf = new PythonUdf("f", 2, "db",
Function.BinaryType.PYTHON_UDF, signature,
+ Function.NullableMode.DEPEND_ON_ARGUMENT, volatility,
identity,
+ "", "f", "", "", "", false, 360, "3.12", "", slot);
+ Assertions.assertEquals(volatility ==
FunctionVolatility.IMMUTABLE, javaUdf.isDeterministic());
+ Assertions.assertTrue(uncacheable(javaUdf));
+ Assertions.assertTrue(uncacheable(pythonUdf));
+ }
+ }
+
+ @Test
+ void testShuffleIsVolatileWithAndWithoutSeed() {
+ Expression array = new SlotReference("a",
ArrayType.of(IntegerType.INSTANCE));
+ for (ArrayShuffle shuffle : ImmutableList.of(new ArrayShuffle(array),
+ new ArrayShuffle(array, new BigIntLiteral(1)))) {
+ Assertions.assertTrue(uncacheable(shuffle));
+ Assertions.assertFalse(shuffle.foldable());
+ Assertions.assertEquals(shuffle,
shuffle.withChildren(shuffle.children()));
+ }
+ ArrayShuffle first = new ArrayShuffle(array, new BigIntLiteral(1));
+ ArrayShuffle second = new ArrayShuffle(array, new BigIntLiteral(1));
+ Assertions.assertNotEquals(first, second);
+ Assertions.assertEquals(first.withIgnoreUniqueId(true),
second.withIgnoreUniqueId(true));
+ }
+
+ @Test
+ void testTimeCastsDependOnQueryDate() {
+ Expression time = new SlotReference("t", TimeV2Type.SYSTEM_DEFAULT);
+ for (DataType target : ImmutableList.of(DateType.INSTANCE,
DateV2Type.INSTANCE,
+ DateTimeType.INSTANCE, DateTimeV2Type.SYSTEM_DEFAULT,
DateTimeV2Type.of(6),
+ TimeStampTzType.SYSTEM_DEFAULT)) {
+ Cast cast = new Cast(time, target);
+ Assertions.assertFalse(cast.isDeterministic());
+ Assertions.assertFalse(cast.foldable());
+ Assertions.assertTrue(uncacheable(new Cast(cast,
StringType.INSTANCE)));
+ }
+ Assertions.assertFalse(uncacheable(new Cast(time,
StringType.INSTANCE)));
+ Assertions.assertFalse(uncacheable(new Cast(time, TimeV2Type.of(6))));
+ Assertions.assertFalse(uncacheable(new Cast(
+ new SlotReference("dt", DateTimeV2Type.SYSTEM_DEFAULT),
DateV2Type.INSTANCE)));
+ }
+
+ @Test
+ void testNestedTimeCastsDependOnQueryDate() {
+ for (DataType date : ImmutableList.of(DateV2Type.INSTANCE,
DateTimeV2Type.SYSTEM_DEFAULT)) {
+ DataType time = TimeV2Type.SYSTEM_DEFAULT;
+ for (DataType[] types : ImmutableList.of(
+ new DataType[] {ArrayType.of(time), ArrayType.of(date)},
+ new DataType[] {MapType.of(time, IntegerType.INSTANCE),
MapType.of(date, IntegerType.INSTANCE)},
+ new DataType[] {MapType.of(IntegerType.INSTANCE, time),
MapType.of(IntegerType.INSTANCE, date)},
+ new DataType[] {structWithSecondField(time),
structWithSecondField(date)},
+ new DataType[]
{ArrayType.of(MapType.of(IntegerType.INSTANCE, structWithSecondField(time))),
+ ArrayType.of(MapType.of(IntegerType.INSTANCE,
structWithSecondField(date)))})) {
+ Cast cast = new Cast(new SlotReference("nested", types[0]),
types[1]);
+ Assertions.assertFalse(cast.isDeterministic(), cast.toSql());
+ Assertions.assertFalse(cast.foldable(), cast.toSql());
+ Assertions.assertTrue(uncacheable(cast), cast.toSql());
+ Expression nullCast = new Cast(new NullLiteral(types[0]),
types[1]);
+ Assertions.assertEquals(new NullLiteral(types[1]),
+ FoldConstantRuleOnFE.evaluateWithoutContext(nullCast));
+ }
+ }
+ }
+
+ @Test
+ void testDeterministicNestedCastsRemainCacheable() {
+ DataType time = TimeV2Type.SYSTEM_DEFAULT;
+ for (DataType[] types : ImmutableList.of(
+ new DataType[] {ArrayType.of(time),
ArrayType.of(StringType.INSTANCE)},
+ new DataType[] {MapType.of(IntegerType.INSTANCE, time),
MapType.of(BigIntType.INSTANCE, time)},
+ new DataType[] {structWithSecondField(time),
structWithSecondField(TimeV2Type.of(6))},
+ new DataType[] {ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT),
ArrayType.of(DateV2Type.INSTANCE)})) {
+ Cast cast = new Cast(new SlotReference("nested", types[0]),
types[1]);
+ Assertions.assertTrue(cast.isDeterministic(), cast.toSql());
+ Assertions.assertTrue(cast.foldable(), cast.toSql());
+ Assertions.assertFalse(uncacheable(cast), cast.toSql());
+ }
+ // An invalid shape must still reach the existing cast-legality check
before NULL folding.
+ Expression invalid = new Cast(new
NullLiteral(structWithSecondField(time)),
+ new StructType(ImmutableList.of(new StructField("x",
DateV2Type.INSTANCE, true, ""))));
+ Assertions.assertSame(invalid,
FoldConstantRuleOnFE.evaluateWithoutContext(invalid));
+ }
+
+ private DataType structWithSecondField(DataType type) {
+ return new StructType(ImmutableList.of(new StructField("stable",
IntegerType.INSTANCE, true, ""),
+ new StructField("value", type, true, "")));
+ }
+
+ @Test
+ void testNullTimeCastsFoldBeforeCacheEligibility() {
+ for (DataType target : ImmutableList.of(DateType.INSTANCE,
DateV2Type.INSTANCE,
+ DateTimeType.INSTANCE, DateTimeV2Type.SYSTEM_DEFAULT,
DateTimeV2Type.of(6))) {
+ Expression cast = new Cast(new Cast(NullLiteral.INSTANCE,
TimeV2Type.SYSTEM_DEFAULT), target);
+ Expression folded =
FoldConstantRuleOnFE.evaluateWithoutContext(cast);
+ Assertions.assertEquals(new NullLiteral(target), folded);
+ Assertions.assertFalse(uncacheable(folded));
+ }
+ // NULL folding must not bypass the legality check for unsupported
cast pairs.
+ Expression unsupported = new Cast(new
NullLiteral(TimeV2Type.SYSTEM_DEFAULT),
+ TimeStampTzType.SYSTEM_DEFAULT);
+ Assertions.assertSame(unsupported,
FoldConstantRuleOnFE.evaluateWithoutContext(unsupported));
+ }
+
+ @Test
+ void testNullShuffleArgumentsFoldBeforeCacheEligibility() {
+ ArrayType arrayType = ArrayType.of(IntegerType.INSTANCE);
+ Expression nullArray = new Cast(NullLiteral.INSTANCE, arrayType);
+ Expression array = new SlotReference("a", arrayType);
+ for (Expression shuffle : ImmutableList.of(new ArrayShuffle(nullArray),
+ new ArrayShuffle(nullArray, new BigIntLiteral(1)),
+ new ArrayShuffle(array, new
NullLiteral(BigIntType.INSTANCE)))) {
+ Expression folded =
FoldConstantRuleOnFE.evaluateWithoutContext(shuffle);
+ Assertions.assertEquals(new NullLiteral(arrayType), folded);
+ Assertions.assertFalse(uncacheable(folded));
+ }
+ }
+
+ private boolean uncacheable(Expression expression) {
+ return
ExpressionUtils.containsNonCacheableExpression(ImmutableList.of(expression));
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/planner/ConditionCacheEligibilityTest.java
b/fe/fe-core/src/test/java/org/apache/doris/planner/ConditionCacheEligibilityTest.java
new file mode 100644
index 00000000000..b3f6dd0ed6e
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/planner/ConditionCacheEligibilityTest.java
@@ -0,0 +1,173 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.planner;
+
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.SqlCacheContext;
+import org.apache.doris.planner.normalize.QueryCacheNormalizer;
+import org.apache.doris.thrift.TPlanNode;
+import org.apache.doris.thrift.TPlanNodeType;
+import org.apache.doris.utframe.TestWithFeService;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+class ConditionCacheEligibilityTest extends TestWithFeService {
+ @Override
+ protected void runBeforeAll() throws Exception {
+ createDatabase("cache_eligibility");
+ useDatabase("cache_eligibility");
+ createTable("create table t(k int, a array<int>, dt1 datetime, dt2
datetime, s string) "
+ + "duplicate key(k) distributed by hash(k) buckets 3
properties('replication_num'='1')");
+
connectContext.getSessionVariable().setDisableNereidsRules("PRUNE_EMPTY_PARTITION");
+ connectContext.getSessionVariable().setEnableQueryCache(true);
+ connectContext.getSessionVariable().parallelPipelineTaskNum = 2;
+ connectContext.getSessionVariable().enableConditionCache = true;
+ }
+
+ @Test
+ void testBothCachesRejectUnstableFilters() throws Exception {
+ for (String predicate : ImmutableList.of(
+ "k < 0 or rand(1) < 0.0001",
+ "k < 0 or rand() < 0.0001",
+ "concat(cast(k as string), uuid()) = 'x'",
+ "array_shuffle(a) = [1,2,3]",
+ "array_shuffle(a, 1) = [1,2,3]",
+ "shuffle(a, 1) = [1,2,3]",
+ "cast(timediff(dt1, dt2) as date) = date '2026-10-01'",
+ "cast(timediff(dt1, dt2) as datetime) = cast(dt1 as datetime)",
+ "cast(array(timediff(dt1, dt2)) as array<datetime>)[1] = dt1",
+ "cast(map(1, timediff(dt1, dt2)) as map<int, datetime>)[1] =
dt1",
+ "struct_element(cast(struct(k, timediff(dt1, dt2)) as
struct<id:int, d:datetime>), 2) = dt1")) {
+ Planner planner = getSqlStmtExecutor("select k, sum(k) from t
where " + predicate + " group by k")
+ .planner();
+ List<TPlanNode> scans = scans(planner);
+ Assertions.assertEquals(1, scans.size(), predicate);
+ Assertions.assertFalse(scans.get(0).isEnableConditionCache(),
predicate);
+ for (PlanFragment fragment : planner.getFragments()) {
+ Assertions.assertFalse(new QueryCacheNormalizer(fragment,
planner.getDescTable())
+ .normalize(connectContext).isPresent(), predicate);
+ }
+ }
+ }
+
+ @Test
+ void testDeterministicFiltersAndRandomOutput() throws Exception {
+ Planner deterministic = getSqlStmtExecutor("select k, sum(k) from t
where k > 100 group by k").planner();
+
Assertions.assertTrue(scans(deterministic).get(0).isEnableConditionCache());
+
Assertions.assertTrue(deterministic.getFragments().stream().anyMatch(fragment ->
+ new QueryCacheNormalizer(fragment,
deterministic.getDescTable()).normalize(connectContext).isPresent()));
+ Planner randomOutput = getSqlStmtExecutor("select rand(), k from t
where k > 100").planner();
+
Assertions.assertTrue(scans(randomOutput).get(0).isEnableConditionCache());
+ }
+
+ @Test
+ void testScanEligibilityIsIndependentAcrossUnionBranches() throws
Exception {
+ Planner planner = getSqlStmtExecutor("select k from t where k < 0 or
rand(1) < 0.0001 "
+ + "union all select k from t where k > 100").planner();
+ List<TPlanNode> scans = scans(planner);
+ Assertions.assertEquals(2, scans.size());
+ Assertions.assertEquals(1,
scans.stream().filter(TPlanNode::isEnableConditionCache).count());
+ }
+
+ @Test
+ void testSqlCacheRejectsClockDependentCastsAndShuffle() throws Exception {
+ boolean enabled =
connectContext.getSessionVariable().isEnableSqlCache();
+ connectContext.getSessionVariable().setEnableSqlCache(true);
+ try {
+ for (String expression : ImmutableList.of("k + 1", "year(now())"))
{
+ NereidsPlanner planner = (NereidsPlanner)
getSqlStmtExecutor("select " + expression + " from t")
+ .planner();
+
Assertions.assertTrue(planner.getStatementContext().getSqlCacheContext().orElseThrow()
+ .supportSqlCache(), expression);
+ }
+ for (String expression : ImmutableList.of("cast(timediff(dt1, dt2)
as date)",
+ "cast(timediff(dt1, dt2) as datetime)",
"array_shuffle(a)", "shuffle(a, 1)",
+ "cast(array(timediff(dt1, dt2)) as array<datetime>)",
+ "cast(map(1, timediff(dt1, dt2)) as map<int, datetime>)",
+ "cast(struct(k, timediff(dt1, dt2)) as struct<id:int,
d:datetime>)")) {
+ NereidsPlanner planner = (NereidsPlanner)
getSqlStmtExecutor("select " + expression + " from t")
+ .planner();
+ SqlCacheContext cache =
planner.getStatementContext().getSqlCacheContext().orElseThrow();
+ Assertions.assertFalse(cache.supportSqlCache(), expression);
+ }
+ } finally {
+ connectContext.getSessionVariable().setEnableSqlCache(enabled);
+ }
+ }
+
+ @Test
+ void testRuntimeFilterProbeEligibility() throws Exception {
+ for (boolean clockDependent : ImmutableList.of(true, false)) {
+ String probe = clockDependent ? "cast(cast(p.s as time) as
datetime)" : "cast(p.s as datetime)";
+ Planner planner = getSqlStmtExecutor("select /*+
SET_VAR(disable_join_reorder=true, "
+ + "enable_runtime_filter_prune=false) */ p.k from t p join
[broadcast] t b on "
+ + probe + " = b.dt1").planner();
+ List<TPlanNode> targets = new ArrayList<>();
+ for (TPlanNode scan : scans(planner)) {
+ if (scan.isSetRuntimeFilters() &&
scan.getRuntimeFilters().stream().anyMatch(filter ->
+
filter.planId_to_target_expr.containsKey(scan.getNodeId()))) {
+ targets.add(scan);
+ Assertions.assertEquals(!clockDependent,
scan.isEnableConditionCache(), probe);
+ }
+ }
+ Assertions.assertFalse(targets.isEmpty(), "Expected a runtime
filter target for " + probe);
+ }
+ }
+
+ @Test
+ void testNullExpressionsRemainCacheable() throws Exception {
+ boolean enabled =
connectContext.getSessionVariable().isEnableSqlCache();
+ connectContext.getSessionVariable().setEnableSqlCache(true);
+ try {
+ for (String expression : ImmutableList.of("cast(cast(null as time)
as date)",
+ "cast(cast(null as time) as datetime)",
+ "array_shuffle(cast(null as array<int>))",
+ "array_shuffle(cast(null as array<int>), 1)",
+ "shuffle(cast(null as array<int>), 1)", "array_shuffle(a,
null)")) {
+ NereidsPlanner planner = (NereidsPlanner) getSqlStmtExecutor(
+ "select k, sum(k) from t where " + expression + " is
null group by k").planner();
+
Assertions.assertTrue(planner.getStatementContext().getSqlCacheContext().orElseThrow()
+ .supportSqlCache(), expression);
+
Assertions.assertTrue(scans(planner).get(0).isEnableConditionCache(),
expression);
+
Assertions.assertTrue(planner.getFragments().stream().anyMatch(fragment ->
+ new QueryCacheNormalizer(fragment,
planner.getDescTable())
+ .normalize(connectContext).isPresent()),
expression);
+ }
+ } finally {
+ connectContext.getSessionVariable().setEnableSqlCache(enabled);
+ }
+ }
+
+ private List<TPlanNode> scans(Planner planner) {
+ List<TPlanNode> scans = new ArrayList<>();
+ for (PlanFragment fragment : planner.getFragments()) {
+ for (TPlanNode node :
fragment.getPlanRoot().treeToThrift().getNodes()) {
+ if (node.getNodeType() == TPlanNodeType.OLAP_SCAN_NODE) {
+ Assertions.assertTrue(node.isSetEnableConditionCache());
+ scans.add(node);
+ }
+ }
+ }
+ return scans;
+ }
+}
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index dfcd5948c71..661bd293463 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -1809,6 +1809,9 @@ struct TPlanNode {
106: optional list<i32> topn_filter_source_node_ids
107: optional i32 nereids_id
+ // FE expression eligibility, independent of the session switch. An old FE
has not checked
+ // volatility, so absence must disable condition cache on a new BE.
+ 108: optional bool enable_condition_cache = false
}
// A flattened representation of a tree of PlanNodes, obtained by depth-first
diff --git
a/regression-test/data/query_p0/cache/condition_cache_eligibility.out
b/regression-test/data/query_p0/cache/condition_cache_eligibility.out
new file mode 100644
index 00000000000..df1e1099449
--- /dev/null
+++ b/regression-test/data/query_p0/cache/condition_cache_eligibility.out
@@ -0,0 +1,16 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !volatile_rf_first --
+10000
+
+-- !volatile_rf_second --
+10000
+
+-- !deterministic_first --
+10
+
+-- !deterministic_second --
+10
+
+-- !null_expressions --
+100000
+
diff --git
a/regression-test/suites/correctness/test_cast_time_to_datetime.groovy
b/regression-test/suites/correctness/test_cast_time_to_datetime.groovy
index 35b45f25c9b..02822d31e53 100644
--- a/regression-test/suites/correctness/test_cast_time_to_datetime.groovy
+++ b/regression-test/suites/correctness/test_cast_time_to_datetime.groovy
@@ -20,20 +20,21 @@ suite("test_cast_time_to_datetime") {
def result1 = sql """ select datediff(now(), from_unixtime(cast(1742194502
as bigint),'yyyy-MM-dd HH:mm:ss')); """
def result2 = sql """ select datediff(current_time(),
from_unixtime(cast(1742194502 as bigint),'yyyy-MM-dd HH:mm:ss')); """
assertEquals(result1[0][0], result2[0][0], "The results of the two SQL
queries should be the same.")
+ // TIME-to-date conversions use the query date and must remain runtime
expressions.
explain {
sql """select cast(cast("500:00:00" as time) as datetime)""" // not
legal for date part
- notContains "CAST"
+ contains "CAST"
}
explain {
sql """select cast(cast("10:10:10" as time) as datetime)""" // also
legal for date part
- notContains "CAST"
+ contains "CAST"
}
explain {
sql """select cast(cast("500:00:00" as time) as date)""" // not legal
for date part
- notContains "CAST"
+ contains "CAST"
}
explain {
sql """select cast(cast("10:10:10" as time) as date)""" // also legal
for date part
- notContains "CAST"
+ contains "CAST"
}
}
\ No newline at end of file
diff --git
a/regression-test/suites/query_p0/cache/condition_cache_eligibility.groovy
b/regression-test/suites/query_p0/cache/condition_cache_eligibility.groovy
new file mode 100644
index 00000000000..8b7fb73efcd
--- /dev/null
+++ b/regression-test/suites/query_p0/cache/condition_cache_eligibility.groovy
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.action.ProfileAction
+
+suite("condition_cache_eligibility") {
+ sql "DROP TABLE IF EXISTS condition_cache_eligibility"
+ sql """
+ CREATE TABLE condition_cache_eligibility (
+ k BIGINT,
+ a ARRAY<INT>,
+ dt1 DATETIME,
+ dt2 DATETIME
+ ) DUPLICATE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO condition_cache_eligibility
+ SELECT number, [1,2,3,4,5,6,7], '2026-09-30 12:00:00', '2026-09-30
12:00:00'
+ FROM numbers("number" = "100000")
+ """
+
+ sql "set enable_parallel_scan = false"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ sql "set enable_condition_cache = true"
+ sql "set enable_profile = true"
+ sql "set profile_level = 2"
+
+ def profileAction = new ProfileAction(context)
+ def checkNoCacheLookup = { String query, Closure execute ->
+ // A unique statement marker identifies this execution, including
repeated queries.
+ // Query profiles are unaffected by concurrent suites or internal
statistics queries.
+ def token = UUID.randomUUID().toString()
+ execute("/* ${token} */ ${query}")
+ def profile = profileAction.getProfileBySql(token,
["ConditionCacheLookup"])
+ def lookups = (profile =~ /ConditionCacheLookup: (\d+)/).findAll()
+ assertFalse(lookups.isEmpty(), "Missing condition-cache lookup
counter: ${profile}")
+ assertTrue(lookups.every { (it[1] as long) == 0 },
+ "Ineligible scan looked up condition cache:
${query}\n${profile}")
+ }
+
+ sql "set experimental_enable_virtual_slot_for_cse = true"
+ def virtualFirst = """
+ SELECT k FROM condition_cache_eligibility
+ WHERE abs(k + 1) + abs(k + 1) = 0 ORDER BY k
+ """
+ def virtualSecond = """
+ SELECT k FROM condition_cache_eligibility
+ WHERE abs(k - 9999) + abs(k - 9999) = 0 ORDER BY k
+ """
+ for (String query : [virtualFirst, virtualSecond]) {
+ explain {
+ sql query
+ verbose true
+ contains "virtualColumn=abs"
+ }
+ }
+ // Compare with uncached base-column predicates to verify virtual-column
results.
+ def emptyReference = """
+ SELECT /*+ SET_VAR(enable_condition_cache=false) */ k
+ FROM condition_cache_eligibility WHERE k = -1 ORDER BY k
+ """
+ def matchingReference = """
+ SELECT /*+ SET_VAR(enable_condition_cache=false) */ k
+ FROM condition_cache_eligibility WHERE k = 9999 ORDER BY k
+ """
+ checkNoCacheLookup(virtualFirst, { query -> check_sqls_result_equal(query,
emptyReference) })
+ 2.times {
+ checkNoCacheLookup(virtualSecond, { query ->
check_sqls_result_equal(query, matchingReference) })
+ }
+
+ // Seeded shuffle is repeatable within a block, but its count changes with
block boundaries.
+ // Check that volatile scans never look up the cache instead of recording
their random counts.
+ for (int batchSize : [1, 1024]) {
+ sql "set batch_size = ${batchSize}"
+ for (String predicate : [
+ "k < 0 OR rand(1) < 0.0001",
+ "k < 0 OR rand() < 0.0001",
+ "array_shuffle(a) = [1,2,3,4,5,6,7]",
+ "array_shuffle(a, 1) = [1,2,3,4,5,6,7]",
+ "shuffle(a, 1) = [1,2,3,4,5,6,7]",
+ "CAST(TIMEDIFF(dt1, dt2) AS DATE) = CURRENT_DATE()",
+ "CAST(TIMEDIFF(dt1, dt2) AS DATETIME) = CAST(CURRENT_DATE() AS
DATETIME)",
+ "CAST(ARRAY(TIMEDIFF(dt1, dt2)) AS ARRAY<DATETIME>)[1] =
CAST(CURRENT_DATE() AS DATETIME)",
+ "CAST(MAP(1, TIMEDIFF(dt1, dt2)) AS MAP<INT, DATETIME>)[1] =
CAST(CURRENT_DATE() AS DATETIME)",
+ "STRUCT_ELEMENT(CAST(STRUCT(k, TIMEDIFF(dt1, dt2)) AS
STRUCT<id:BIGINT, d:DATETIME>), 2)"
+ + " = CAST(CURRENT_DATE() AS DATETIME)"]) {
+ 2.times {
+ // Keep one-row batches inexpensive under ASAN while retaining
the volatile scan predicate.
+ checkNoCacheLookup(
+ "SELECT count(*) FROM condition_cache_eligibility
WHERE k < 1000 AND (${predicate})",
+ { query -> sql query })
+ }
+ }
+ }
+
+ sql "DROP TABLE IF EXISTS condition_cache_rf_probe"
+ sql """
+ CREATE TABLE condition_cache_rf_probe (k BIGINT, s STRING NOT NULL)
+ DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO condition_cache_rf_probe
+ SELECT number, '00:00:00' FROM numbers("number" = "10000")
+ """
+ sql "DROP TABLE IF EXISTS condition_cache_rf_build"
+ sql """
+ CREATE TABLE condition_cache_rf_build (d DATETIME NOT NULL)
+ DUPLICATE KEY(d) DISTRIBUTED BY HASH(d) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ // Keep the expected join count stable even if these executions cross
midnight.
+ sql """
+ INSERT INTO condition_cache_rf_build VALUES
+ (CURRENT_DATE() - INTERVAL 1 DAY), (CURRENT_DATE()),
(CURRENT_DATE() + INTERVAL 1 DAY)
+ """
+ sql "set enable_runtime_filter_prune = false"
+ sql "set runtime_filter_wait_infinitely = true"
+ sql "set disable_join_reorder = true"
+ def rfQuery = """
+ SELECT count(*) FROM condition_cache_rf_probe p
+ JOIN [broadcast] condition_cache_rf_build b
+ ON CAST(CAST(p.s AS TIME) AS DATETIME) = b.d
+ """
+ explain {
+ sql rfQuery
+ contains "RF"
+ contains "CAST(CAST"
+ }
+ checkNoCacheLookup(rfQuery, { query -> order_qt_volatile_rf_first query })
+ checkNoCacheLookup(rfQuery, { query -> order_qt_volatile_rf_second query })
+
+ sql "set enable_profile = false"
+ def token = UUID.randomUUID().toString()
+ def deterministic = """
+ SELECT '${token}', count(*) FROM condition_cache_eligibility WHERE k %
10000 = 0
+ """
+ // Execute the same SQL twice so both scans have the same digest; omit the
profile token from output.
+ quickRunTest("deterministic_first", deterministic, true, { row -> [row[1]]
})
+ // Only the second execution records a profile, so the token cannot select
the warm-up query.
+ sql "set enable_profile = true"
+ sql "set profile_level = 2"
+ quickRunTest("deterministic_second", deterministic, true, { row ->
[row[1]] })
+ def profile = profileAction.getProfileBySql(token,
["ConditionCacheLookup", "ConditionCacheHit"])
+ def lookups = (profile =~ /ConditionCacheLookup: (\d+)/)
+ assertTrue(lookups.findAll().any { (it[1] as long) > 0 },
+ "Second deterministic scan did not look up condition cache:
${profile}")
+ def hits = (profile =~ /ConditionCacheHit: (\d+)/)
+ assertTrue(hits.findAll().any { (it[1] as long) > 0 },
+ "Second deterministic scan did not hit condition cache:
${profile}")
+
+ order_qt_null_expressions """
+ SELECT count(*) FROM condition_cache_eligibility
+ WHERE CAST(CAST(NULL AS TIME) AS DATETIME) IS NULL
+ AND array_shuffle(CAST(NULL AS ARRAY<INT>), 1) IS NULL
+ AND shuffle(CAST(NULL AS ARRAY<INT>)) IS NULL
+ AND array_shuffle(a, NULL) IS NULL
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
