HappenLee opened a new pull request, #68932:
URL: https://github.com/apache/doris/pull/68932
### What problem does this PR solve?
Issue Number: None
Related PR: #68672
Problem Summary:
Backport #68672 (`326c13634971d8cf641f0be6f56928c327d3f7d2`) to `branch-4.1`.
Condition cache can reuse false ranges for volatile expressions or
virtual-column predicates whose definitions are absent from the digest. For
example, evaluating `abs(k + 1) + abs(k + 1) = 0` before `abs(k - 9999) + abs(k
- 9999) = 0` can omit the matching `k = 9999` row.
Share query/condition cache expression eligibility, check nested
TIME-to-date casts and runtime-filter probes including implicit casts, and pass
scan eligibility to BE before computing the digest. Preserve supported NULL
folding and disable virtual-column digests. Add per-query lookup counters and
retain deterministic cache-hit coverage.
Conflict resolution for 4.1:
- Retain 4.1 file-scan signatures, the shared lazy TVF translation path, and
PlanNode serialization APIs.
- Check every target in 4.1's multi-target runtime filters and its separate
`RunTimeFilterTranslatorV2` path through one shared helper. Port translator
tests for both paths, including grouped targets and implicit casts.
- Port the BE lookup miss/hit/bypass tests into a focused fixture using
4.1's `Schema` API; the master candidate-pushdown fixture is absent on this
branch.
- Retain 4.1's `ConditionCacheSegmentHit` profile name and adapt the
regression assertions. The original generated `.out` file is unchanged.
- Adapt container-type imports, UDF enums, Optional usage and
constant-folding test APIs to 4.1. Preserve 4.1's existing typed-NULL folding
for mismatched struct shapes and TIME-to-TIMESTAMPTZ casts; master's separate
cast-legality change is not part of this backport.
### Release note
Prevent incorrect cached results for volatile predicates,
query-date-dependent TIME casts, and virtual-column predicates. Preserve
supported NULL folding. Expose `ConditionCacheLookup` in OLAP scan profiles.
Rolling upgrade: upgrade BE before FE. The optional plan flag defaults to
false, so a new BE disables condition cache for old FE plans; an old BE cannot
enforce the flag sent by a new FE.
### Check List (For Author)
- Test:
- [x] FE production and test compilation; 34 targeted FE tests passed
(CacheExpressionTest 8, ConditionCacheEligibilityTest 6,
RuntimeFilterTranslatorCacheEligibilityTest 2, UniqueFunctionTest 18).
- [x] FE Checkstyle: all 11 changed Java files, zero violations.
Repository C++ formatting/check scripts with clang-format 16.0.6; final focused
check for the ported BE test. Patch whitespace checks passed with the original
generated output's trailing blank retained.
- [ ] BE build/UT and regression: not completed locally. The installed
dependencies lack the Paimon archive/header; the BE UT configure attempt also
stops at missing OpenMP_C. No BE runtime pass is claimed. The 4.1 branch does
not contain the master build-hygiene/clang-tidy scripts, and no compilation
database was produced for clang-tidy.
Final FE test command (reuse the production classes after successful
compilation):
```bash
MAVEN_OPTS='-Dskip.clean=true -Dmaven.main.skip=true
-Dmaven.compiler.useIncrementalCompilation=false' FE_UT_PARALLEL=48 bash
run-fe-ut.sh --run
'org.apache.doris.nereids.util.CacheExpressionTest,org.apache.doris.planner.ConditionCacheEligibilityTest,org.apache.doris.nereids.glue.translator.RuntimeFilterTranslatorCacheEligibilityTest,org.apache.doris.nereids.trees.expressions.functions.scalar.UniqueFunctionTest'
```
- Behavior changed: Yes. Unsafe scans bypass condition cache; deterministic
scans remain eligible. The backport preserves 4.1's existing APIs and profile
names.
- Does this need documentation: No. Correctness fix using existing cache
switches; rolling-upgrade requirements are stated above.
<details>
<summary>Backport self-review</summary>
| Checkpoint | Conclusion |
| --- | --- |
| Goal and evidence | Preserve #68672 on 4.1; expression, planner,
translator and BE digest/lookup coverage retained or ported. See validation
above for executed checks. |
| Scope and reuse | Production changes remain focused; both RF translators
share one helper. Absent master test fixtures are replaced with cache-only
fixtures. |
| Concurrency and locks | No new thread, shared production mutable state,
lock, or lock ordering. Lookup statistics follow the existing per-reader
collection path. |
| Lifecycle and static initialization | Eligibility is set during planning
and read by the scan operator. No new cross-translation-unit initialization or
ownership lifecycle. The test cache is restored with RAII. |
| Configuration | No new configuration or session variable. |
| Compatibility | Optional Thrift field 108 defaults false; BE-first rolling
upgrades retain conservative behavior. No storage format change. |
| Parallel paths | OLAP, file and TVF scan eligibility and both 4.1 RF
translator paths are covered. Lazy TVF scans reuse the existing translation
method. |
| Conditional checks | Volatility and implicit-cast checks handle concrete
expression cases; 4.1's existing NULL folding is preserved. |
| Coverage | Tests cover volatile/deterministic expressions, NULL folding,
nested casts, single/multiple RF targets, V1/V2 translation, digest propagation
and lookup counters. Runtime limitations are stated above. |
| Expected output | The generated regression output is copied unchanged from
#68672. Only the 4.1 profile counter name is adapted. |
| Observability | ConditionCacheLookup counts actual lookups including
misses; existing hit and filtered-row counters remain. |
| Transactions and persistence | No EditLog, version visibility, transaction
or persistence path changes. |
| Data writes and atomicity | No data-write path changes; cache bypass uses
the existing zero-digest mechanism. |
| FE/BE propagation | PlanNode copies and scan serialization carry the flag;
BE reads it before digest construction. Unchecked plans remain disabled by
default. |
| Performance | Eligibility uses the shared expression walk; the BE counter
adds one increment per segment lookup. No new row-level allocation or locking. |
| Other findings | No additional correctness issue found in the backport
review; unexecuted BE/runtime validation remains explicit. |
</details>
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
