HappenLee opened a new pull request, #68672:
URL: https://github.com/apache/doris/pull/68672
### What problem does this PR solve?
Issue Number: None
Problem Summary:
Condition cache can reuse non-matching ranges for predicates whose results
change across queries or scan batches. For example, skipping ranges for `k < 0
OR rand(1) < 0.0001` changes which random values subsequent rows receive and
can produce incorrect results.
Extract query cache's expression eligibility into `ExpressionUtils` and
share it with condition cache. Nereids checks scan predicates and OLAP virtual
columns, then sends eligibility to BE. Ineligible scans reuse the existing
zero-digest path to disable both cache reads and writes. Deterministic scans
remain eligible, including scans with random expressions only in their output.
Also fix two missing classifications:
- Treat `array_shuffle` / `shuffle`, including seeded calls, as
`UniqueFunction`. A seed fixes the sequence within a block, but does not make a
row's permutation stable across different scan batches.
- Mark TIME-to-date/datetime casts as nondeterministic and non-foldable
because they supply the current query date. Extend SQL cache's check to
unresolved nondeterministic expression nodes, including Cast, while preserving
existing folding and revalidation for expressions such as `year(now())`.
The new Thrift field is optional and defaults to false, so an old FE cannot
accidentally enable condition cache on a new BE. The BE part is required for
this fix to take effect during upgrades.
### Release note
Prevent incorrect cached results for nondeterministic scan predicates.
Disable caching and constant folding for shuffle calls and TIME conversions
that depend on the query date.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (details below)
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Unsafe predicates bypass condition cache; deterministic scans
remain cacheable.
- Does this need documentation?
- [x] No.
- [ ] Yes.
Validation of the patch before transplanting it unchanged onto current
master:
- FE build with Checkstyle and BE ASAN build passed. The local toolchain
required `ASAN_OPTIONS=detect_leaks=0` for its dictionary generator; address
checks remained enabled.
- 57 FE tests passed across `CacheExpressionTest`,
`ConditionCacheEligibilityTest`, `QueryCacheNormalizerTest`, `CastTest`, and
`UniqueFunctionTest`.
- Four regression suites passed: `condition_cache_eligibility` (generated
output and a separate verification run), `test_cast_time_to_datetime`,
`test_array_functions`, and `test_array_functions_by_literal`.
- Runtime profiles showed zero condition-cache hits for repeated rand,
shuffle and TIME-cast predicates. A deterministic predicate hit on its second
execution and skipped 79,872 rows.
- clang-format 16.0.6 and BE header hygiene passed. clang-tidy was blocked
by an existing unmatched `NOLINTEND` in `be/src/core/types.h`; no diagnostics
pointed to changed lines.
The targeted FE tests are being rerun on the final PR branch.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
