This is an automated email from the ASF dual-hosted git repository. HappenLee pushed a commit to branch happenlee/internal-topn-condition-cache in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1022768e2c1745f1027505af0abb6ce9ff1e6019 Author: happenlee <[email protected]> AuthorDate: Thu Nov 12 16:39:44 2026 +0800 [improvement](be) Reuse static condition cache for internal TopN scans ### Background Condition cache stores per-segment granule results for scan predicates. Later scans with the same digest can skip granules that have no matching rows. TopN runtime filters tighten a threshold during execution, so their exclusions cannot be stored under a static predicate's cache key. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: Internal-table scans currently disable condition cache whenever a TopN filter source is present, even when a matching static predicate entry already exists. Allow those scans to consume the existing cache in read-only mode. Carry a write-permission flag through the reader options and guard bitmap allocation, survivor collection, and publication with it. Most of this diff adds tests for the policy and cache behavior. ### Details - **Change map:** 89 production lines changed across 11 files and 268 test lines changed across two files; no generated files or documentation changes. Production changes select the cache policy, pass one flag through existing reader options, and centralize the write guard. Tests cover policy selection and real cache lookup/publication behavior. - **Before and after:** Internal scans with TopN runtime filters previously skipped condition cache entirely. They can now read matching static predicate entries; on a miss they scan normally without allocating, collecting, or publishing a cache bitmap. External TopN scans retain their existing opt-out. - **Configs and defaults:** Reuse the existing condition-cache setting and per-scan enable flag. No new settings, cache keys, bitmap formats, or FE/BE protocol fields. - **Impact on existing users:** Warm static predicate entries can prune internal TopN scans. Ordinary scans keep their existing cache population behavior, including the guard against publishing after a pushed-down LIMIT truncates the scan. ### Testing - Passed locally: clang-format 16.0.6 on all changed C++ files, build-hygiene checks, and `git diff --check`. - Passed locally: compiler diagnostics and the `bugprone-unused-raii` check on all five changed production translation units and both test translation units, using the generated compilation database. The complete configured clang-tidy check set has not been completed. - Added eight unit tests covering static digest reuse, cache safety gates, external TopN opt-out, read-only misses and hits, unchanged cached bitmaps, ordinary scan population, and partial LIMIT scans. - Attempted `./run-be-ut.sh -j 48 --run --filter='ScanOperatorTest.*:SegmentIteratorConditionCacheTest.*'` on the PR base. Full build configuration is blocked by missing `bson-2.0` and `mongoc-2.0` headers in this machine's third-party installation. The tests and SQL end-to-end regressions have not run on this revision. ### Notes for Reviewers The digest is computed before `_process_conjuncts()` injects TopN expressions. It therefore remains compatible with entries produced by the original predicates. Read-only mode is selected from the plan's TopN source IDs before scanners are created, including when the runtime threshold has not arrived yet. ### Release note Internal-table scans with TopN runtime filters can reuse existing static predicate condition-cache entries. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: full build and test execution require the current third-party dependencies. ### AI Disclosure - Tool: Codex; the exact model identifier is not exposed in this session. - Generated: Implementation, unit tests, and this description. - Human review: Not reported in this session; author review is pending. --- be/src/exec/operator/olap_scan_operator.h | 1 + be/src/exec/operator/scan_operator.cpp | 36 +++-- be/src/exec/operator/scan_operator.h | 4 + be/src/exec/scan/olap_scanner.cpp | 1 + be/src/storage/iterators.h | 2 + be/src/storage/rowset/beta_rowset_reader.cpp | 1 + be/src/storage/rowset/rowset_reader_context.h | 1 + be/src/storage/segment/segment_iterator.cpp | 36 ++--- be/src/storage/segment/segment_iterator.h | 5 + be/src/storage/tablet/tablet_reader.cpp | 1 + be/src/storage/tablet/tablet_reader.h | 1 + be/test/exec/scan/scan_operator_test.cpp | 108 +++++++++++++- .../segment_iterator_condition_cache_test.cpp | 160 +++++++++++++++++++++ 13 files changed, 327 insertions(+), 30 deletions(-) diff --git a/be/src/exec/operator/olap_scan_operator.h b/be/src/exec/operator/olap_scan_operator.h index 0c3c0099c18..bed51601aba 100644 --- a/be/src/exec/operator/olap_scan_operator.h +++ b/be/src/exec/operator/olap_scan_operator.h @@ -78,6 +78,7 @@ private: void set_scan_ranges(RuntimeState* state, const std::vector<TScanRangeParams>& scan_ranges) override; Status _init_profile() override; + bool _supports_topn_condition_cache() const override { return true; } Status _process_conjuncts(RuntimeState* state) override; Status _on_runtime_filter_update(const VExprContextSPtrs& new_conjuncts) override; bool _is_key_column(const std::string& col_name) override; diff --git a/be/src/exec/operator/scan_operator.cpp b/be/src/exec/operator/scan_operator.cpp index 0874c525c98..885267d99a4 100644 --- a/be/src/exec/operator/scan_operator.cpp +++ b/be/src/exec/operator/scan_operator.cpp @@ -237,19 +237,8 @@ Status ScanLocalState<Derived>::open(RuntimeState* state) { RETURN_IF_ERROR(_on_runtime_filter_update(new_conjuncts)); } - // Disable condition cache in topn filter valid. TODO:: Try to support the topn filter in condition cache - if (p._enable_condition_cache && state->query_options().condition_cache_digest && - p._topn_filter_source_node_ids.empty()) { - _condition_cache_digest = state->query_options().condition_cache_digest; - for (auto& conjunct : _conjuncts) { - _condition_cache_digest = conjunct->get_digest(_condition_cache_digest); - if (!_condition_cache_digest) { - break; - } - } - } else { - _condition_cache_digest = 0; - } + // Compute the static predicate digest before _process_conjuncts() adds TopN filters. + _init_condition_cache(state); RETURN_IF_ERROR(_process_conjuncts(state)); @@ -268,6 +257,27 @@ Status ScanLocalState<Derived>::open(RuntimeState* state) { return status; } +template <typename Derived> +void ScanLocalState<Derived>::_init_condition_cache(RuntimeState* state) { + const auto& p = _parent->cast<typename Derived::Parent>(); + const bool has_topn_filter = !p._topn_filter_source_node_ids.empty(); + // TopN can consume a static predicate's cache, but its dynamic threshold must never + // contribute exclusions to that cache. External TopN scans keep condition cache disabled. + _condition_cache_write_enabled = !has_topn_filter; + _condition_cache_digest = 0; + if (!p._enable_condition_cache || !state->query_options().condition_cache_digest || + (has_topn_filter && !_supports_topn_condition_cache())) { + return; + } + _condition_cache_digest = state->query_options().condition_cache_digest; + for (const auto& conjunct : _conjuncts) { + _condition_cache_digest = conjunct->get_digest(_condition_cache_digest); + if (!_condition_cache_digest) { + break; + } + } +} + void ScanLocalStateBase::_init_slot_value_range( phmap::flat_hash_map<int, ColumnValueRangeType>& slot_id_to_value_range, SlotDescriptor* slot, const DataTypePtr& type_desc) { diff --git a/be/src/exec/operator/scan_operator.h b/be/src/exec/operator/scan_operator.h index 7c5e5eb11b9..76cdc442340 100644 --- a/be/src/exec/operator/scan_operator.h +++ b/be/src/exec/operator/scan_operator.h @@ -101,6 +101,7 @@ public: [[nodiscard]] std::string get_name() { return _parent->get_name(); } uint64_t get_condition_cache_digest() const { return _condition_cache_digest; } + bool condition_cache_write_enabled() const { return _condition_cache_write_enabled; } Status update_late_arrival_runtime_filter(RuntimeState* state, int& arrived_rf_num); @@ -150,6 +151,7 @@ protected: RuntimeFilterConsumerHelper _helper; // magic number as seed to generate hash value for condition cache uint64_t _condition_cache_digest = 0; + bool _condition_cache_write_enabled = true; // condition cache filter stats RuntimeProfile::Counter* _condition_cache_hit_counter = nullptr; RuntimeProfile::Counter* _condition_cache_filtered_rows_counter = nullptr; @@ -296,6 +298,8 @@ protected: friend class Scanner; Status _init_profile() override; + void _init_condition_cache(RuntimeState* state); + virtual bool _supports_topn_condition_cache() const { return false; } virtual Status _process_conjuncts(RuntimeState* state) { return _normalize_conjuncts(state); } virtual bool _should_push_down_common_expr(const VExprSPtr&) { return false; } diff --git a/be/src/exec/scan/olap_scanner.cpp b/be/src/exec/scan/olap_scanner.cpp index 212d182d37f..5f65823ebbd 100644 --- a/be/src/exec/scan/olap_scanner.cpp +++ b/be/src/exec/scan/olap_scanner.cpp @@ -114,6 +114,7 @@ OlapScanner::OlapScanner(ScanLocalStateBase* parent, OlapScanner::Params&& param _initial_file_cache_stats(std::move(params.initial_file_cache_stats)) { _tablet_reader_params.set_read_source(std::move(params.read_source), _state->skip_delete_bitmap()); + _tablet_reader_params.condition_cache_write_enabled = parent->condition_cache_write_enabled(); _has_prepared = false; _vector_search_params = params.state->get_vector_search_params(); } diff --git a/be/src/storage/iterators.h b/be/src/storage/iterators.h index cad005f5c3e..2274fed7a3f 100644 --- a/be/src/storage/iterators.h +++ b/be/src/storage/iterators.h @@ -174,6 +174,8 @@ public: CollectionStatisticsPtr collection_statistics; uint64_t condition_cache_digest = 0; + // A TopN scan may read the static predicate cache but cannot populate it. + bool condition_cache_write_enabled = true; }; struct CompactionSampleInfo { diff --git a/be/src/storage/rowset/beta_rowset_reader.cpp b/be/src/storage/rowset/beta_rowset_reader.cpp index 3cd7b88cc99..89b84de9075 100644 --- a/be/src/storage/rowset/beta_rowset_reader.cpp +++ b/be/src/storage/rowset/beta_rowset_reader.cpp @@ -204,6 +204,7 @@ Status BetaRowsetReader::get_segment_iterators(RowsetReaderContext* read_context _read_options.enable_unique_key_merge_on_write = _read_context->enable_unique_key_merge_on_write; _read_options.topn_filter_source_node_ids = _read_context->topn_filter_source_node_ids; + _read_options.condition_cache_write_enabled = _read_context->condition_cache_write_enabled; _read_options.read_orderby_key_reverse = _read_context->read_orderby_key_reverse; _read_options.use_insert_order_when_same = _read_context->use_insert_order_when_same; _read_options.read_changelog = _read_context->read_changelog; diff --git a/be/src/storage/rowset/rowset_reader_context.h b/be/src/storage/rowset/rowset_reader_context.h index 3c24a4c1b6e..c70ea2d28dd 100644 --- a/be/src/storage/rowset/rowset_reader_context.h +++ b/be/src/storage/rowset/rowset_reader_context.h @@ -110,6 +110,7 @@ struct RowsetReaderContext { std::shared_ptr<segment_v2::AnnTopNRuntime> ann_topn_runtime; uint64_t condition_cache_digest = 0; + bool condition_cache_write_enabled = true; // When true, push down value predicates for MOR tables bool enable_mor_value_predicate_pushdown = false; diff --git a/be/src/storage/segment/segment_iterator.cpp b/be/src/storage/segment/segment_iterator.cpp index cb77918e8c8..8d662d0a446 100644 --- a/be/src/storage/segment/segment_iterator.cpp +++ b/be/src/storage/segment/segment_iterator.cpp @@ -200,7 +200,7 @@ void SegmentIterator::_init_row_bitmap_by_condition_cache() { // Record rows filtered by condition cache hit _opts.stats->condition_cache_filtered_rows += filtered_blocks * SegmentIterator::CONDITION_CACHE_OFFSET; - } else { + } else if (_should_write_condition_cache()) { _condition_cache = std::make_shared<std::vector<bool>>( num_rows / CONDITION_CACHE_OFFSET + 1, false); } @@ -210,6 +210,22 @@ void SegmentIterator::_init_row_bitmap_by_condition_cache() { } } +void SegmentIterator::_write_condition_cache() { + if (!_should_write_condition_cache() || + (_opts.read_limit > 0 && _rows_returned >= _opts.read_limit)) { + return; + } + DCHECK(_condition_cache); + auto* condition_cache = ConditionCache::instance(); + ConditionCache::CacheKey cache_key(_opts.rowset_id, _segment->id(), + _opts.condition_cache_digest); + VLOG_DEBUG << "Condition cache insert, query id: " << print_id(_opts.runtime_state->query_id()) + << ", rowset id: " << _opts.rowset_id.to_string() + << ", segment id: " << _segment->id() + << ", cache digest: " << _opts.condition_cache_digest; + condition_cache->insert(cache_key, std::move(_condition_cache)); +} + // A fast range iterator for roaring bitmap. Output ranges use closed-open form, like [from, to). // Example: // input bitmap: [0 1 4 5 6 7 10 15 16 17 18 19] @@ -3118,18 +3134,7 @@ Status SegmentIterator::next_batch(Block* block) { block->replace_by_position(cid, type->create_column()); } - if (_opts.condition_cache_digest && !_find_condition_cache && - !(_opts.read_limit > 0 && _rows_returned >= _opts.read_limit)) { - auto* condition_cache = ConditionCache::instance(); - ConditionCache::CacheKey cache_key(_opts.rowset_id, _segment->id(), - _opts.condition_cache_digest); - VLOG_DEBUG << "Condition cache insert, query id: " - << print_id(_opts.runtime_state->query_id()) - << ", rowset id: " << _opts.rowset_id.to_string() - << ", segment id: " << _segment->id() - << ", cache digest: " << _opts.condition_cache_digest; - condition_cache->insert(cache_key, std::move(_condition_cache)); - } + _write_condition_cache(); return res; } @@ -3336,10 +3341,9 @@ Status SegmentIterator::_next_batch_internal(Block* block) { if (!_output_ordinals.empty()) { RETURN_IF_ERROR(_read_columns_by_rowids( _output_ordinals, _block_rowids, _sel_rowid_idx.data(), _selected_size, - &_current_columns, _opts.condition_cache_digest && !_find_condition_cache, - false)); + &_current_columns, _should_write_condition_cache(), false)); } else { - if (_opts.condition_cache_digest && !_find_condition_cache) { + if (_should_write_condition_cache()) { auto& condition_cache = *_condition_cache; for (size_t i = 0; i < _selected_size; ++i) { auto rowid = _block_rowids[_sel_rowid_idx[i]]; diff --git a/be/src/storage/segment/segment_iterator.h b/be/src/storage/segment/segment_iterator.h index 0de8a0b6752..c0919a170d0 100644 --- a/be/src/storage/segment/segment_iterator.h +++ b/be/src/storage/segment/segment_iterator.h @@ -335,6 +335,11 @@ private: void _prepare_score_column_materialization(); void _init_row_bitmap_by_condition_cache(); + bool _should_write_condition_cache() const { + return _opts.condition_cache_digest && _opts.condition_cache_write_enabled && + !_find_condition_cache; + } + void _write_condition_cache(); void _init_segment_prefetchers(); diff --git a/be/src/storage/tablet/tablet_reader.cpp b/be/src/storage/tablet/tablet_reader.cpp index e26c87f2a73..1066d6d54cd 100644 --- a/be/src/storage/tablet/tablet_reader.cpp +++ b/be/src/storage/tablet/tablet_reader.cpp @@ -199,6 +199,7 @@ Status TabletReader::_capture_rs_readers(const ReaderParams& read_params) { _reader_context.ann_topn_runtime = read_params.ann_topn_runtime; _reader_context.condition_cache_digest = read_params.condition_cache_digest; + _reader_context.condition_cache_write_enabled = read_params.condition_cache_write_enabled; _reader_context.all_access_paths = read_params.all_access_paths; _reader_context.predicate_access_paths = read_params.predicate_access_paths; diff --git a/be/src/storage/tablet/tablet_reader.h b/be/src/storage/tablet/tablet_reader.h index 62159f6572d..73d2c7cc9f6 100644 --- a/be/src/storage/tablet/tablet_reader.h +++ b/be/src/storage/tablet/tablet_reader.h @@ -223,6 +223,7 @@ public: std::shared_ptr<segment_v2::AnnTopNRuntime> ann_topn_runtime; uint64_t condition_cache_digest = 0; + bool condition_cache_write_enabled = true; // General LIMIT budget forwarded to SegmentIterator. -1 means no limit. int64_t general_read_limit = -1; diff --git a/be/test/exec/scan/scan_operator_test.cpp b/be/test/exec/scan/scan_operator_test.cpp index 4554c1c03f5..75ff780a3cf 100644 --- a/be/test/exec/scan/scan_operator_test.cpp +++ b/be/test/exec/scan/scan_operator_test.cpp @@ -21,7 +21,9 @@ #include <gtest/gtest.h> #include "common/object_pool.h" +#include "exec/operator/file_scan_operator.h" #include "exec/operator/olap_scan_operator.h" +#include "exprs/vliteral.h" #include "runtime/descriptors.h" namespace doris { @@ -109,4 +111,108 @@ TEST_F(ScanOperatorTest, adaptive_pipeline_task_serial_read_on_limit) { // Without conjuncts, enable_adaptive_pipeline_task_serial_read_on_limit is false ASSERT_EQ(scan_operator->_should_run_serial, false); } -} // namespace doris \ No newline at end of file + +namespace { + +VExprContextSPtr cacheable_conjunct() { + TExprNode node; + node.__set_node_type(TExprNodeType::BOOL_LITERAL); + node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN)); + node.__set_is_nullable(false); + TBoolLiteral literal; + literal.__set_value(true); + node.__set_bool_literal(literal); + return VExprContext::create_shared(VLiteral::create_shared(node)); +} + +class UndigestibleLiteral : public VLiteral { +public: + explicit UndigestibleLiteral(const TExprNode& node) : VLiteral(node, false) {} + uint64_t get_digest(uint64_t) const override { return 0; } +}; + +} // namespace + +TEST_F(ScanOperatorTest, topn_reuses_static_digest_without_writing) { + TPlanNode tnode; + tnode.__set_enable_condition_cache(true); + tnode.__set_row_tuples({0, 1}); + auto parent = std::make_unique<OlapScanOperatorX>(obj_pool.get(), tnode, 0, *descs, 1, + TQueryCacheParam {}); + auto local = OlapScanLocalState::create_shared(state.get(), parent.get()); + auto conjunct = cacheable_conjunct(); + local->_conjuncts = {conjunct}; + TQueryOptions options; + options.__set_condition_cache_digest(12345); + state->set_query_options(options); + + local->_init_condition_cache(state.get()); + const auto static_digest = local->get_condition_cache_digest(); + ASSERT_NE(static_digest, 0); + EXPECT_EQ(static_digest, conjunct->get_digest(12345)); + EXPECT_TRUE(local->condition_cache_write_enabled()); + + // Source IDs are query-local and must not change the static predicate cache key. + parent->_topn_filter_source_node_ids = {7}; + local->_init_condition_cache(state.get()); + EXPECT_EQ(local->get_condition_cache_digest(), static_digest); + EXPECT_FALSE(local->condition_cache_write_enabled()); + + parent->_topn_filter_source_node_ids = {11, 13}; + local->_init_condition_cache(state.get()); + EXPECT_EQ(local->get_condition_cache_digest(), static_digest); + EXPECT_FALSE(local->condition_cache_write_enabled()); +} + +TEST_F(ScanOperatorTest, topn_preserves_digest_safety_gates) { + TPlanNode tnode; + tnode.__set_enable_condition_cache(true); + tnode.__set_row_tuples({0, 1}); + auto parent = std::make_unique<OlapScanOperatorX>(obj_pool.get(), tnode, 0, *descs, 1, + TQueryCacheParam {}); + parent->_topn_filter_source_node_ids = {7}; + auto local = OlapScanLocalState::create_shared(state.get(), parent.get()); + auto conjunct = cacheable_conjunct(); + local->_conjuncts = {conjunct}; + + TQueryOptions options; + options.__set_condition_cache_digest(0); + state->set_query_options(options); + local->_init_condition_cache(state.get()); + EXPECT_EQ(local->get_condition_cache_digest(), 0); + + options.__set_condition_cache_digest(12345); + state->set_query_options(options); + parent->_enable_condition_cache = false; + local->_init_condition_cache(state.get()); + EXPECT_EQ(local->get_condition_cache_digest(), 0); + + parent->_enable_condition_cache = true; + local->_conjuncts = {VExprContext::create_shared( + std::make_shared<UndigestibleLiteral>(conjunct->root()->clone_texpr_node()))}; + local->_init_condition_cache(state.get()); + EXPECT_EQ(local->get_condition_cache_digest(), 0); +} + +TEST_F(ScanOperatorTest, external_topn_condition_cache_remains_disabled) { + TPlanNode tnode; + tnode.__set_enable_condition_cache(true); + tnode.__set_row_tuples({0, 1}); + auto parent = std::make_unique<FileScanOperatorX>(obj_pool.get(), tnode, 0, *descs, 1); + auto local = FileScanLocalState::create_shared(state.get(), parent.get()); + local->_conjuncts = {cacheable_conjunct()}; + TQueryOptions options; + options.__set_condition_cache_digest(12345); + state->set_query_options(options); + + local->_init_condition_cache(state.get()); + EXPECT_NE(local->get_condition_cache_digest(), 0); + EXPECT_TRUE(local->condition_cache_write_enabled()); + + parent->_topn_filter_source_node_ids = {7}; + local->_init_condition_cache(state.get()); + EXPECT_EQ(local->get_condition_cache_digest(), 0); + EXPECT_FALSE(local->condition_cache_write_enabled()); +} + +} // namespace doris diff --git a/be/test/storage/segment/segment_iterator_condition_cache_test.cpp b/be/test/storage/segment/segment_iterator_condition_cache_test.cpp new file mode 100644 index 00000000000..be5452b1713 --- /dev/null +++ b/be/test/storage/segment/segment_iterator_condition_cache_test.cpp @@ -0,0 +1,160 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include <gtest/gtest.h> + +#include <memory> +#include <vector> + +#include "runtime/exec_env.h" +#include "runtime/runtime_state.h" +#include "storage/olap_common.h" +#include "storage/predicate/null_predicate.h" +#include "storage/segment/condition_cache.h" +#include "storage/segment/segment.h" +#include "storage/segment/segment_iterator.h" +#include "storage/tablet/tablet_schema.h" + +namespace doris::segment_v2 { + +class SegmentIteratorConditionCacheTest : public testing::Test { +protected: + void SetUp() override { + _previous_cache = ExecEnv::GetInstance()->get_condition_cache(); + _cache.reset(ConditionCache::create_global_cache(1024 * 1024, 4)); + ExecEnv::GetInstance()->_condition_cache = _cache.get(); + + TabletSchemaPB schema; + schema.set_keys_type(KeysType::DUP_KEYS); + auto* column = schema.add_column(); + column->set_unique_id(0); + column->set_name("k1"); + column->set_type("INT"); + column->set_is_key(true); + column->set_is_nullable(true); + _tablet_schema = std::make_shared<TabletSchema>(); + _tablet_schema->init_from_pb(schema); + _read_schema = std::make_shared<ReadSchema>(_tablet_schema->columns()); + _segment = + std::make_shared<Segment>(7, RowsetId(), _tablet_schema, InvertedIndexFileInfo()); + _segment->_num_rows = 3 * ConditionCacheContext::GRANULE_SIZE; + } + + void TearDown() override { ExecEnv::GetInstance()->_condition_cache = _previous_cache; } + + std::unique_ptr<SegmentIterator> make_iterator(bool write_enabled) { + StorageReadOptions opts(_stats); + opts.runtime_state = &_runtime_state; + opts.condition_cache_digest = 12345; + opts.condition_cache_write_enabled = write_enabled; + auto iterator = std::make_unique<SegmentIterator>(_segment, _read_schema, opts); + iterator->_col_predicates = { + NullPredicate::create_shared(0, "k1", false, PrimitiveType::TYPE_INT)}; + iterator->_row_bitmap.addRange(0, _segment->num_rows()); + return iterator; + } + + ConditionCache::CacheKey cache_key() { return {RowsetId(), 7, 12345}; } + + ConditionCache* _previous_cache = nullptr; + std::unique_ptr<ConditionCache> _cache; + TabletSchemaSPtr _tablet_schema; + ReadSchemaSPtr _read_schema; + std::shared_ptr<Segment> _segment; + RuntimeState _runtime_state; + OlapReaderStatistics _stats; +}; + +TEST_F(SegmentIteratorConditionCacheTest, ReadOnlyMissDoesNotCollectOrPublish) { + auto iterator = make_iterator(false); + iterator->_init_row_bitmap_by_condition_cache(); + EXPECT_FALSE(iterator->_find_condition_cache); + EXPECT_EQ(iterator->_condition_cache, nullptr); + EXPECT_EQ(iterator->_row_bitmap.cardinality(), _segment->num_rows()); + EXPECT_FALSE(iterator->_should_write_condition_cache()); + + // This is also the finalization path when a TopN scan reaches EOF. It must neither + // publish threshold-based exclusions nor dereference a missing MISS bitmap. + iterator->_write_condition_cache(); + ConditionCacheHandle handle; + EXPECT_FALSE(_cache->lookup(cache_key(), &handle)); +} + +TEST_F(SegmentIteratorConditionCacheTest, ReadOnlyHitUsesStaticExclusionsWithoutMutatingCache) { + auto bitmap = std::make_shared<std::vector<bool>>( + std::initializer_list<bool> {false, true, false, false}); + _cache->insert(cache_key(), bitmap); + auto iterator = make_iterator(false); + iterator->_init_row_bitmap_by_condition_cache(); + + ASSERT_TRUE(iterator->_find_condition_cache); + EXPECT_EQ(iterator->_condition_cache, nullptr); + EXPECT_EQ(iterator->_row_bitmap.cardinality(), ConditionCacheContext::GRANULE_SIZE); + EXPECT_TRUE(iterator->_row_bitmap.contains(ConditionCacheContext::GRANULE_SIZE)); + EXPECT_FALSE(iterator->_row_bitmap.contains(0)); + EXPECT_FALSE(iterator->_row_bitmap.contains(2 * ConditionCacheContext::GRANULE_SIZE)); + EXPECT_EQ(_stats.condition_cache_hit_seg_nums, 1); + EXPECT_FALSE(iterator->_should_write_condition_cache()); + + iterator->_write_condition_cache(); + ConditionCacheHandle handle; + ASSERT_TRUE(_cache->lookup(cache_key(), &handle)); + EXPECT_EQ(handle.get_filter_result(), bitmap); + EXPECT_EQ(*bitmap, (std::vector<bool> {false, true, false, false})); +} + +TEST_F(SegmentIteratorConditionCacheTest, OrdinaryScanStillPopulatesCacheForReadOnlyScan) { + auto writer = make_iterator(true); + writer->_init_row_bitmap_by_condition_cache(); + ASSERT_NE(writer->_condition_cache, nullptr); + EXPECT_TRUE(writer->_should_write_condition_cache()); + (*writer->_condition_cache)[1] = true; + writer->_write_condition_cache(); + + auto reader = make_iterator(false); + reader->_init_row_bitmap_by_condition_cache(); + EXPECT_TRUE(reader->_find_condition_cache); + EXPECT_EQ(reader->_row_bitmap.cardinality(), ConditionCacheContext::GRANULE_SIZE); + EXPECT_EQ(reader->_condition_cache, nullptr); +} + +TEST_F(SegmentIteratorConditionCacheTest, DisabledDigestDoesNotReadOrWrite) { + auto iterator = make_iterator(true); + iterator->_opts.condition_cache_digest = 0; + iterator->_init_row_bitmap_by_condition_cache(); + EXPECT_FALSE(iterator->_find_condition_cache); + EXPECT_EQ(iterator->_condition_cache, nullptr); + EXPECT_FALSE(iterator->_should_write_condition_cache()); + iterator->_write_condition_cache(); + ConditionCacheHandle handle; + EXPECT_FALSE(_cache->lookup(cache_key(), &handle)); +} + +TEST_F(SegmentIteratorConditionCacheTest, ReadLimitDoesNotPublishPartialBitmap) { + auto iterator = make_iterator(true); + iterator->_init_row_bitmap_by_condition_cache(); + ASSERT_NE(iterator->_condition_cache, nullptr); + (*iterator->_condition_cache)[0] = true; + iterator->_opts.read_limit = 1; + iterator->_rows_returned = 1; + + iterator->_write_condition_cache(); + ConditionCacheHandle handle; + EXPECT_FALSE(_cache->lookup(cache_key(), &handle)); +} + +} // namespace doris::segment_v2 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
