This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new c8ef2c01410 branch-4.2: [refactor](index) Move collection similarity
to shared index code (#68602) (#68678)
c8ef2c01410 is described below
commit c8ef2c01410cd7338c5aba802026b997738cf262
Author: Jack <[email protected]>
AuthorDate: Thu Oct 15 19:12:10 2026 +0800
branch-4.2: [refactor](index) Move collection similarity to shared index
code (#68602) (#68678)
### What problem does this PR solve?
Issue Number: None
Related PR: #68602 (master)
Problem Summary:
`CollectionSimilarity` is used by both inverted-index and SNII queries
but lived under `storage/compaction`. This backport moves its
implementation, header and test to `storage/index` and updates every
direct include. The scoring code is unchanged. The 4.2 compile-check
markers and surrounding branch-specific declarations remain intact.
### Release note
None. This changes source layout only.
### Check List (For Author)
- Test:
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No. Source layout only.
### Backport audit
Source merge commit: `7c2f99b1bc03c0b43a877fe90837172f4de637c3`;
backport commit: `5c0e125b07240d8fd4344bbe0793ddc533ae657b`.
| Source file | Source hunk | Disposition |
| --- | --- | --- |
| `be/src/storage/index/collection_similarity.cpp` | `@@ -18 +18 @@` |
Ported: Source include-path change is present. |
| `be/src/storage/index/collection_similarity.cpp` | `@@ -168 +168 @@
void CollectionSimilarity::find_top_k_scores(const roaring::Roaring*
row_bitmap,` | Adapted: 4.2 compile-check end marker retained. |
| `be/src/storage/index/collection_similarity.h` | `R100 (no @@ hunk)` |
Adapted: File moved; 4.2 compile-check markers retained. |
| `be/src/storage/index/index_query_context.h` | `@@ -23 +23 @@` |
Adapted: Source include path updated; 4.2 compile-check markers and
local type includes retained. |
| `be/src/storage/index/inverted/query_v2/collect/doc_set_collector.h` |
`all source hunks` | Ported(verbatim): Target file matches source blob.
|
| `be/src/storage/index/inverted/query_v2/collect/top_k_collector.h` |
`@@ -29 +29 @@` | Ported: Source include-path change is present. |
|
`be/src/storage/index/inverted/query_v2/scored_bit_set_query/scored_bit_set_scorer.h`
| `all source hunks` | Ported(verbatim): Target file matches source
blob. |
| `be/src/storage/segment/segment_iterator.cpp` | `@@ -80 +79,0 @@` |
Ported: Source include-path change is present. |
| `be/src/storage/segment/segment_iterator.cpp` | `@@ -85,0 +85 @@` |
Ported: Source include-path change is present. |
| `be/test/storage/index/collection_similarity_test.cpp` | `all source
hunks` | Ported(verbatim): Target file matches source blob. |
|
`be/test/storage/index/inverted/inverted_index_reader_analysis_purpose_test.cpp`
| `@@ -32 +32 @@` | Ported: Source include-path change is present. |
| `be/test/storage/index/inverted/query/phrase_prefix_query_test.cpp` |
`all source hunks` | Ported(verbatim): Target file matches source blob.
|
| `be/test/storage/index/inverted/query/query_helper_test.cpp` | `all
source hunks` | Ported(verbatim): Target file matches source blob. |
| `be/test/storage/index/snii/snii_index_reader_candidate_test.cpp` |
`all source hunks` | Ported(verbatim): Target file matches source blob.
|
`git range-diff` shows only the expected branch differences: the 4.2
compile-check markers remain in the moved files and index query context;
the 4.2 segment iterator and collector headers retain unrelated
branch-specific declarations. Every source include-path change is
present.
### Validation
- BE ASAN build: passed.
- BE ASAN unit tests: 58/58 passed across `CollectionSimilarityTest`
(27), `SniiIndexReaderCandidateTest` (9),
`InvertedIndexReaderAnalysisPurposeTest` (5), `PhrasePrefixQueryTest`
(7), and `QueryHelperTest` (10). No sanitizer report.
- Isolated regression:
`inverted_index_p0/test_phrase_candidate_pushdown` and
`storage_format/test_storage_format_snii_norms`; one suite each, zero
failures.
- Complete PR diff English check: passed with no data exceptions.
- Merged-tree clang-format 16 preflight: passed against
`origin/branch-4.2` at `7e4909b904ff49fc799e5a4efa02f03fe4a8638d`.
- Advisory clang-tidy did not complete: the branch's warning-as-error
setting promoted existing diagnostics in `segment_iterator.cpp`, and
Clang 16 reported existing `jni-util.h` static assertions. No tidy pass
is claimed.
---
be/src/storage/{compaction => index}/collection_similarity.cpp | 4 ++--
be/src/storage/{compaction => index}/collection_similarity.h | 0
be/src/storage/index/index_query_context.h | 7 +++++--
be/src/storage/index/inverted/query_v2/collect/doc_set_collector.h | 2 +-
be/src/storage/index/inverted/query_v2/collect/top_k_collector.h | 2 +-
.../inverted/query_v2/scored_bit_set_query/scored_bit_set_scorer.h | 2 +-
be/src/storage/segment/segment_iterator.cpp | 2 +-
.../storage/{compaction => index}/collection_similarity_test.cpp | 6 +++---
.../index/inverted/inverted_index_reader_analysis_purpose_test.cpp | 2 +-
be/test/storage/index/inverted/query/phrase_prefix_query_test.cpp | 2 +-
be/test/storage/index/inverted/query/query_helper_test.cpp | 4 ++--
be/test/storage/index/snii/snii_index_reader_candidate_test.cpp | 2 +-
12 files changed, 19 insertions(+), 16 deletions(-)
diff --git a/be/src/storage/compaction/collection_similarity.cpp
b/be/src/storage/index/collection_similarity.cpp
similarity index 98%
rename from be/src/storage/compaction/collection_similarity.cpp
rename to be/src/storage/index/collection_similarity.cpp
index a4e5e64a807..03ed169006b 100644
--- a/be/src/storage/compaction/collection_similarity.cpp
+++ b/be/src/storage/index/collection_similarity.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "core/column/column_nullable.h"
#include "core/column/column_vector.h"
@@ -167,4 +167,4 @@ void CollectionSimilarity::find_top_k_scores(const
roaring::Roaring* row_bitmap,
}
#include "common/compile_check_end.h"
-} // namespace doris
\ No newline at end of file
+} // namespace doris
diff --git a/be/src/storage/compaction/collection_similarity.h
b/be/src/storage/index/collection_similarity.h
similarity index 100%
rename from be/src/storage/compaction/collection_similarity.h
rename to be/src/storage/index/collection_similarity.h
diff --git a/be/src/storage/index/index_query_context.h
b/be/src/storage/index/index_query_context.h
index 3e529a8078b..4f458bf36d7 100644
--- a/be/src/storage/index/index_query_context.h
+++ b/be/src/storage/index/index_query_context.h
@@ -17,7 +17,10 @@
#pragma once
-#include "storage/compaction/collection_similarity.h"
+#include <memory>
+#include <roaring/roaring.hh>
+
+#include "storage/index/collection_similarity.h"
#include "storage/index/inverted/similarity/collection_statistics.h"
namespace doris::segment_v2 {
@@ -95,4 +98,4 @@ struct IndexQueryContext {
using IndexQueryContextPtr = std::shared_ptr<IndexQueryContext>;
#include "common/compile_check_end.h"
-} // namespace doris::segment_v2
\ No newline at end of file
+} // namespace doris::segment_v2
diff --git a/be/src/storage/index/inverted/query_v2/collect/doc_set_collector.h
b/be/src/storage/index/inverted/query_v2/collect/doc_set_collector.h
index bad94d724bc..4cc0f09c9a7 100644
--- a/be/src/storage/index/inverted/query_v2/collect/doc_set_collector.h
+++ b/be/src/storage/index/inverted/query_v2/collect/doc_set_collector.h
@@ -20,7 +20,7 @@
#include <memory>
#include <roaring/roaring.hh>
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/inverted/query_v2/weight.h"
namespace doris::segment_v2::inverted_index::query_v2 {
diff --git a/be/src/storage/index/inverted/query_v2/collect/top_k_collector.h
b/be/src/storage/index/inverted/query_v2/collect/top_k_collector.h
index 889aff9fa55..809c0f5a87a 100644
--- a/be/src/storage/index/inverted/query_v2/collect/top_k_collector.h
+++ b/be/src/storage/index/inverted/query_v2/collect/top_k_collector.h
@@ -27,7 +27,7 @@
#include <string>
#include <vector>
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/inverted/query_v2/weight.h"
namespace doris::segment_v2::inverted_index::query_v2 {
diff --git
a/be/src/storage/index/inverted/query_v2/scored_bit_set_query/scored_bit_set_scorer.h
b/be/src/storage/index/inverted/query_v2/scored_bit_set_query/scored_bit_set_scorer.h
index 59aae9a09f1..503606d91a8 100644
---
a/be/src/storage/index/inverted/query_v2/scored_bit_set_query/scored_bit_set_scorer.h
+++
b/be/src/storage/index/inverted/query_v2/scored_bit_set_query/scored_bit_set_scorer.h
@@ -21,7 +21,7 @@
#include <utility>
#include "roaring/roaring.hh"
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/inverted/query_v2/bit_set_query/bit_set_scorer.h"
#include "storage/index/inverted/query_v2/scorer.h"
diff --git a/be/src/storage/segment/segment_iterator.cpp
b/be/src/storage/segment/segment_iterator.cpp
index bec2c7f34ba..9cbb1d247a1 100644
--- a/be/src/storage/segment/segment_iterator.cpp
+++ b/be/src/storage/segment/segment_iterator.cpp
@@ -79,12 +79,12 @@
#include "runtime/runtime_predicate.h"
#include "runtime/runtime_state.h"
#include "runtime/thread_context.h"
-#include "storage/compaction/collection_similarity.h"
#include "storage/id_manager.h"
#include "storage/index/ann/ann_index.h"
#include "storage/index/ann/ann_index_iterator.h"
#include "storage/index/ann/ann_index_reader.h"
#include "storage/index/ann/ann_topn_runtime.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/index_file_reader.h"
#include "storage/index/index_iterator.h"
#include "storage/index/index_query_context.h"
diff --git a/be/test/storage/compaction/collection_similarity_test.cpp
b/be/test/storage/index/collection_similarity_test.cpp
similarity index 99%
rename from be/test/storage/compaction/collection_similarity_test.cpp
rename to be/test/storage/index/collection_similarity_test.cpp
index 7b5bf796353..3657b7089a0 100644
--- a/be/test/storage/compaction/collection_similarity_test.cpp
+++ b/be/test/storage/index/collection_similarity_test.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include <gtest/gtest.h>
@@ -643,7 +643,7 @@ TEST_F(CollectionSimilarityTest, LargeDataSparseBitmapTest)
{
for (size_t i = 0; i < NUM_SCORED_ROWS; ++i) {
float score = static_cast<float>(i + 1) /
static_cast<float>(NUM_SCORED_ROWS);
- similarity->collect(static_cast<uint32_t>(i * 100), score); // 稀疏分布
+ similarity->collect(static_cast<uint32_t>(i * 100), score);
}
std::vector<uint32_t> all_ids;
@@ -763,4 +763,4 @@ TEST_F(CollectionSimilarityTest,
LargeDataBoundaryThresholdTest) {
}
}
-} // namespace doris
\ No newline at end of file
+} // namespace doris
diff --git
a/be/test/storage/index/inverted/inverted_index_reader_analysis_purpose_test.cpp
b/be/test/storage/index/inverted/inverted_index_reader_analysis_purpose_test.cpp
index e0e9620e964..6d55def1516 100644
---
a/be/test/storage/index/inverted/inverted_index_reader_analysis_purpose_test.cpp
+++
b/be/test/storage/index/inverted/inverted_index_reader_analysis_purpose_test.cpp
@@ -29,7 +29,7 @@
#include "io/fs/local_file_system.h"
#include "runtime/exec_env.h"
#include "runtime/runtime_state.h"
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/index_file_reader.h"
#include "storage/index/inverted/analyzer/analyzer.h"
#include "storage/index/inverted/analyzer/analyzer_provider.h"
diff --git a/be/test/storage/index/inverted/query/phrase_prefix_query_test.cpp
b/be/test/storage/index/inverted/query/phrase_prefix_query_test.cpp
index 94b0eca42b3..07863f708a8 100644
--- a/be/test/storage/index/inverted/query/phrase_prefix_query_test.cpp
+++ b/be/test/storage/index/inverted/query/phrase_prefix_query_test.cpp
@@ -23,7 +23,7 @@
#include "io/fs/local_file_system.h"
#include "runtime/exec_env.h"
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/index_file_reader.h"
#include "storage/index/index_file_writer.h"
#include "storage/index/inverted/inverted_index_cache.h"
diff --git a/be/test/storage/index/inverted/query/query_helper_test.cpp
b/be/test/storage/index/inverted/query/query_helper_test.cpp
index 95c1a3a6fad..51a47bdcbd9 100644
--- a/be/test/storage/index/inverted/query/query_helper_test.cpp
+++ b/be/test/storage/index/inverted/query/query_helper_test.cpp
@@ -22,7 +22,7 @@
#include <memory>
#include "common/be_mock_util.h"
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/index_query_context.h"
#include "storage/index/inverted/similarity/similarity.h"
#include "storage/index/inverted/util/mock_iterator.h"
@@ -276,4 +276,4 @@ TEST_F(QueryHelperTest, CollectMixedScenario) {
ASSERT_TRUE(_context->collection_similarity->_bm25_scores.count(1002) > 0);
ASSERT_TRUE(_context->collection_similarity->_bm25_scores.count(2000) > 0);
ASSERT_TRUE(_context->collection_similarity->_bm25_scores.count(2001) > 0);
-}
\ No newline at end of file
+}
diff --git a/be/test/storage/index/snii/snii_index_reader_candidate_test.cpp
b/be/test/storage/index/snii/snii_index_reader_candidate_test.cpp
index a1f4f71c36d..d58512f82b8 100644
--- a/be/test/storage/index/snii/snii_index_reader_candidate_test.cpp
+++ b/be/test/storage/index/snii/snii_index_reader_candidate_test.cpp
@@ -31,7 +31,7 @@
#include "io/fs/local_file_system.h"
#include "runtime/exec_env.h"
#include "runtime/runtime_state.h"
-#include "storage/compaction/collection_similarity.h"
+#include "storage/index/collection_similarity.h"
#include "storage/index/index_file_reader.h"
#include "storage/index/index_query_context.h"
#include "storage/index/inverted/analyzer/custom_analyzer.h"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
