github-actions[bot] commented on code in PR #67180: URL: https://github.com/apache/doris/pull/67180#discussion_r4090108007
########## regression-test/suites/inverted_index_p0/test_phrase_candidate_pushdown.groovy: ########## @@ -0,0 +1,115 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// Phrase queries restricted to the scan candidates must return the same rows as the unrestricted +// evaluation, for CLucene (V2) and SNII indexes alike. "k < 5" and "tag = 1" keep 5 of 100 rows, +// below the default 0.3 candidate ratio, so the phrase conjuncts run under a candidate set. Each +// query runs first with pushdown (a cold query cache), then with pushdown disabled, then with +// pushdown again over the cached full-segment results. +// It flips a BE config, so it must not share the cluster with other suites. +suite("test_phrase_candidate_pushdown", "p0,nonConcurrent") { + sql "SET enable_inverted_index_query = true" Review Comment: [P2] Disable the FE SQL cache before comparing these paths. `enable_sql_cache` defaults to true, and this suite reruns the same query strings after changing only a BE config. The 30-second quiet window only makes interception timing-dependent: once a first-phase query becomes cache-eligible on a slow run, the ratio-0 and final ratio-0.3 assertions can replay rows without constructing a SegmentIterator, so they do not prove either the unrestricted BE path or the intended full inverted-index-cache hit. Please set `enable_sql_cache = false` for the suite while leaving `enable_inverted_index_query_cache` enabled. ########## be/test/storage/index/snii/bench/phrase_candidate_pushdown_bench_test.cpp: ########## @@ -0,0 +1,353 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Candidate pushdown into multi-term phrase queries, measured for both inverted index storage +// formats. One synthetic log corpus is indexed through the production column writer as CLucene (V2) +// and as SNII. Every phrase then runs through the production reader with and without +// IndexQueryContext::candidate_rows, and every restricted result is checked against the full result +// intersected with the candidates. +// +// The test is DISABLED_ so CI never runs it; it is still compiled into doris_be_test. Use a RELEASE +// UT build (BUILD_TYPE_UT=RELEASE in custom_env.sh) for representative numbers: +// +// GTEST_ALSO_RUN_DISABLED_TESTS=1 ./run-be-ut.sh --run \ +// --filter='*PhraseCandidatePushdownBench*' -j <N> +// +// PHRASE_CANDIDATE_BENCH_DOCS sets the segment size (default 200000) and +// PHRASE_CANDIDATE_BENCH_ITERATIONS the samples per measurement (default 10). Times are medians of +// per-query thread CPU time, which moves far less than wall time on a shared machine. + +#include <fmt/format.h> +#include <gen_cpp/PaloInternalService_types.h> +#include <gtest/gtest.h> + +#include <algorithm> +#include <cstdint> +#include <cstdlib> +#include <ctime> +#include <iomanip> +#include <iostream> +#include <memory> +#include <random> +#include <string> +#include <string_view> +#include <vector> + +#include "io/fs/local_file_system.h" +#include "runtime/exec_env.h" +#include "runtime/runtime_state.h" +#include "storage/index/index_file_reader.h" +#include "storage/index/index_file_writer.h" +#include "storage/index/index_query_context.h" +#include "storage/index/index_writer.h" +#include "storage/index/inverted/inverted_index_cache.h" +#include "storage/index/inverted/inverted_index_desc.h" +#include "storage/index/inverted/inverted_index_reader.h" +#include "storage/index/snii/snii_index_reader.h" +#include "storage/olap_common.h" +#include "storage/tablet/tablet_schema.h" +#include "util/slice.h" + +namespace doris::segment_v2 { +namespace { + +constexpr const char* kBenchDir = "./ut_dir/phrase_candidate_pushdown_bench"; +constexpr const char* kColumnName = "1"; +constexpr double kCandidateRatios[] = {0.001, 0.01, 0.05, 0.1, 0.224, 0.3, 0.5}; + +struct BenchQuery { + const char* label; + InvertedIndexQueryType type; + const char* text; +}; + +// Frequent two- and four-term phrases, a frequent lead with an expanding numeric tail, a +// log-search shape with a long lead, and a rare phrase whose result is far below any candidate set. +constexpr BenchQuery kQueries[] = {{.label = "exact_2", + .type = InvertedIndexQueryType::MATCH_PHRASE_QUERY, + .text = "retry attempt"}, + {.label = "exact_4", + .type = InvertedIndexQueryType::MATCH_PHRASE_QUERY, + .text = "retry attempt 2 job"}, + {.label = "prefix_2", + .type = InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY, + .text = "order 12"}, + {.label = "prefix_5", + .type = InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY, + .text = "retry attempt 1 job 88"}, + {.label = "rare_exact", + .type = InvertedIndexQueryType::MATCH_PHRASE_QUERY, + .text = "request 424242 completed"}}; Review Comment: [P2] Make `rare_exact` reach the rare-posting path. `Request` rows are emitted only when `r % 4 == 3`; because 1,000,000 is divisible by four, every generated request id is also 3 modulo four, but 424242 is 2 modulo four. This phrase is therefore absent, so both results are empty and SNII exits during missing-term planning instead of benchmarking the rare-term filtering strategy. Please seed a deterministic matching row and assert that the unrestricted result is non-empty. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
