matanper opened a new issue, #70588: URL: https://github.com/apache/doris/issues/70588
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4. master (`42b148bac33`) has the same code. ### What's Wrong? On a column whose fulltext inverted index was built without `"support_phrase" = "true"`, the same `MATCH_PHRASE` predicate either fails the query or returns correct rows by scanning every row. Which one happens depends on whether the predicate is evaluated through the index or as an expression: - **Pushed down to the index:** the query fails with `[INDEX_INVALID_PARAMETERS] phrase queries require setting support_phrase = true`. - **Not pushed down** (for example inside an `OR` with a non-indexed predicate, above a join or projection, or on segments that have no index for the column): the query succeeds, evaluating the phrase row by row. So a query can fail or succeed depending on the plan shape, and on which segments of the scan carry the index. Adding an unrelated `OR` turns a failing query into a full scan that succeeds. ### Why `FunctionMatchBase::evaluate_inverted_index` (`be/src/exprs/function/match.cpp:340-352` on master) returns an error for phrase functions when the fulltext reader lacks phrase support: ```cpp if (... segment_v2::IndexReaderHelper::is_fulltext_index(reader.value()) && !segment_v2::IndexReaderHelper::is_support_phrase(reader.value())) { return Status::Error<ErrorCode::INDEX_INVALID_PARAMETERS>( "phrase queries require setting support_phrase = true"); } ``` The row path, `FunctionMatchPhrase::execute_match`, has no such requirement. It tokenizes each row with the same analyzer and checks for consecutive tokens, gated only by `enable_match_without_inverted_index` (`FunctionMatchBase::check`, `:551-555`, default true). The caller already has a clean way to decline: `VExpr::_evaluate_inverted_index` records an index result only when the returned bitmap is non-empty. Returning OK without a bitmap makes the expression evaluate row by row on the remaining rows. `evaluate_inverted_index` already does exactly that for a segment without the index (`iter == nullptr`). ### What You Expected? `MATCH_PHRASE` returns the same result whether or not it is pushed to an index without phrase support. An index that cannot answer a phrase query declines, and the predicate is evaluated row by row, as it is on a segment without the index. When `enable_match_without_inverted_index = false`, the existing "not support execute_match" error still applies. ### How to Reproduce? ```sql CREATE TABLE t ( id BIGINT, msg STRING, INDEX idx_msg (msg) USING INVERTED PROPERTIES ("parser" = "unicode") -- no support_phrase ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ("replication_num" = "1"); INSERT INTO t VALUES (1, 'failed password for root'), (2, 'password failed for root'); SELECT id FROM t WHERE msg MATCH_PHRASE 'failed password'; -- ERROR: phrase queries require setting support_phrase = true SELECT id FROM t WHERE msg MATCH_PHRASE 'failed password' OR id = -1; -- id has no inverted index, so the OR is evaluated as an expression and -- FunctionMatchPhrase::execute_match answers the phrase row by row (expected: 1) ``` We saw both outcomes in production on the same predicate: some statements failed with this error, while another statement with the same predicate completed after scanning every row of the column. ### Anything Else? Proposed fix: in `FunctionMatchBase::evaluate_inverted_index`, when a phrase function meets a fulltext reader without phrase support, return `Status::OK()` without setting the bitmap, the same as when the segment has no index. The expression is then evaluated row by row. Two notes: - The row path does not parse phrase options such as slop (`'a b ~2'`); only the index phrase query understands them. An index without positions never answered phrase queries, so this adds no new semantics, but it could be documented. - A follow-up could make the fallback cheap. When a phrase meets a fulltext index without positions, the planner could add the index-answerable `MATCH_ALL` of the same terms as a pre-filter, so only rows containing every term are checked row by row. We have a patch and can open a PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
