matanper opened a new issue, #70588:
URL: https://github.com/apache/doris/issues/70588

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### Version
   
   4.1.4. master (`42b148bac33`) has the same code.
   
   ### What's Wrong?
   
   On a column whose fulltext inverted index was built without 
`"support_phrase" = "true"`, the same `MATCH_PHRASE` predicate either fails the 
query or returns correct rows by scanning every row. Which one happens depends 
on whether the predicate is evaluated through the index or as an expression:
   
   - **Pushed down to the index:** the query fails with 
`[INDEX_INVALID_PARAMETERS] phrase queries require setting support_phrase = 
true`.
   - **Not pushed down** (for example inside an `OR` with a non-indexed 
predicate, above a join or projection, or on segments that have no index for 
the column): the query succeeds, evaluating the phrase row by row.
   
   So a query can fail or succeed depending on the plan shape, and on which 
segments of the scan carry the index. Adding an unrelated `OR` turns a failing 
query into a full scan that succeeds.
   
   ### Why
   
   `FunctionMatchBase::evaluate_inverted_index` 
(`be/src/exprs/function/match.cpp:340-352` on master) returns an error for 
phrase functions when the fulltext reader lacks phrase support:
   
   ```cpp
   if (... segment_v2::IndexReaderHelper::is_fulltext_index(reader.value()) &&
       !segment_v2::IndexReaderHelper::is_support_phrase(reader.value())) {
       return Status::Error<ErrorCode::INDEX_INVALID_PARAMETERS>(
               "phrase queries require setting support_phrase = true");
   }
   ```
   
   The row path, `FunctionMatchPhrase::execute_match`, has no such requirement. 
It tokenizes each row with the same analyzer and checks for consecutive tokens, 
gated only by `enable_match_without_inverted_index` 
(`FunctionMatchBase::check`, `:551-555`, default true).
   
   The caller already has a clean way to decline: 
`VExpr::_evaluate_inverted_index` records an index result only when the 
returned bitmap is non-empty. Returning OK without a bitmap makes the 
expression evaluate row by row on the remaining rows. `evaluate_inverted_index` 
already does exactly that for a segment without the index (`iter == nullptr`).
   
   ### What You Expected?
   
   `MATCH_PHRASE` returns the same result whether or not it is pushed to an 
index without phrase support. An index that cannot answer a phrase query 
declines, and the predicate is evaluated row by row, as it is on a segment 
without the index. When `enable_match_without_inverted_index = false`, the 
existing "not support execute_match" error still applies.
   
   ### How to Reproduce?
   
   ```sql
   CREATE TABLE t (
     id BIGINT,
     msg STRING,
     INDEX idx_msg (msg) USING INVERTED PROPERTIES ("parser" = "unicode")  -- 
no support_phrase
   ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
   PROPERTIES ("replication_num" = "1");
   
   INSERT INTO t VALUES (1, 'failed password for root'), (2, 'password failed 
for root');
   
   SELECT id FROM t WHERE msg MATCH_PHRASE 'failed password';
   -- ERROR: phrase queries require setting support_phrase = true
   
   SELECT id FROM t WHERE msg MATCH_PHRASE 'failed password' OR id = -1;
   -- id has no inverted index, so the OR is evaluated as an expression and
   -- FunctionMatchPhrase::execute_match answers the phrase row by row 
(expected: 1)
   ```
   
   We saw both outcomes in production on the same predicate: some statements 
failed with this error, while another statement with the same predicate 
completed after scanning every row of the column.
   
   ### Anything Else?
   
   Proposed fix: in `FunctionMatchBase::evaluate_inverted_index`, when a phrase 
function meets a fulltext reader without phrase support, return `Status::OK()` 
without setting the bitmap, the same as when the segment has no index. The 
expression is then evaluated row by row.
   
   Two notes:
   
   - The row path does not parse phrase options such as slop (`'a b ~2'`); only 
the index phrase query understands them. An index without positions never 
answered phrase queries, so this adds no new semantics, but it could be 
documented.
   - A follow-up could make the fallback cheap. When a phrase meets a fulltext 
index without positions, the planner could add the index-answerable `MATCH_ALL` 
of the same terms as a pre-filter, so only rows containing every term are 
checked row by row.
   
   We have a patch and can open a PR.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to