matanper opened a new issue, #69916: URL: https://github.com/apache/doris/issues/69916
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues. ### Version master (checked at 5b2c066b4d), branch-4.1, 4.1.5 ### What's wrong? `FunctionLike` and `FunctionRegexpLike` build their state, including the Hyperscan database (`hs_compile`) or the RE2/Boost regex for a constant pattern, only in their `THREAD_LOCAL` open. `VExprContext::clone` opens `THREAD_LOCAL` again for every clone, and a pushed-down predicate is cloned once per scanner and once per segment iterator (`SegmentIterator::_construct_compound_expr_context`). A scan over N segments therefore compiles the same constant pattern about N times. A case-insensitive alternation such as `(?i)(alpha|bravo|...|juliet)[.](exe|dll)$` takes milliseconds to compile. On a table with about ten thousand segments, the open phase (`SegmentIteratorInitTimer` in the profile) dominates the query and costs minutes of CPU. Matching itself (`ExprFilterEvalTime`) stays in the tens of milliseconds. `LikeSearchState::clone`, used by `LikeColumnPredicate`, also runs `hs_prepare` again for each clone. ### What you expected to happen A constant pattern is compiled once per fragment, and its clones reuse it. ### How to reproduce ```sql CREATE TABLE t (k BIGINT, s STRING) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 64 PROPERTIES ('replication_num' = '1', 'disable_auto_compaction' = 'true'); -- 60 times: INSERT INTO t SELECT number, concat('file', cast(number AS STRING), '.exe') FROM numbers('number' = '5000'); SELECT count(*) FROM t WHERE s REGEXP 'file1[.]exe$'; SELECT count(*) FROM t WHERE s REGEXP '(?i)(alpha|bravo|charlie|delta|echo|foxtrot|golf|hotel|india|juliet|file1)[.](exe|dll)$'; ``` With about 3.9k rowsets on one BE, the second query takes about 150 ms against about 72 ms for the first: about 77 ms of compiling. With the change below, both take about 47 ms. The gap grows linearly with the segment count. ### Anything else? Proposed change: - In the `FRAGMENT_LOCAL` open, compile a constant pattern and store it as fragment state. `FunctionContext::clone()` already shares that state. - In the `THREAD_LOCAL` open, share the fragment's Hyperscan database, which is read-only once compiled, and give each state its own scratch through `hs_clone_scratch`. - Keep compiling RE2 and Boost per state, because RE2 locks its lazily built DFA cache on every search, so one shared RE2 would serialise matching across scanner threads. - Non-constant patterns keep the current per-call path. - `LikeSearchState::clone` shares the database the same way. Other regexp functions (`regexp_extract`, `regexp_replace` and similar) also compile in their `THREAD_LOCAL` open, and could follow the same pattern. Related: #68977 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
