airborne12 opened a new pull request, #69887:
URL: https://github.com/apache/doris/pull/69887
### What problem does this PR solve?
Issue Number: None
Related PR: #69600
Problem Summary: A regexp without a literal prefix, such as `output
MATCH_REGEXP 'transi.*'`,
expands by walking every term of the SNII dictionary and testing each key
against the pattern. On
a 99M-row table, about 40% of the SNII stack samples of that query were
spent decoding dictionary
entries, twice as much as the pattern match:
- `visit_prefix_range` and `visit_term_range` decoded every key into a fresh
`DictEntry`: the whole
struct reset, a new `std::string`, and a `Status` for every varint.
- `DictBlockReader::open` fully decoded every anchor of the block, about 1
entry in 16, only to get
its key. Every block open pays for this, exact term lookups included.
Both now read an entry's length and front-coded key on the raw cursor. A
walk rebuilds the key in
one buffer it keeps across entries and a rejected key skips its body by the
entry length; `open()`
reads only the keys of the anchors. An entry that does not read cleanly goes
to the checked
decoders, which report what is wrong with it as before, and a walk still
decodes the body of every
entry it accepts with the checked decoder. The on-disk format and the write
path do not change.
`open()` no longer decodes the bodies of the anchors. The block's crc32c,
verified just before,
covers them, and they are checked when they are decoded, like the body of
every other entry.
| Query (99M rows) | Hot, new/previous | Cold, new/previous |
|---|---|---|
| regexp `'transi.*'` | 0.69 | 0.77 |
| regexp that matches no term | 0.67 | |
| regexp `'.*ing'` | 0.72 | |
| regexp `'e.*'` (stops at the expansion limit) | 0.90 | |
| phrase query, term lookup cache off | 0.78 | |
| phrase query, term lookup cache on | 0.99 | 1.01 |
| AND of two terms | 0.99 | 1.01 |
| the other 22 queries | around 1.0 (reruns of those above 1.05: 0.94-1.03)
| |
Hot: both versions interleaved in one BE process. Cold: BE restarted and the
index files evicted
before every run, configuration order alternated, 5 runs each. The build
also had #69879 (the term
lookup cache) and #69882. With that cache off, which is master today,
queries open dictionary
blocks more often: the phrase query gains as the 0.78 row shows, and the
five other lookup-heavy
queries measured that way stayed at 0.96-1.04.
### Release note
Speed up SNII regexp queries that have no literal prefix.
### Check List (For Author)
- Test: Unit Test (`SniiDictBlock`: walks match `decode_all()` for prefixes,
ranges and key filters;
every one-byte change to a block's entries gives a walk the same status,
keys and end of range as
a walk over the checked decoders; `open()` rejects an anchor key it cannot
read; a corrupt anchor
body is reported when it is decoded; removing a bounds check from the key
reader, or the fallback
from `open()`, fails them; all 1507 *Snii* tests pass under ASAN).
Manual test: the hot and cold A/B above; the 25 queries of the bench
return the same results with
both versions.
- Behavior changed: No (results are the same; a block with a valid crc and a
corrupt anchor body now
fails when that entry is decoded instead of when the block is opened)
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
