airborne12 opened a new pull request, #70259:
URL: https://github.com/apache/doris/pull/70259
> Part of the SNII / text analysis / ANN CLucene removal series (index at
the end). This PR depends on no other PR of the series. Its commits:
`5794906702a` [improvement](be) Resolve a tokenizer's char filter once per
value.
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
DorisTokenizer::correct_source_offset and
correct_source_start_offset ran dynamic_cast<const DorisCharFilter*> on the
reader for every token. With no char filter configured the cast always fails,
and a failing cast walks the type hierarchy comparing names with strcmp,
twice
per token. A perf profile puts the casts and their strcmp at about 55% of the
basic parser and about 39% of a custom standard + lowercase analyzer.
The reader only changes in DorisTokenizer::reset(), so the cast now runs
there
and the result is kept for the value.
Alternating A/B over 3 rounds on 4 corpora, single thread, release build,
geomean of be/benchmark/benchmark_analyzer.hpp:
basic 64.8 -> 123.2 MB/s 1.90x
custom basic + lowercase 65.7 -> 127.0 MB/s 1.93x
custom standard + lowercase 59.6 -> 93.0 MB/s 1.56x
custom standard + folding 55.0 -> 82.1 MB/s 1.49x
icu 18.6 -> 20.5 MB/s 1.10x
english and standard do not use DorisTokenizer and moved 1.07x and 1.03x,
which
sizes the host noise.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- The benchmark above. Tokens of basic and custom standard + lowercase
were
compared with an independent implementation on 194 MB of real text, no
row differed.
- Unit Test: ./run-be-ut.sh --run
--filter='*Tokenizer*:*Analyzer*:*CharFilter*', 392 tests passed.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
