airborne12 opened a new pull request, #70266:
URL: https://github.com/apache/doris/pull/70266
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70265 (T6), whose commits show in this diff until
they merge; review only this PR's own commits: `82aa141fe10` [refactor](be)
Move jieba into the Doris tree for the chinese parser, `058fb5c4aef`
[improvement](be) Cut chinese text over a flat jieba dictionary. It stays a
draft until its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70265
Problem Summary:
**[refactor](be) Move jieba into the Doris tree for the chinese parser**
(`82aa141fe10`)
The chinese parser was the last builtin parser still on a
CLucene analyzer: LanguageBasedAnalyzer behind LuceneBackedTextAnalyzer, over
the copy of cppjieba in CLucene's contribs-lib. jieba's segmenters now live
in
the BE as analysis/tokenizer/jieba/, ported without a change in how they cut,
and the parser runs them through a tokenizer of the new stream interface.
This commit only moves the code; a later one changes its data layout.
- analysis/tokenizer/jieba/: trie.h, dict_trie.h, mp_segment.h, hmm_model.h,
hmm_segment.h, mix_segment.h, full_segment.h, pre_filter.h, segment_base.h,
unicode.h and jieba.h from cppjieba, string_util.h from limonp, with
cppjieba's and limonp's namespaces replaced by
doris::segment_v2::inverted_index::jieba, so that doris_be_test links both
copies without merging them. No namespace std, XLOG or XCHECK: a
dictionary,
user dictionary or model file that cannot be read, or has an illegal line,
throws INVERTED_INDEX_ANALYZER_ERROR instead of aborting the process.
Decoding comes from analysis/common/rune.h, which ik already uses. The
keyword extractor, POS tagger, query segment, logging and std extensions
are
not brought over, so idf.utf8 is no longer read; it stays in the package.
The asserts on dictionary data are DCHECKs, active in the same builds.
- JiebaTokenizer: copies the value, cuts it with MixSegment and the HMM in
the
coarse mode or with FullSegment in the fine mode, and hands the words out
in
batches, cut to 255 bytes. With lower_case, a word whose first byte is
ASCII
is lowercased where it lies when it is handed out, as the old tokenizer did
to its buffer, so a word sees the lowercasing of the words before it. The
fine mode's words overlap: it hands out a run of ASCII letters and digits
whole and then the runs of letters and of digits in it, and jieba reads a
stray continuation byte and the byte after it as one character, so a run
can start with a byte that is not ASCII, keep its capitals, and be lowered
by the next word. A word that still has a capital after its turn is
therefore copied into the batch; the other terms are views of the copy of
the value. The dictionaries are loaded once per directory, and a failed
load is tried again by the next analyzer.
- analyzer.cpp builds the chinese parser on JiebaTokenizer, and with
lucene_builtin_analyzer and the CLuceneError catch gone it includes no
CLucene: it leaves the CLucene exception tables of check-header-deps.py.
- LICENSE.txt, dist/LICENSE-dist.txt, dist/licenses/LICENSE-cppjieba.txt,
dist/licenses/LICENSE-limonp.txt and .licenserc.yaml carry the MIT terms of
the derived files.
**[improvement](be) Cut chinese text over a flat jieba dictionary**
(`058fb5c4aef`)
jieba, moved into the BE unchanged in the previous commit,
still walked a trie of hash maps behind pointers, kept a 20-byte RuneStr and
a
DAG entry with an inline vector per character, and looked separators, stop
words and HMM emissions up in hash sets and maps. The chinese parser now cuts
over flat arrays built once from the same dictionaries, with every decision
the old segmenters made kept, so the tokens do not change.
- flat_dict.h: FlatDict, built from the loaded DictTrie, HMMModel and stop
word file, which are dropped afterwards. The trie is breadth-first arrays
(child_begin, key, weight, has_value), so the children of a node are
consecutive and a node's count is the difference of two child_begin
entries; the first rune has a table of 0x10000 entries, supplementary first
runes are searched for among the root's children, up to 8 children are
scanned and more are bisected. User single-rune words are a bitmap with a
set for supplementary runes, the emission probabilities a rune-major table
of 0x10000 x 4 with a map for supplementary runes, and stop words of up to
8 bytes an open-addressing table keyed by their bytes, longer ones a sorted
vector. The pointer trie is never built: a stable sort keeps the last unit
of each word, which is what inserting the units in order kept.
- segmenter.h: the mix and full cuts of MPSegment, HMMSegment, MixSegment,
FullSegment and PreFilter over a JiebaScratch (jieba_scratch.h): runes as
code point and byte offset arrays, the DAG walked straight into the DP or
the full cut in its old order instead of being stored, one column of
Viterbi weights. Kept as they decide tokens: the lenient decoding of
rune.h, no token for a value that does not decode, the DP adding weights in
the old order and keeping the first maximum (strict >), MAX_WORD_LENGTH
512, FullSegment's word_len carried from one rune to the next, the whole
alphanumeric run before its runs of letters and digits, user single-rune
words, stop words dropped before lowercasing. The separators are compared
directly: space, \t, \n, U+FF0C and U+3002, all SPECIAL_SEPARATORS held.
- jieba.h loads the files and keeps only the FlatDict; JiebaTokenizer owns a
JiebaScratch, releases it above the high water mark at reset() like its
other buffers and counts it in reserved_bytes(). next_batch() is unchanged.
- trie.h, mp_segment.h, hmm_segment.h, mix_segment.h, full_segment.h,
pre_filter.h and segment_base.h are gone; LICENSE.txt and .licenserc.yaml
list the derived files that remain.
### Release note
The chinese parser tokenizes about 3.2 times as fast in the coarse mode and
2.3 times as fast in the fine mode, and its dictionaries take less memory.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- `82aa141fe10` [refactor](be) Move jieba into the Doris tree for the
chinese parser:
- BE UT under ASAN: the analysis, index and search suites with the
T5 to
T7 ones, 4191 of 4198 tests pass; the 7 skipped are opt-in reports
and
generators, the opt-in code point scan and the vector width check
of the
build without AVX2. The code point scan passes with
ANALYSIS_GOLDEN_FULL=1. JiebaTokenizerTest runs the parser beside
LanguageBasedAnalyzer in both modes, with and without lowercase, on
random values with invalid bytes, on every word of jieba.dict.utf8
and
user.dict.utf8 (64 to a value, half with raised letters) and on
mixed-case, unknown and undecodable words. A death test does the
same on
a dictionary of its own, with words such as 卡拉OK, Q版本 and words of
a
supplementary plane, in a process of its own because CLucene's
jieba
keeps the first dictionary it loads. Other new cases: 你好\xE4\xB8
makes
no token, the first byte decides the lowercase, supplementary-plane
characters and user words, loading without idf.utf8, a failed load
tried
again, eight threads on one analyzer, batches around 256 tokens,
the
stream contract, and the buffers released after a long value.
- Mutation: lowercasing the words of a batch in reverse order fails
AnalysisDifferential and JiebaTokenizerTest's random values, both
on
the shipped dictionaries, where a stray continuation byte starts a
run
of letters that keeps its capitals, and the batching test. The
source
was restored before the final runs.
- The analysis golden files are unchanged, and the benchmark's
digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060
rows,
chinese_coarse and chinese_fine among them.
- The port was also compared with the original headers directly,
outside
the BE build: 3,408,272 calls of cut and cut_all with 0
differences, on
the shipped dictionaries (every byte and pairs and triples of
chosen
bytes, 300k random sentences with invalid bytes, every dictionary
word
alone and joined to the next, about 70 MB of corpora) and on
edited ones
(an undecodable word, user dictionaries of 1 to 4 columns, CRLF);
15
broken files fail on both sides.
- check-build-hygiene.sh passes, with analyzer.cpp out of both
CLucene
exception tables (12+11 now); clang-format 16 leaves the changed
files
as they are; run-clang-tidy.sh reports nothing on the changed
lines.
- `058fb5c4aef` [improvement](be) Cut chinese text over a flat jieba
dictionary:
- BE UT under ASAN: the analysis, index and search suites with the
T5 to
T7 ones, 4191 of 4198 tests pass and the 7 skipped are the opt-in
ones;
the code point scan passes with ANALYSIS_GOLDEN_FULL=1.
JiebaTokenizerTest
passes unchanged: the live differential with LanguageBasedAnalyzer
on
random values, every dictionary word, mixed-case, unknown and
undecodable words, and on a dictionary of its own in a death test.
ComponentStreamTest counts the scratch among the bytes a long value
leaves. IndexFileWriterTest.CacheEvictionTest failed once in a run
of the
wider suite list and passes on its own: it depends on what earlier
tests
leave in a shared searcher cache, not on this change.
- The analysis golden files are unchanged, and the benchmark's
digest mode
gives the same tokens as the T1 baseline binary in all 5,240,060
rows,
chinese_coarse and chinese_fine among them.
- The new code was also compared with the original CLucene headers
directly, outside the BE build and with the real
jieba_tokenizer.cpp:
4,208,058 cuts and 7,364,072 tokenizer runs (both modes, lowercase
on
and off, the scratch reused and fresh) on every byte and pairs and
triples of chosen bytes, 300k random sentences with invalid bytes,
every
dictionary word alone and joined to the next, long values, all
corpora,
and edited dictionaries (an undecodable main word, duplicate main
words,
supplementary-plane user words including words of 512 and 513
runes,
supplementary-plane emissions, CRLF files, 12k stop words): 0
mismatches, in a release and in an ASAN+UBSan build. 9 of 10
deliberate
mutations of the new code were caught; the tenth swaps E and S on
a tie
at the last rune, which ends a word either way.
- Throughput of the benchmark binary of this change with
be/benchmark/analyzer_ab.py: five alternating rounds, the median
of the
per-round ratios on each corpus and their geometric mean, reuse/
and
row/. Against the T1 baseline: coarse 3.62 and 3.23 (zh_wiki 2.85
and
2.70, ja_wiki 3.18 and 2.93, the four mixed corpora 3.97 and
3.47), fine
2.44 and 2.28 (zh_wiki 2.48 and 2.41, ja_wiki 2.66 and 2.54, mixed
2.37
and 2.18). Against the previous commit: coarse 3.54 and 3.10, fine
2.43
and 2.19, so the move itself kept the old speed.
- Memory of loading the shipped dictionaries, each in a process of
its own
with glibc malloc: the old cppjieba::Jieba with its five files
+132.3 MB
of RSS after loading (+145 MB at the peak); the new holder +40.5 MB
right after loading and +11.2 MB after malloc_trim (+51 MB at the
peak),
the difference being freed loader memory that glibc keeps and the
BE's
jemalloc returns on decay. Loading takes 0.19 s instead of 0.51 s.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed
files
as they are; run-clang-tidy.sh reports nothing on the changed
lines.
- Behavior changed:
- [x] No.
- [ ] Yes.
`82aa141fe10`: Every token of the chinese parser is the same as
before (above). A missing dictionary file still fails the analyzer, now with
INVERTED_INDEX_ANALYZER_ERROR from the BE.
`058fb5c4aef`: Every token of the chinese parser is the same as
before (above).
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
