airborne12 opened a new pull request, #70262:
URL: https://github.com/apache/doris/pull/70262
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70261 (T2), whose commits show in this diff until
they merge; review only this PR's own commits: `25ead53be63` [refactor](be)
Switch every consumer of text analysis to the TextAnalyzer interface. It stays
a draft until its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70261
Problem Summary:
The SNII text analysis is moving off CLucene. The previous
change added the CLucene-free TextAnalyzer interface and two adapters with no
production caller. This one moves every consumer onto the interface, so the
analyzers can follow one at a time. The analyzers themselves still run on
CLucene behind LuceneBackedTextAnalyzer, except the gram family, which moves
now
because the SNII writer depends on its stream.
- AnalyzerProvider::get_analyzer(), InvertedIndexAnalyzerCtx::analyzer,
InvertedIndexAnalyzer::create_analyzer() and the IndexPolicyMgr getters
hand
out a TextAnalyzerPtr. index_policy_mgr.h and inverted_index_parser.h no
longer
name a CLucene type, and no AnalyzerPtr is left outside the analysis layer.
- The index-level char filter (char_filter_type and friends) moves from the
readers each consumer built (create_reader) into the analyzer:
create_analyzer() and create_analyzer_provider() put it in front of
whatever
they build, builtin, custom or normalizer, so every consumer reads every
value
through it exactly as before. IndexPolicyMgr's unused outer_char_filter_map
parameter goes.
- get_analyse_result() has one loop: a reusable form over a stream and a
batch,
and a one-shot form over an analyzer. analyze() and normalize() use it.
- The SNII writer keeps one stream and one batch for all its rows, made in
init() and destroyed by finish() and close_on_error(). The two lanes
(_plain_lane_token_stream, _finish_plain_lane_row) and _char_string_reader
go;
the memory report covers the stream and the batch of every analyzer
(_report_analysis_capacity), not only the gram tokenizer's buffers. The
solved
gram density is set directly on the stream, which replaces
_pending_density_permille.
- GramTokenizer is a native TextTokenizer that hands out its grams as views,
and
GramTextAnalyzer is the analyzer of a gram-family policy, which the FE
allows
no filter. The CLucene version stays as LuceneGramTokenizer for a CLucene
chain, which only a policy created before the FE check (a gram tokenizer
plus
a filter) can still build. GramExtractor and GramTokenizer release a buffer
above 64 KiB at the next row instead of keeping it until the segment ends.
- MATCH without an index and tokenize() tokenize a block with one stream and
one batch.
- The V1/V2 writer reaches its analyzer through LuceneAnalyzerBridge and
hands
it plain readers.
- create_builtin_analyzer() and normalize() turn a CLuceneError, such as a
missing dictionary file, into an INVERTED_INDEX_ANALYZER_ERROR exception,
so
the CLuceneError catches of analyze(), the SNII writer, the scoring
collector,
the search functions and tokenize() go. function_search.cpp keeps the one
around the whole evaluation, which covers index reads too.
- The analyzer benchmark drives TextAnalyzers built through the production
entry points: "row/" with a stream per row, "reuse/" with one stream.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- ./run-be-ut.sh --run with the analysis, golden, gram, SNII, MATCH,
tokenize, search, inverted index reader/writer, policy and collector
suites passes on ASAN (3945 tests).
- analysis_golden.txt only gains lines: four frozen write/v2-array
digests
of an ARRAY<STRING> column on V2, added at 52719502834, whose bridge
path wrote them first; there the icu one already read 282 postings,
digest 118e86a496cf6071, as here. The other digests are unchanged.
- The benchmark's digest mode over all six corpora gives the same tokens
as the T1 baseline binary in all 5,240,060 rows; row/ and reuse/ run.
- New tests: the index char filter is part of create_analyzer() and
create_analyzer_provider(); a missing chinese dictionary fails SNII
init(), analyze() and tokenize() with INVERTED_INDEX_ANALYZER_ERROR;
after a 2 MiB row the SNII batch is under 64 KiB and the writer charges
exactly the stream's and the batch's capacity; finish() and
close_on_error() destroy the stream; the solved density is on the
writer's GramTokenizer; the native GramTokenizer gives the CLucene
tokenizer's grams and releases a long row's scratch at the next row.
- Fixed while widening the run: the golden write tests and the adapter
tests now set up ExecEnv's tmp file directory themselves
(test::ensure_tmp_file_dirs()), since some earlier test may have
cleared it.
- ./run-regression-test.sh --run -d inverted_index_p0 on an ASAN cluster
of this build: 232 of 252 suites pass on the first run. The other 20
need debug points (18) or room under the FE's limit of 100 index
policies, which the earlier suites fill (2); all 20 pass on a fresh
cluster with enable_debug_points = true. The BE logged no sanitizer
error.
- build-support/run-clang-tidy.sh reports no warning on the changed
lines. build-support/check-build-hygiene.sh passes, with the CLucene
exception tables down from 31 + 51 to 29 + 46 entries: the SNII writer,
the compaction eligibility check, analyzer_provider.h and
inverted_index_parser.h no longer reach or name CLucene.
- Behavior changed:
- [x] No.
- [ ] Yes.
Two differences are only reachable outside SQL or on an error path:
a V1/V2 ARRAY index with the icu parser now indexes its elements (before,
ICUAnalyzer::tokenStream() and invertField() both reset the stream and the
second reset found the reader consumed), but the FE accepts no parser on an
array column; and a CLuceneError raised while tokenize() or MATCH without an
index analyzes a value now reports INVERTED_INDEX_ANALYZER_ERROR.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
