airborne12 opened a new pull request, #70261:
URL: https://github.com/apache/doris/pull/70261
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70260 (T1), whose commits show in this diff until
they merge; review only this PR's own commits: `52719502834` [refactor](be) Add
the CLucene-free text analysis interface and its two CLucene adapters. It stays
a draft until its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70260
Problem Summary:
The SNII text analysis is moving off CLucene. Today every
consumer (the SNII writer, the query side, the V1/V2 writers) reaches an
analyzer through lucene::analysis::Analyzer, TokenStream and Token, one
virtual
call per token, so no analyzer can leave CLucene before every consumer does,
and the other way round. This adds the interface both sides move to, with no
production caller yet, and the two adapters that let consumers and analyzers
move one at a time.
The interface is in storage/index/inverted/analysis/ and includes no CLucene
header; the gate adds no exception for it.
- TokenBatch: the tokens of one next_batch() call, each a term view and a
position increment. A term points into the value, into memory the stream
keeps, or into the batch's arena, which never moves a term it handed out.
clear() keeps the token array and the arena's largest block;
release_oversized() frees everything once the batch keeps more than 64 KiB,
so one huge value does not pin its memory until the segment ends.
- TextTokenStream, with TextTokenizer and TextTokenFilter: reset(text) binds
a
value, next_batch() hands out up to 256 tokens and returns false, with an
empty batch, only at the end, and reserved_bytes() reports what survives a
value.
- PerTokenTokenizer and PerTokenFilter: bases for components that make one
token at a time; the base copies each term into the batch. A filter reads
its
input through a private batch, so it may hold an input token across
batches,
as word_delimiter holds a token while it hands out its parts.
- TextAnalyzer: immutable after it is built, shared by threads, makes
streams.
The 64 KiB high water mark and release_oversized_scratch() move from
token_stream.h to a new scratch.h, so the batch uses them without CLucene.
The adapters are in storage/index/inverted/clucene/:
- LuceneBackedTextAnalyzer puts a CLucene analyzer behind TextAnalyzer, for
the
consumers that switch before their analyzers do. Each value gets a new
reader
through the index's char filter and a new CLucene stream, as a query reads
a
value today (the SNII writer re-initializes one reader instead); terms are
copied into the batch, and a CLuceneError comes out as a doris::Exception.
It
goes once every analyzer is native.
- LuceneAnalyzerBridge puts a TextAnalyzer behind lucene::analysis::Analyzer,
for the V1/V2 writers. reusableTokenStream() rebinds one stream per scalar
value, tokenStream() makes an owned stream per array element, reset()
starts
the value as SDocumentsWriter::invertField() expects, and terms go out with
setTextNoCopy(), empty ones included, because those writers give an empty
term a position.
One difference turned up and is pinned by a test: on the V1/V2 array path,
ICUAnalyzer::tokenStream() resets the stream and invertField() resets it
again, and ICUTokenizer, which reads its reader in reset(), finds it
consumed:
array elements of an icu index are indexed as nothing. The bridge starts each
element once and indexes them. No table can have such an index, because the
FE
accepts no parser and no analyzer on an array column
(InvertedIndexUtil.checkInvertedIndexParser).
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- ./run-be-ut.sh --run
--filter='TokenBatchTest.*:PerTokenStreamTest.*:LuceneBackedTextAnalyzerTest.*:LuceneAnalyzerBridgeTest.*:LuceneBridgeWriterTest.*:AnalysisGolden*:AnalysisDifferential*:AnalysisWriteGolden*:AnnWriteGolden*:SniiWriterGoldenBytes*:SniiGoldenCorpus*:InvertedIndexWriterTest.*'
passes on ASAN, with no leak report.
- TokenBatchTest: 10000 copied terms over many arena blocks keep their
bytes; clear() keeps the token array and the largest block, and a batch
that fits allocates nothing; release_oversized() leaves at most 64 KiB
after 10000 tokens or a 1 MiB term, and keeps an ordinary batch.
- PerTokenStreamTest: batches of at most 256 tokens; a filter making two
tokens out of one; a filter dropping every token returns false, never
an
empty batch; a filter holding a 300-character token across two batches;
next_token() is not called after it returned false; reset() in the
middle
of a value does not replay it; reserved_bytes() adds up the chain and
reset() releases an oversized input batch.
- expect_text_stream_contract(), a checker later components reuse: false
only with an empty batch and again after, no token for an empty value
with
a null or a non-null pointer, a reused stream equal to a new one, and a
reset after the first batch. It passes on the test streams, on a
scripted
CLucene analyzer and on the standard analyzer through
LuceneBackedTextAnalyzer.
- AnalysisGolden.LuceneBackedTextAnalyzerMatchesTheFrozenDigests: all 122
analyzer configurations through LuceneBackedTextAnalyzer, with a new
stream per line and with one stream for all lines, match
analysis_golden.txt (3172 jobs).
- AnalysisGolden.EightThreadsShareOneTextAnalyzer: eight threads share
each
configuration's analyzer on mixed_zh_en and invalid_utf8, half with a
new
stream per line, and all match the frozen digests.
- AnalysisWriteGolden.LuceneAnalyzerBridgeWritesTheSameV2Index: the
production V2 writer with its analyzer behind the bridge writes the
frozen string-column digests of chinese, ik, icu and the custom chain,
and the same array-column index as the analyzer itself, except the icu
case above.
- LuceneBridgeWriterTest: a doris::Exception from reset() or next_batch()
reaches add_values() and add_array_values() as
INVERTED_INDEX_ANALYZER_ERROR through the CLucene frames.
- analysis_golden.txt and ann_write_golden.txt unchanged.
build-support/check-build-hygiene.sh passes with the same 31 + 51
exceptions. build-support/run-clang-tidy.sh reports no warning on the
changed lines.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
