airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4067855127
##########
be/src/storage/index/inverted/token_filter/word_delimiter_filter.cpp:
##########
@@ -175,6 +195,11 @@ void WordDelimiterFilter::reset() {
_buffered_pos = 0;
_buffered_len = 0;
_first = true;
+ _saved_source_byte_offsets.clear();
Review Comment:
Fixed in f4f79a496e0. Reproduced first: the same 96 KiB value left about 2
MiB (roughly 21x the input) of retained capacity across WordDelimiter's
saved/current offset vectors, attribute buffers, buffered states and both
concatenations.
`reset()` now applies the shared high-water release to all of them; the
`_states` vector is re-created at its initial size when the vector itself has
grown past the limit, otherwise each buffered state is trimmed in place.
Covered by the same test as the tokenizer case; the analyzer suites (543 tests)
pass under ASAN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
