airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4061509457
##########
be/src/storage/index/inverted/tokenizer/icu/icu_tokenizer.cpp:
##########
@@ -73,6 +83,24 @@ void ICUTokenizer::reset() {
if (!buffer_.isEmpty() && buffer_.isBogus()) {
_CLTHROWT(CL_ERR_Runtime, "Failed to convert UTF-8 string to
UnicodeString.");
}
+ utf16ToUtf8Offset_.assign(buffer_.length() + 1, 0);
Review Comment:
Fixed in 6ecc7e14. ICUTokenizer no longer materializes a full
UTF-16-to-UTF-8 table. It now resolves monotonically increasing break-iterator
boundaries with a constant-size UTF-8/UTF-16 cursor while preserving exact
source spans and surrogate-interior clipping behavior. Added ASAN large-input
allocation coverage; the 4 MiB reset case now stays below 4x input auxiliary
allocation. Validation: 203/203 related ASAN tests, full ASAN BE build, and
clang-tidy passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
