HappenLee opened a new pull request, #68406:
URL: https://github.com/apache/doris/pull/68406
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68369
Problem Summary:
Doris maintains separate handwritten SSE, NEON and scalar UTF-8 validators.
Replace them with the simdutf dependency added in #68369, keeping the existing
boolean validation entry point and file-scan validation setting. SIMD selection
and short-input handling use simdutf's runtime dispatch; no CPU-specific or
length-specific dispatch is added in Doris.
Remove the obsolete implementation and scalar-only API, update its remaining
test caller, and drop the old license and tooling exclusions. Fix the existing
UTF-8 tests, whose `sizeof(vector)` loops only exercised the first test case,
and extend coverage for Unicode boundaries, malformed and truncated sequences,
embedded NUL, empty/null buffers, SIMD boundaries, unaligned exact-sized
buffers, long inputs, independent row validation and the file-scan bypass.
### Release note
None
### Check List (For Author)
- Test: Unit tests and static checks
- Passed all 9 UTF-8 utility tests in a standalone harness compiling the
actual modified implementation, test source and generated Thrift types. Both
the implementation and simdutf 9.2.0 were built with ASAN/UBSAN. Passed with
automatic dispatch and the supported icelake, haswell, westmere and fallback
implementations.
- Passed repository formatting and build-hygiene checks; clang-tidy
passed for the UTF-8 implementation and tests.
- Attempted the standard BE ASAN test runner with UTF-8, ColumnString,
SQL-function and Variant filters. The build stopped in unchanged
`common/cpp/aws_common.cpp` because the local AWS dependency lacks
`aws/core/auth/GeneralHTTPCredentialsProvider.h`; these integrated tests did
not run.
- clang-tidy for the one-line Variant test-helper update is blocked by
an existing unmatched `NOLINTEND` in unchanged `be/src/core/types.h:576`.
- Behavior changed: No; validation semantics and the file-scan bypass are
preserved.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
