HappenLee opened a new pull request, #70754: URL: https://github.com/apache/doris/pull/70754
### Background The single-argument MD5 function hashes a batch of independent strings. The existing AVX2 path processes multiple messages together, while ARM64 calls the single-message MD5 implementation for each row. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: Add four-lane NEON operations and reuse the existing two-group MD5 rounds, padding, and output conversion to process eight messages per batch on ARM64. Batches smaller than four messages use the existing serial path, and heavily imbalanced groups use serial hashing to avoid spending SIMD work on lanes that have already finished. Digest values and the AVX2 path are preserved. ### Details - **Why it was slow:** ARM64 did not use the existing multi-message execution framework. - **Optimization:** Native NEON add, rotate, Boolean operations, and a 4x4 message-word transpose; architecture-specific selection of serial hashing for poorly utilized batches. - **Results:** Neoverse-N2, Clang 20.1.8, OpenSSL 3.0.13, `-O3 -march=armv8-a+crc`, one pinned core; median of seven randomly interleaved repetitions (0.15 s minimum). At 4096 rows/batch, 16/64/256/1024-byte inputs achieve 2.51/2.94/3.40/3.70x throughput over the previous per-message ARM algorithm; mixed 0–1024-byte inputs achieve 2.05x. Highly skewed inputs are within 0.6% of serial in the measured cases after fallback. Single-row short inputs and four-row mixed inputs retain about 3% overhead. GCC 14 confirms 2.79–3.50x on the equal-length cases. - **Applicability and cost:** Applies to batched single-argument MD5 on ARM64. A NEON batch uses a temporary 16-byte digest per row, as the existing AVX2 path does. Small batches and skewed message lengths are included in the benchmark. These are MD5 API microbenchmarks, not end-to-end SQL measurements. ARM Clang 20 results (4096 rows per batch): | Batch rows | Input bytes | Serial ns/row | Batch ns/row | Speedup | | ---: | --- | ---: | ---: | ---: | | 4096 | 0 | 141.72 | 54.66 | 2.593× | | 4096 | 16 | 138.45 | 55.11 | 2.512× | | 4096 | 64 | 250.58 | 85.21 | 2.941× | | 4096 | 256 | 591.60 | 173.99 | 3.400× | | 4096 | 1024 | 1954.94 | 528.34 | 3.700× | | 4096 | mixed 0–1024 | 1005.06 | 489.25 | 2.054× | | 4096 | 8192 + seven 16-byte strings | 1956.91 | 1962.15 | 0.997× | ### Testing - Actual `be/src/util/md5.cpp` and `be/test/util/md5_test.cpp` compiled in isolation with ASAN/UBSAN; seven tests cover known digests, padding boundaries, partial SIMD groups, binary data, unaligned input, zero-length null pointers, and guarded output. - Native ARM64 validation with GCC 14 and Clang 20; x86 AVX2 validation with Clang 20. [Reproducible workflow and artifacts](https://github.com/HappenLee/incubator-doris/actions/runs/44500133646). - Added `benchmark_md5.hpp` and registered it in the standard benchmark entry point; serial and batch implementations run on the same inputs with setup excluded from timing. - clang-format 16, build hygiene, changed-line clang-tidy, and `git diff --check` passed locally. clang-tidy used an isolated compilation database after the full UT build was blocked by missing third-party libraries. - The standard `run-be-ut.sh -j 48 --run --filter='Md5Test.*'` was attempted locally but could not finish because the installed third-party package lacks `libgoogle_cloud_cpp_oauth2.a`. A complete BE build and SQL regression run are not claimed. ### Release note Improve the performance of batched single-argument MD5 on ARM64 with NEON, retaining serial hashing for small or highly imbalanced groups. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: isolated native ARM64/x86 tests and benchmarks completed; full BE build and SQL regression are pending CI. ### AI Disclosure - Tool: OpenAI Codex - Generated: NEON implementation, tests, benchmark, and PR description. - Human review: Not yet performed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
