HappenLee opened a new pull request, #70520:
URL: https://github.com/apache/doris/pull/70520
### What problem does this PR solve?
Issue Number: None
Related PR: #70513
Problem Summary:
String bucket hashing and SQL `crc32()` call zlib for every row, including
short inputs where library setup and serial byte processing are expensive.
Extend the existing constexpr CRC32 table to eight slices and reuse the
four-byte step for tails and fixed-size values. Route SQL `crc32()` through the
common helper. Inputs below 256 bytes use the short path, with IEEE CRC32
instructions on ARM64 builds that enable the CRC extension; longer inputs keep
zlib's braided implementation. Preserve null-buffer reset, empty-input, seed
and incremental hashing semantics, and use `size_t` lengths to retain SQL
input-size support.
This PR targets master independently of #70513.
### Performance
CRC kernel microbenchmark on an Intel Xeon Platinum 8457C, clang 20.1.8
`-O3`, zlib 1.3.1. Each batch contains 1024 rows stored as bytes and offsets.
Results are median CPU nanoseconds per row from seven randomly interleaved
repetitions pinned to CPU 24. Seeds vary by row to represent bucket hashing.
| Input | Before, ns/row | After, ns/row | Speedup |
|---|---:|---:|---:|
| 8 bytes | 12.08 | 5.65 | 2.14x |
| 32 bytes | 61.83 | 17.80 | 3.47x |
| 64 bytes | 131.28 | 37.96 | 3.46x |
| Mixed 0-255 bytes | 162.80 | 87.33 | 1.86x |
| 4096 bytes | 1803.78 | 1837.49 | 0.98x |
These are CRC kernel measurements, not end-to-end SQL speedups. The machine
had concurrent load and CPU frequency scaling; the long-input difference is
within the observed variation. The submitted string helper is identical to the
benchmarked implementation. The benchmark includes both the original `crc32`
and `crc32_z` calls, seeded and zero-seed optimized paths, boundary lengths,
and mixed short/long inputs.
### Release note
Improve CRC32 performance for short strings while preserving checksum values.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- All 17 targeted fixed-size/string CRC32 tests passed in an
independently linked x86_64 ASAN binary built from this PR's header and test
source.
- Added SQL `crc32()` NULL and binary-boundary coverage.
- [x] Manual test (add detailed scripts or steps below)
- Registered reproducible before/after Google Benchmarks; results
and method are above.
- clang-format 16 on changed lines, build hygiene, diff checks and
scoped core/benchmark clang-tidy passed.
- An extracted implementation from the PR header cross-compiled for
ARM64 with and without CRC extensions; IEEE instruction selection and the
software fallback were verified.
- Standard `./run-be-ut.sh -j 48 --run
--filter='ZlibCRC32Fixed.*:ZlibCRC32Strings.*:function_string_test.crc32_binary_boundaries'`
was blocked by existing local workspace dependency and
`ColumnVector`/`PODArray` compilation errors. SQL wrapper/test clang-tidy was
also blocked by those column errors; SQL test execution remains pending.
- Native ARM64 correctness/performance and end-to-end SQL testing
remain pending.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
