Baymine opened a new pull request, #66164:
URL: https://github.com/apache/doris/pull/66164
## Proposed changes
Issue Number: close #66163
## Problem Summary
`HiveTextFieldSplitter` splits every Hive text line on the hot scan path.
This PR removes two avoidable costs, both behavior-preserving (split results
are identical):
1. **Single-char separator, no escape char (common case).** The `escape_char
== 0` branch of `_split_field_single_char` scanned the line byte-by-byte. It
now uses `memchr` to locate each separator, letting glibc use its vectorized
search. The escape path is unchanged and still uses
`is_hive_text_separator_escaped`.
2. **Multi-char separator (MultiDelimitSerDe).** `_split_field_multi_char`
rebuilt the KMP prefix table from the separator on every line. It is now
computed once in the constructor (`_kmp_next`) and reused per call. This is a
strict win (no downside).
### Benchmark (memchr vs for+if, single-char path)
There is an existing note in `new_plain_text_line_reader.cpp` that `for +
if` beats `memchr` "under normal short fields case". A micro-benchmark of the
split loop (plain `-O2`, 4M iters/case) shows the tradeoff precisely:
| field width | for+if | memchr | speedup |
|-----------------|---------:|---------:|--------:|
| 3 bytes (tiny) | 182 ns | 192 ns | 0.95x |
| 6 bytes | 210 ns | 189 ns | 1.11x |
| 20 bytes | 283 ns | 175 ns | 1.62x |
| 100 bytes | 958 ns | 175 ns | 5.48x |
| 400 bytes | 949 ns | 58 ns | 16.3x |
`memchr` wins for field widths >= ~6 bytes (the normal case for real Hive
columns: strings, numbers, timestamps, paths) and is dramatically faster for
wide values. The only regression is ~5% for pathologically tiny 1-3 byte
fields, which matches the existing note. If reviewers prefer, the fast path can
be gated behind a small line-length threshold to keep `for + if` for tiny
lines; happy to add that.
## Release note
None
## Check List (For committer)
- [ ] Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Reason:
- [ ] Behavior changed:
- [x] No.
- [ ] Yes.
- [ ] Does this need documentation?
- [x] No.
- [ ] Yes.
## Check List (For author)
- [x] Unit Test: `be/test/format/text/hive_text_field_splitter_test.cpp`
gains 5 cases covering the memchr fast-path boundaries (Hive `\x01` default
separator, long fields spanning SIMD chunks, consecutive/leading/trailing
separators, multi-byte UTF-8), the escape vs no-escape divergence, the cached
KMP table reused across rows, and the multi-char escape boundary conditions.
`HiveTextFieldSplitterTest` 10/10 pass locally.
- [x] Manual test: micro-benchmark of the split loop (results above).
- [x] I have unit-tested and manually tested this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
