morrySnow opened a new pull request, #67452:
URL: https://github.com/apache/doris/pull/67452
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Every SQL lexer currently copies its input through
`CharStreams.fromString()` and calls `Character.toUpperCase()` for every
case-insensitive lookahead. This is visible in lexer allocation and
identifier-heavy parsing time.
Add two strictly compatible fast paths:
- Fold ASCII `a-z` with arithmetic and retain `Character.toUpperCase()` for
every other code point.
- Read strings without UTF-16 surrogates directly, avoiding ANTLR's copied
code-point buffer. Inputs containing any surrogate retain the original ANTLR
stream so code-point indices, `getText`, navigation, and errors remain
unchanged.
The public arbitrary-`CharStream` constructor is preserved. All production
String entry points now use the factory. ANTLR's native `caseInsensitive`
option was evaluated but rejected because it changed existing Unicode behavior
such as the handling of `ſelect`.
### Benchmark
Environment and method:
- Baseline: `cf33a08bcd5`, artifact SHA-256
`51700bef953361218912abc6aec348cecdec3fe61031eba10cc1ebc534bc9183`
- Candidate: `7aa7b5e6487`, artifact SHA-256
`6e8edf78befc0dc349be398aae454c172b5cb968ee98d88a0e056d4c21a9c5d0`
- macOS 15.0.1 arm64, OpenJDK 17.0.20.1, 1 GiB heap, JMH 1.37
- 2 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc`
- Interleaved baseline/candidate runs; one candidate run affected by
unrelated host load was discarded and repeated. The table averages two valid
runs per artifact. Individual JMH scores use 99.9% confidence intervals.
```shell
java -Xms1g -Xmx1g -jar <benchmark.jar> \
'CaseInsensitiveStreamBenchmark.(createLexer|foldPrebuiltCharacters|parseStatement|tokenize)'
\
-p workload=shortQuery,lowercaseIdentifiers,stringAndComment,unicode \
-f 2 -wi 4 -i 7 -w 300ms -r 400ms -prof gc -rf json
```
| Benchmark | Baseline us/op (B1 / B2) | Candidate us/op (C1 / C3) | Mean
change | Baseline -> candidate B/op |
|---|---:|---:|---:|---:|
| Tokenize 64 lowercase identifiers | 10.826±0.078 / 11.408±0.125 |
8.708±0.071 / 8.704±0.096 | -21.7% | 14,799 -> 10,464 (-29.3%) |
| Parse 64 lowercase identifiers | 54.775±7.360 / 56.365±4.224 |
49.860±0.191 / 52.340±1.160 | -8.0% | 99,395 -> 95,017 (-4.4%) |
| Tokenize `select 1` | 0.451±0.004 / 0.485±0.011 | 0.348±0.069 /
0.324±0.010 | -28.2% | 1,008 -> 816 (-19.0%) |
| Parse strings/comments with Unicode | 27.047±5.371 / 28.435±2.932 |
24.744±0.401 / 24.743±0.425 | -10.8% | 24,167 -> 23,784 (-1.6%) |
| Tokenize supplementary Unicode control | 32.034±1.398 / 33.910±0.752 |
32.303±1.108 / 32.020±0.738 | -2.5% | 24,630 -> 24,594 (-0.1%) |
The isolated `createLexer` measurement for the surrogate-containing
string/comment workload regresses by 5.9% because the compatibility guard scans
for surrogates before falling back. The corresponding complete tokenize and
parse paths improve by 3.6% and 10.8%; no end-to-end control workload
regressed. Prebuilt lowercase character folding improves by 10.6%.
The performance gains come from eliminating the copied input buffer for
BMP-only SQL, avoiding its allocation, and replacing the common lowercase ASCII
`Character.toUpperCase()` call with an arithmetic branch.
Correctness corpus:
- 4,610 tracked SQL files; baseline and candidate parse signatures match in
legacy and ANSI modes. SHA-256:
`69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a`.
- 9,220 lexer cases (4,610 SQL files x both `noBackslashEscapes` modes);
complete token tuple and lexer-error signatures match. SHA-256:
`66780d4e1d9224ea27c8f715ab33edd247faed1571030ba9e90a8ef3f6212180`.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Tested with:
- `mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark
-Dmaven.build.cache.enabled=false clean package`: 170 tests passed; Checkstyle
passed.
- `./run-fe-ut.sh --run
org.apache.doris.httpv2.websql.SingleStatementValidatorTest,org.apache.doris.nereids.parser.NereidsParserTest,org.apache.doris.nereids.parser.NereidsParserDigestTest`:
91 tests passed.
- Token and parse corpus comparisons described above: no differences.
- FE-only `./build.sh --fe` with a fresh output directory: passed and
produced `Successfully build Doris`.
- Full `./run-fe-ut.sh` was attempted, but the host's Surefire JVM stopped
during the pre-existing `fe-connector-trino` `TrinoPredicateConverterTest`
dynamic-attach path before reaching `fe-core`; it was terminated after 90
minutes with no P4-related failure.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
