morrySnow opened a new pull request, #67434:
URL: https://github.com/apache/doris/pull/67434
### What problem does this PR solve?
Issue Number: None
Related PR: #67429
Problem Summary: This draft PR is stacked on #67429. Internal Nereids AST
parsing allocated whitespace and ordinary line-comment tokens even though ANTLR
consumes only default-channel tokens. This change adds an opt-in lean lexer
mode for direct AST construction and skips only `WS` and `SIMPLE_COMMENT`.
Public lexer, scan, and comment-normalization paths remain in full-token mode;
`BRACKETED_COMMENT` remains on channel 2 for hints; token character offsets are
unchanged. Syntax failures are retried with full tokens so the existing
exception type, first-error position, and diagnostic text are preserved.
### Benchmark
Lower latency and allocation are better.
- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB heap
- JMH: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc`
- Baseline: `773c4ee1027`; parser jar SHA-256
`65730a22a5262370a832bda7a04ae4b8e0de5c107be65c5ae80f95c711842f59`; benchmark
jar SHA-256 `47284ffd6779e9d534a19a682de5a4d468f83a62a98e048982807497f1f665b6`
- Candidate: `1317d48b035`; parser jar SHA-256
`e1bf35e20619d4a2881db6bbd1cbce2e7abcb0743da62abbb7060f76e6dfcff5`; benchmark
jar SHA-256 `2f9a7bfeac4204224b97542309bb647f051f44575734a9f5df5f3ce941b716e7`
- Harness:
`fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/LeanTokenModeBenchmark.java`
- Workloads: typical SELECT; 32-column comment-heavy SELECT; 128-column wide
SELECT; hinted join SELECT
#### Normalized allocation
Full and lean are measured in the same candidate artifact so only token mode
changes. Values are bytes/op.
| Path | Workload | Full | Lean | Change |
|---|---|---:|---:|---:|
| Tokenize | Typical | 3,405.4 | 2,525.4 | -25.8% |
| Tokenize | Comment-heavy | 12,898.8 | 6,544.1 | -49.3% |
| Tokenize | Wide SELECT | 63,831.6 | 40,743.5 | -36.2% |
| Tokenize | Hinted | 3,328.0 | 2,565.4 | -22.9% |
| Parser end-to-end | Typical | 16,421.9 | 15,838.4 | -3.6% |
| Parser end-to-end | Comment-heavy | 56,797.8 | 50,403.1 | -11.3% |
| Parser end-to-end | Wide SELECT | 278,525.3 | 255,490.8 | -8.3% |
| Parser end-to-end | Hinted | 15,594.2 | 14,768.1 | -5.3% |
| FE LogicalPlan | Typical | 30,088.7 | 29,155.1 | -3.1% |
| FE LogicalPlan | Comment-heavy | 93,372.7 | 86,967.1 | -6.9% |
| FE LogicalPlan | Wide SELECT | 501,914.9 | 478,821.1 | -4.6% |
| FE LogicalPlan | Hinted | 35,621.9 | 34,901.9 | -2.0% |
The target reduction comes from avoiding `CommonToken` allocation for
whitespace and ordinary line comments. Hint tokens and all default-channel
tokens are still allocated.
#### Latency and full-mode control
The stable parser cases improved from 31.322 +/- 3.135 to 29.741 +/- 1.648
us/op for comment-heavy SQL (-5.0%), and from 139.441 +/- 6.714 to 127.317 +/-
4.184 us/op for the wide SELECT (-8.7%). Other latency runs had large
host-scheduling error intervals, so they are not used as acceptance evidence;
normalized allocation above remained stable.
A frozen-artifact B1/C1 full-mode control, which leaves lean mode disabled,
measured:
| Control | Baseline | Candidate | Change | Baseline B/op | Candidate B/op |
|---|---:|---:|---:|---:|---:|
| `SELECT 1` | 2.516 +/- 0.057 us/op | 2.313 +/- 0.038 us/op | -8.0% |
4,792.0 | 4,856.0 |
| Typical SELECT | 5.481 +/- 0.404 us/op | 5.310 +/- 0.305 us/op | -3.1% |
8,858.8 | 8,824.1 |
No full-mode control metric regressed beyond the 3% threshold. A second
candidate timing run was affected by host scheduling noise; its allocation
remained consistent and it is retained in `/tmp/antlr-p3-control-c2.json`
rather than averaged into the table.
Build and benchmark commands:
```shell
mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark -am package \
-Dmaven.build.cache.enabled=false -Dskip.doc=true
java -jar /tmp/antlr-p3-candidate-benchmarks.jar LeanTokenModeBenchmark \
-prof gc -rf json -rff /tmp/antlr-p3-lean.json
```
Raw local results: `/tmp/antlr-p3-lean.json`,
`/tmp/antlr-p3-fe-end-to-end.json`, and
`/tmp/antlr-p3-control-{b1,c1,c2,b2}.json`.
### Semantic differential
- Corpus: all 4,610 tracked `*.sql` files; manifest SHA-256
`567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10`
- Baseline versus candidate: 4,275 parsed and 335 rejected in both legacy
and ANSI modes
- Per-file parse/error signature SHA-256:
`69c811d0d80c52b40d8cd854e925ff4930aa6601c7d1e66541f2eb29f35db50a` for every
baseline/candidate output
- Candidate full versus lean: 18,440 combinations across the corpus,
`noBackslashEscapes`, and ANSI modes had identical filtered token tuples and
CST signatures
- Focused FE tests verify hint extraction, scan/normalization full-token
behavior, CREATE VIEW and sync-MV source intervals, encryption offsets, and
exact invalid-SQL messages
- An 8-thread test verifies deterministic lexing with ANTLR's shared static
DFA
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Parser module: 241 tests passed
- `LeanTokenModeTest`: 4 passed, 0 failures
- `EncryptSQLTest`: 3 passed, 0 failures
- Parser, benchmark, and FE Checkstyle: 0 violations
- Tracked SQL corpus and 18,440-mode differential completed
- JMH target, FE end-to-end, GC-profiler, and full-mode control runs
completed
- `./build.sh --fe` passed the parser module, then stopped during
unrelated connector packaging with `Java heap space`; both targeted
`run-fe-ut.sh` invocations completed their 60-module reactors successfully
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
