morrySnow opened a new pull request, #67457:
URL: https://github.com/apache/doris/pull/67457
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`PostProcessor` was installed as a global ANTLR parse listener, so every
rule exit paid listener-dispatch cost even though only three local actions were
needed: normalizing non-reserved identifiers, normalizing quoted identifiers,
and reporting malformed unquoted identifiers.
This PR moves those actions into the corresponding grammar rules and removes
the listener from both `DorisSqlParser` and `NereidsParser`. It preserves
identifier token text/type/positions, double-backtick unescaping,
malformed-identifier errors, and parsing with `buildParseTree=false`.
### Benchmark
Environment:
- Baseline: `760b14ee62c` (benchmark-only commit on `f054492cbb9`)
- Candidate: `5b45223bc61`
- JDK 17.0.20.1, JMH 1.37, macOS arm64
- 1 thread, 3 forks, `-Xms1g -Xmx1g`, 4 x 300 ms warmup, 7 x 400 ms
measurement, `-prof gc`
- Baseline is the average of two complete runs (B1/B2). Candidate error is
JMH's 99.9% confidence interval.
- A preliminary candidate run affected by host interference was discarded
(`typical` parser-only was 88.8 us/op instead of the stable 7-9 us/op range);
the final artifact was rebuilt and measured in a clean run.
Build and run:
```shell
LC_ALL=en_US.UTF-8 mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark
-am \
-Dmaven.build.cache.enabled=false package
java -jar
fe/fe-sql-parser-benchmark/target/fe-sql-parser-benchmark-*-benchmark.jar \
'IdentifierPostProcessorBenchmark.*' -prof gc -rf json -rff result.json
```
Time is `us/op`; positive change means faster. The workloads cover a control
query, a typical aggregate query, a 64-column query, a
non-reserved-keyword-heavy query, and a quoted-identifier-heavy query.
| Path / workload | Baseline B1 / B2 | Candidate | Change | B/op baseline ->
candidate |
|---|---:|---:|---:|---:|
| end-to-end / control | 2.817 / 2.694 | 2.191 +/- 0.041 | +25.8% | 4,933.4
-> 4,848.0 (-1.73%) |
| end-to-end / typical | 17.143 / 18.288 | 17.173 +/- 1.742 | +3.2% |
18,754.0 -> 18,658.9 (-0.51%) |
| end-to-end / wide | 66.555 / 72.165 | 60.816 +/- 0.731 | +14.0% |
141,501.2 -> 141,432.9 (-0.05%) |
| end-to-end / nonReserved | 12.278 / 14.034 | 11.044 +/- 0.276 | +19.1% |
22,844.4 -> 22,530.9 (-1.37%) |
| end-to-end / quoted | 13.154 / 11.128 | 8.089 +/- 1.121 | +50.1% |
16,081.5 -> 15,984.1 (-0.61%) |
| parser-only / control | 1.703 / 1.841 | 1.550 +/- 0.016 | +14.3% | 3,824.0
-> 3,744.0 (-2.09%) |
| parser-only / typical | 8.655 / 8.565 | 7.868 +/- 0.096 | +9.4% | 13,872.1
-> 13,792.1 (-0.58%) |
| parser-only / wide | 60.700 / 59.489 | 55.784 +/- 0.624 | +7.7% |
115,632.9 -> 115,552.8 (-0.07%) |
| parser-only / nonReserved | 10.127 / 10.148 | 9.863 +/- 0.756 | +2.8% |
19,261.5 -> 19,189.5 (-0.37%) |
| parser-only / quoted | 6.985 / 7.116 | 6.658 +/- 0.186 | +5.9% | 13,440.1
-> 13,346.8 (-0.69%) |
The direct parser path improves by 2.8%-14.3% and allocates less in every
workload. End-to-end means also improve in every workload; the larger
quoted/control figures have more host-level variance and are not used as the
primary conclusion. The gain comes from removing global rule-exit listener
dispatch; the remaining allocation reduction comes from avoiding listener
bookkeeping.
Artifact/result hashes:
- Baseline benchmark jar:
`b9941cb95ea3cc8797d0ad896259049613e822c0f72319403a062bfc9bcb51a4`
- Candidate benchmark jar:
`2d33163832f90e0d1c1929e522e012becbab56e3bc463dcc19b42f0e9d27526a`
- B1 JSON: `3629629849730548e3e8ff5ab54bb9d02b14e3dde3aff3b8aa51c31cc74c8c5e`
- B2 JSON: `ca94f64515f2b904244f78139c7c43f636bff1f72f5daa00a748f7b9800167e6`
- Candidate JSON:
`98227ae4ce630bf306084573a2a315a5f3d3d0081c787a08c6eca9cea6c73000`
- Raw JSON retained locally under `/private/tmp/doris-p7-benchmark.YSnF7t/`.
### Semantic verification
- Parsed all 4,610 tracked `.sql` files in default and ANSI modes.
Acceptance/error class/error position/statement count matched the frozen
baseline in all 9,220 cases.
- Tracked SQL path-list SHA-256:
`c1fdf48a22d311f7164516b512e850bef373126be1d3079148c109e5f80f80c1`.
- Detailed CST/token/error snapshots cover ordinary, non-reserved, quoted,
doubled-backtick and multipart identifiers, expressions, DDL, both parser entry
paths, and `test-table` / `test-tbl` errors. Every rule node and token field
matched the baseline.
- Verified the advanced parser path with `buildParseTree=false`.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- `mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark -am
-Dmaven.build.cache.enabled=false package` (193 parser tests passed)
- `./run-fe-ut.sh --run
org.apache.doris.nereids.parser.NereidsParserTest`
- [x] Manual test
- Baseline/candidate corpus, CST/token/error differential checks,
and JMH benchmark described above
- `./build.sh --fe`
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
