daniellansun commented on PR #2921:
URL: https://github.com/apache/groovy/pull/2921#issuecomment-5639709184
# Parser performance verification: GROOVY-12398
**Question.** Does commit `4c89f6774d5cbb431940b0ca9d1581a9434b0d1e` parse a
fixed Groovy corpus faster than commit
`4aa7b94b83bd29814b1449cde1b239bdbe8c0ff0`, when both use the same parse-only
path that production uses (SLL, then LL on failure)?
**Answer.** Yes. On this host, with a frozen corpus of 245 files (570 533
characters), the target parser used **about 35% less wall time** than the
baseline (pooled mean **1.53×**). Every post-warmup sample of the target was
faster than every post-warmup sample of the baseline. Parse outcomes (SLL
success, LL fallback, the one expected failure) are identical.
This is a **parse-only** measurement. It does not claim the same factor for
end-to-end `groovyc`.
---
## 1. What was compared
| | Baseline | Target |
|---|---|---|
| Commit | `4aa7b94b83bd29814b1449cde1b239bdbe8c0ff0` |
`4c89f6774d5cbb431940b0ca9d1581a9434b0d1e` |
| Subject | `prepare for version 7 (cont'd)` | `GROOVY-12398: classify
supplementary-plane identifiers by code point` |
| Raw jar SHA-256 |
`83130ad2d6ce26e1fee7eccca7a9f387151142defb923d3af1087f324cb7f8c7` |
`17946a113b5123939a6ecff762a3ffa62535bbb8d0444e39a928acf7a736cbe8` |
The range contains two commits:
1. `cfcc6d50fc` — lexer/parser grammar and predicate work (the performance
change).
2. `4c89f6774d` — review follow-up (code-point capitalization, invalid-octal
caret, tests). That follow-up is not expected to move the hot path; the
measurement is of the **full** GROOVY-12398 series against the parent of the
first commit.
Each commit was built in its own Git worktree (`./gradlew :jar`). The jars
were checked before timing: baseline still contains `invalidDigitCount` and
`java.util.regex.Pattern`; target contains `lastConsumedCodePoint` and
`isIdentifierAssign`.
---
## 2. Method
### Corpus
`src/main/groovy` + `src/test-resources/core`, `*.groovy` only, walked in
sorted path order.
The tree was **frozen from the baseline commit** and reused for every run,
so a change in source text cannot be mistaken for a parser change. The same two
directories at the target commit hash to the same SHA-256 (39 + 206 files).
Input is therefore identical in both trees.
| Root | Files | UTF-16/UTF-8 chars | Bytes |
|---|---:|---:|---:|
| `src/main/groovy` | 39 | 290 264 | 290 270 |
| `src/test-resources/core` | 206 | 280 269 | 280 277 |
| **Total** | **245** | **570 533** | **570 547** |
### Parse path
Matches `AstBuilder`:
1. `GroovyLangLexer` + `CommonTokenStream` + `GroovyLangParser`
2. `BailErrorStrategy`, no console listeners
3. `PredictionMode.SLL` first
4. On failure: `tokens.seek(0)`, `parser.reset()`, `PredictionMode.LL`
5. New lexer/parser per file; ATN/DFA shared through `ParserAtnManager`
(production-like cache warmup)
ANTLR runtime: `me.sunlan:antlr4-runtime:4.13.2.16` (the runtime Groovy
already uses).
### Timing protocol
- Independent JVM per invocation (`-Xms1g -Xmx1g -XX:+AlwaysPreTouch`).
- 5 untimed warmup passes (DFA fill + JIT), then **15 timed** full-corpus
passes.
- Wall time is `System.nanoTime()` around the whole corpus, not per file.
- Invocation order was interleaved **A B A B A B A** (baseline / target /
baseline / …) so a slow host cannot be blamed on “target ran first while the
machine was idle”.
- 4 baseline JVMs × 15 = **60** timed samples; 3 target JVMs × 15 = **45**
timed samples.
### Host
| | |
|---|---|
| Kernel | Linux 6.15.5 x86_64 (`hera`) |
| CPU | AMD EPYC 7763, 6 visible CPUs, 1 thread/core |
| Memory | 24 GiB |
| JVM | Corretto 25.0.4+7-LTS, mixed mode |
| Date | 2026-09-11 (UTC) |
This is a shared developer VM. Variance across baseline JVMs is reported
rather than hidden.
---
## 3. Correctness (same corpus, both parsers)
| Outcome | Baseline | Target |
|---|---:|---:|
| SLL success | 241 | 241 |
| LL fallback | 3 | 3 |
| Hard failure | 1 | 1 |
LL fallback (both parsers):
- `src/test-resources/core/Command_01.groovy`
- `src/test-resources/core/Command_05.groovy`
- `src/test-resources/core/Lambda_01x.groovy`
Hard failure (both parsers): `src/test-resources/core/Unicode_01.groovy` —
`GroovySyntaxError: Illegal escape character: '\u'`. That file is an
illegal-escape fixture, not a regression.
The speedup is **not** from fewer LL retries. The SLL/LL split is unchanged.
---
## 4. Results
Times are seconds to parse the whole corpus after warmup. Lower is better.
### Per JVM (15 timed rounds each)
| Run | Role | Min | Median | Mean | Stdev | Max |
|---|---|---:|---:|---:|---:|---:|
| A | baseline | 0.218082 | 0.233826 | 0.242895 | 0.022486 | 0.293010 |
| B | target | 0.148911 | 0.177118 | 0.177364 | 0.019880 | 0.211886 |
| A2 | baseline | 0.254599 | 0.295294 | 0.300075 | 0.036579 | 0.381228 |
| B2 | target | 0.147442 | 0.165518 | 0.166205 | 0.009369 | 0.185643 |
| A3 | baseline | 0.222311 | 0.236567 | 0.242742 | 0.018481 | 0.291574 |
| B3 | target | 0.147241 | 0.162760 | 0.167201 | 0.016246 | 0.204381 |
| A4 | baseline | 0.219386 | 0.260223 | 0.255703 | 0.027844 | 0.326735 |
A2 is a noisy baseline JVM (mean 0.300 s vs ~0.243 s for A and A3). It is
kept in the pool rather than discarded.
### Pooled post-warmup samples
| | n | Min | Median | Mean | Stdev | Max |
|---|---:|---:|---:|---:|---:|---:|
| Baseline | 60 | 0.218082 | 0.252503 | **0.260354** | 0.035573 | 0.381228 |
| Target | 45 | 0.147241 | 0.165395 | **0.170257** | 0.016237 | 0.211886 |
### Speedup
| Estimator | Baseline | Target | Factor | Time saved |
|---|---:|---:|---:|---:|
| Mean | 0.260354 s | 0.170257 s | **1.53×** | 34.6% |
| Median | 0.252503 s | 0.165395 s | **1.53×** | 34.5% |
| Best (min) | 0.218082 s | 0.147241 s | **1.48×** | 32.5% |
Throughput at the pooled means: **941 → 1 439 files/s**, **2.19e6 → 3.35e6
chars/s**.
### Separation
The slowest target round (0.211886 s) is still faster than the fastest
baseline round (0.218082 s). The two sample sets do not overlap.
Cohen’s *d* on the pooled samples is **3.1** (large). That is a description
of this data set, not a claim about every machine.
### Conservative cross-check
Against the **fastest** baseline JVM mean (A3, 0.242742 s) and the
**slowest** target JVM mean (B, 0.177364 s), the factor is still **1.37×**.
That is the least flattering pairing of JVM means in this run.
---
## 5. What this does and does not show
**Shown**
- On this host, the GROOVY-12398 parser is consistently faster on this
corpus than its parent, by roughly one and a half times in parse-only wall time.
- The improvement is in the SLL-successful majority (241/245 files), not in
a change of prediction mode mix.
- The corpus and the parse outcomes are the same, so the delta is in the
recognisers, not in “different files” or “more failures skipped”.
**Not shown**
- End-to-end compile time (semantic analysis, classgen, I/O). Parse is only
part of `groovyc`; a 1.5× parse-only gain will shrink to a smaller fraction of
a full compile, as already noted on the PR.
- Other hardware, JDK 17/21, or a Grails/Spock-sized corpus. Those would be
the right next measurement if the change is considered for 6.0.
- That every future Groovy file will see 1.5×. Command-heavy scripts that
already fall back to LL (three files here) are not the bulk of this corpus.
**Noise**
Baseline JVM A2 was ~24% slower than A/A3. The machine is not a quiet
benchmark box. Interleaving and pooling are how that is handled; dropping A2
would only make the target look better, so it was not dropped.
---
## 6. Conclusion
Relative to `4aa7b94b83`, commit `4c89f6774d` (GROOVY-12398 as it stands on
that branch) delivers a **real, repeatable parse-only speedup** of about
**1.5×** (≈35% less time) on `src/main/groovy` + `src/test-resources/core`.
Parse behaviour on that corpus is unchanged.
That is sufficient evidence for a **master** merge from a performance
standpoint. It is **not** by itself a case for slipping the same grammar/ATN
rewrite into a 6.0 RC without a further compile-level corpus check.
---
## Appendix: how to reproduce
```bash
# worktrees
git worktree add --detach /tmp/parse-verify/wt-baseline
4aa7b94b83bd29814b1449cde1b239bdbe8c0ff0
git worktree add --detach /tmp/parse-verify/wt-target
4c89f6774d5cbb431940b0ca9d1581a9434b0d1e
( cd /tmp/parse-verify/wt-baseline && ./gradlew :jar --offline -q )
( cd /tmp/parse-verify/wt-target && ./gradlew :jar --offline -q )
# freeze corpus from baseline
mkdir -p /tmp/parse-verify/corpus
cp -a /tmp/parse-verify/wt-baseline/src/main/groovy
/tmp/parse-verify/corpus/main-groovy
cp -a /tmp/parse-verify/wt-baseline/src/test-resources/core
/tmp/parse-verify/corpus/test-resources-core
ANTLR=$HOME/.gradle/caches/modules-2/files-2.1/me.sunlan/antlr4-runtime/4.13.2.16/9c7caa836c70ef09ee7b70f77c11b803f75e19c0/antlr4-runtime-4.13.2.16.jar
# ParseBench.java as used for this report (5 warmup, 15 rounds, SLL then LL)
java -Xms1g -Xmx1g -XX:+AlwaysPreTouch -Dbench.label=... \
-cp ".:<raw-jar>:$ANTLR" ParseBench \
/tmp/parse-verify/corpus/main-groovy \
/tmp/parse-verify/corpus/test-resources-core
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
