[
https://issues.apache.org/jira/browse/GROOVY-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114452#comment-18114452
]
ASF GitHub Bot commented on GROOVY-12398:
-----------------------------------------
daniellansun commented on PR #2921:
URL: https://github.com/apache/groovy/pull/2921#issuecomment-5639709184
# Parser performance verification: GROOVY-12398
**Question.** Does commit `4c89f6774d5cbb431940b0ca9d1581a9434b0d1e` parse a
fixed Groovy corpus faster than commit
`4aa7b94b83bd29814b1449cde1b239bdbe8c0ff0`, when both use the same parse-only
path that production uses (SLL, then LL on failure)?
**Answer.** Yes. On this host, with a frozen corpus of 245 files (570 533
characters), the target parser used **about 35% less wall time** than the
baseline (pooled mean **1.53×**). Every post-warmup sample of the target was
faster than every post-warmup sample of the baseline. Parse outcomes (SLL
success, LL fallback, the one expected failure) are identical.
This is a **parse-only** measurement. It does not claim the same factor for
end-to-end `groovyc`.
---
## 1. What was compared
| | Baseline | Target |
|---|---|---|
| Commit | `4aa7b94b83bd29814b1449cde1b239bdbe8c0ff0` |
`4c89f6774d5cbb431940b0ca9d1581a9434b0d1e` |
| Subject | `prepare for version 7 (cont'd)` | `GROOVY-12398: classify
supplementary-plane identifiers by code point` |
| Raw jar SHA-256 |
`83130ad2d6ce26e1fee7eccca7a9f387151142defb923d3af1087f324cb7f8c7` |
`17946a113b5123939a6ecff762a3ffa62535bbb8d0444e39a928acf7a736cbe8` |
The range contains two commits:
1. `cfcc6d50fc` — lexer/parser grammar and predicate work (the performance
change).
2. `4c89f6774d` — review follow-up (code-point capitalization, invalid-octal
caret, tests). That follow-up is not expected to move the hot path; the
measurement is of the **full** GROOVY-12398 series against the parent of the
first commit.
Each commit was built in its own Git worktree (`./gradlew :jar`). The jars
were checked before timing: baseline still contains `invalidDigitCount` and
`java.util.regex.Pattern`; target contains `lastConsumedCodePoint` and
`isIdentifierAssign`.
---
## 2. Method
### Corpus
`src/main/groovy` + `src/test-resources/core`, `*.groovy` only, walked in
sorted path order.
The tree was **frozen from the baseline commit** and reused for every run,
so a change in source text cannot be mistaken for a parser change. The same two
directories at the target commit hash to the same SHA-256 (39 + 206 files).
Input is therefore identical in both trees.
| Root | Files | UTF-16/UTF-8 chars | Bytes |
|---|---:|---:|---:|
| `src/main/groovy` | 39 | 290 264 | 290 270 |
| `src/test-resources/core` | 206 | 280 269 | 280 277 |
| **Total** | **245** | **570 533** | **570 547** |
### Parse path
Matches `AstBuilder`:
1. `GroovyLangLexer` + `CommonTokenStream` + `GroovyLangParser`
2. `BailErrorStrategy`, no console listeners
3. `PredictionMode.SLL` first
4. On failure: `tokens.seek(0)`, `parser.reset()`, `PredictionMode.LL`
5. New lexer/parser per file; ATN/DFA shared through `ParserAtnManager`
(production-like cache warmup)
ANTLR runtime: `me.sunlan:antlr4-runtime:4.13.2.16` (the runtime Groovy
already uses).
### Timing protocol
- Independent JVM per invocation (`-Xms1g -Xmx1g -XX:+AlwaysPreTouch`).
- 5 untimed warmup passes (DFA fill + JIT), then **15 timed** full-corpus
passes.
- Wall time is `System.nanoTime()` around the whole corpus, not per file.
- Invocation order was interleaved **A B A B A B A** (baseline / target /
baseline / …) so a slow host cannot be blamed on “target ran first while the
machine was idle”.
- 4 baseline JVMs × 15 = **60** timed samples; 3 target JVMs × 15 = **45**
timed samples.
### Host
| | |
|---|---|
| Kernel | Linux 6.15.5 x86_64 (`hera`) |
| CPU | AMD EPYC 7763, 6 visible CPUs, 1 thread/core |
| Memory | 24 GiB |
| JVM | Corretto 25.0.4+7-LTS, mixed mode |
| Date | 2026-09-11 (UTC) |
This is a shared developer VM. Variance across baseline JVMs is reported
rather than hidden.
---
## 3. Correctness (same corpus, both parsers)
| Outcome | Baseline | Target |
|---|---:|---:|
| SLL success | 241 | 241 |
| LL fallback | 3 | 3 |
| Hard failure | 1 | 1 |
LL fallback (both parsers):
- `src/test-resources/core/Command_01.groovy`
- `src/test-resources/core/Command_05.groovy`
- `src/test-resources/core/Lambda_01x.groovy`
Hard failure (both parsers): `src/test-resources/core/Unicode_01.groovy` —
`GroovySyntaxError: Illegal escape character: '\u'`. That file is an
illegal-escape fixture, not a regression.
The speedup is **not** from fewer LL retries. The SLL/LL split is unchanged.
---
## 4. Results
Times are seconds to parse the whole corpus after warmup. Lower is better.
### Per JVM (15 timed rounds each)
| Run | Role | Min | Median | Mean | Stdev | Max |
|---|---|---:|---:|---:|---:|---:|
| A | baseline | 0.218082 | 0.233826 | 0.242895 | 0.022486 | 0.293010 |
| B | target | 0.148911 | 0.177118 | 0.177364 | 0.019880 | 0.211886 |
| A2 | baseline | 0.254599 | 0.295294 | 0.300075 | 0.036579 | 0.381228 |
| B2 | target | 0.147442 | 0.165518 | 0.166205 | 0.009369 | 0.185643 |
| A3 | baseline | 0.222311 | 0.236567 | 0.242742 | 0.018481 | 0.291574 |
| B3 | target | 0.147241 | 0.162760 | 0.167201 | 0.016246 | 0.204381 |
| A4 | baseline | 0.219386 | 0.260223 | 0.255703 | 0.027844 | 0.326735 |
A2 is a noisy baseline JVM (mean 0.300 s vs ~0.243 s for A and A3). It is
kept in the pool rather than discarded.
### Pooled post-warmup samples
| | n | Min | Median | Mean | Stdev | Max |
|---|---:|---:|---:|---:|---:|---:|
| Baseline | 60 | 0.218082 | 0.252503 | **0.260354** | 0.035573 | 0.381228 |
| Target | 45 | 0.147241 | 0.165395 | **0.170257** | 0.016237 | 0.211886 |
### Speedup
| Estimator | Baseline | Target | Factor | Time saved |
|---|---:|---:|---:|---:|
| Mean | 0.260354 s | 0.170257 s | **1.53×** | 34.6% |
| Median | 0.252503 s | 0.165395 s | **1.53×** | 34.5% |
| Best (min) | 0.218082 s | 0.147241 s | **1.48×** | 32.5% |
Throughput at the pooled means: **941 → 1 439 files/s**, **2.19e6 → 3.35e6
chars/s**.
### Separation
The slowest target round (0.211886 s) is still faster than the fastest
baseline round (0.218082 s). The two sample sets do not overlap.
Cohen’s *d* on the pooled samples is **3.1** (large). That is a description
of this data set, not a claim about every machine.
### Conservative cross-check
Against the **fastest** baseline JVM mean (A3, 0.242742 s) and the
**slowest** target JVM mean (B, 0.177364 s), the factor is still **1.37×**.
That is the least flattering pairing of JVM means in this run.
---
## 5. What this does and does not show
**Shown**
- On this host, the GROOVY-12398 parser is consistently faster on this
corpus than its parent, by roughly one and a half times in parse-only wall time.
- The improvement is in the SLL-successful majority (241/245 files), not in
a change of prediction mode mix.
- The corpus and the parse outcomes are the same, so the delta is in the
recognisers, not in “different files” or “more failures skipped”.
**Not shown**
- End-to-end compile time (semantic analysis, classgen, I/O). Parse is only
part of `groovyc`; a 1.5× parse-only gain will shrink to a smaller fraction of
a full compile, as already noted on the PR.
- Other hardware, JDK 17/21, or a Grails/Spock-sized corpus. Those would be
the right next measurement if the change is considered for 6.0.
- That every future Groovy file will see 1.5×. Command-heavy scripts that
already fall back to LL (three files here) are not the bulk of this corpus.
**Noise**
Baseline JVM A2 was ~24% slower than A/A3. The machine is not a quiet
benchmark box. Interleaving and pooling are how that is handled; dropping A2
would only make the target look better, so it was not dropped.
---
## 6. Conclusion
Relative to `4aa7b94b83`, commit `4c89f6774d` (GROOVY-12398 as it stands on
that branch) delivers a **real, repeatable parse-only speedup** of about
**1.5×** (≈35% less time) on `src/main/groovy` + `src/test-resources/core`.
Parse behaviour on that corpus is unchanged.
That is sufficient evidence for a **master** merge from a performance
standpoint. It is **not** by itself a case for slipping the same grammar/ATN
rewrite into a 6.0 RC without a further compile-level corpus check.
---
## Appendix: how to reproduce
```bash
# worktrees
git worktree add --detach /tmp/parse-verify/wt-baseline
4aa7b94b83bd29814b1449cde1b239bdbe8c0ff0
git worktree add --detach /tmp/parse-verify/wt-target
4c89f6774d5cbb431940b0ca9d1581a9434b0d1e
( cd /tmp/parse-verify/wt-baseline && ./gradlew :jar --offline -q )
( cd /tmp/parse-verify/wt-target && ./gradlew :jar --offline -q )
# freeze corpus from baseline
mkdir -p /tmp/parse-verify/corpus
cp -a /tmp/parse-verify/wt-baseline/src/main/groovy
/tmp/parse-verify/corpus/main-groovy
cp -a /tmp/parse-verify/wt-baseline/src/test-resources/core
/tmp/parse-verify/corpus/test-resources-core
ANTLR=$HOME/.gradle/caches/modules-2/files-2.1/me.sunlan/antlr4-runtime/4.13.2.16/9c7caa836c70ef09ee7b70f77c11b803f75e19c0/antlr4-runtime-4.13.2.16.jar
# ParseBench.java as used for this report (5 warmup, 15 rounds, SLL then LL)
java -Xms1g -Xmx1g -XX:+AlwaysPreTouch -Dbench.label=... \
-cp ".:<raw-jar>:$ANTLR" ParseBench \
/tmp/parse-verify/corpus/main-groovy \
/tmp/parse-verify/corpus/test-resources-core
```
> Improve lexer and parser performance
> ------------------------------------
>
> Key: GROOVY-12398
> URL: https://issues.apache.org/jira/browse/GROOVY-12398
> Project: Groovy
> Issue Type: Improvement
> Reporter: Daniel Sun
> Priority: Major
>
> The Groovy ANTLR4 grammar already uses left-recursive expressions, which is
> the main idea of the [optimized Java
> grammar|https://github.com/antlr/grammars-v4/tree/master/java/java]. Several
> remaining lexer and parser costs still match patterns that grammar was
> written to avoid.
> h3. Problem
> * Number literals are assembled from a chain of one-character fragments
> ({{Digit}}, {{Zero}}, {{NonZeroDigit}}, ...). The invalid-octal alternative
> also counts digits in a mid-rule action loop, which prevents DFA compilation
> of {{IntegerLiteral}}.
> * Every identifier, including ASCII names such as {{Foo}} and {{foo}},
> evaluates {{Character.isUpperCase}} to choose {{CapitalizedIdentifier}} vs
> {{Identifier}}.
> * GString and comment predicates ({{isFollowedByJavaLetterInGString}},
> {{isFollowedByWhiteSpaces}}) allocate a one-character {{String}} and run
> {{Pattern.matcher}} on each lookahead.
> * {{elementValues}} is ambiguous with assignment: {{@Foo(a = 1)}} can be
> named pairs or an assignment expression, so {{AdaptivePredict}} explores both
> (see [antlr/grammars-v4#215|https://github.com/antlr/grammars-v4/issues/215]).
> h3. Approach
> Keep token types and language semantics unchanged.
> * Flatten number and float fragments to character classes, as in the
> optimized Java lexer. Validate invalid octal once at the end of the
> alternative.
> * Split ASCII identifiers on a predicate-free DFA path: uppercase {{A-Z}}
> becomes {{CapitalizedIdentifier}}; {{a-z}}, dollar and underscore become
> {{Identifier}}. Unicode still uses predicates.
> * Replace the regex-based predicates with direct character tests.
> * Gate {{elementValue}} with {{!isIdentifierAssign(_input)}}, the same idea
> as Java's {{IsNotIdentifierAssign}}.
> h3. Result
> Local parse-only measurement (SLL then LL) of {{src/main/groovy}} +
> {{src/test-resources/core}} (245 files, approximately 570k characters):
> ||Run||Best||Mean||
> |before|0.231s|0.401s|
> |after|0.146s|0.169s|
> Best run is about 1.6x faster. Mean improves more because prediction is more
> stable after DFA warmup.
> Related: GROOVY-10454, GROOVY-9588
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
