jamesfredley commented on PR #16158: URL: https://github.com/apache/grails-core/pull/16158#issuecomment-5309752640
## CI measurement, second sample - correcting the first A second run of the same commit landed (59/59 jobs green again). It does **not** reproduce the improvement reported above, and the earlier numbers should be treated as retracted. | Job | `8.0.x` baseline median | Sample 1 | Sample 2 | Mean vs baseline | |---|---:|---:|---:|---:| | Ubuntu JDK 21 | 84.8 min | 59.5 | 80.4 | -17.5% | | Ubuntu JDK 25 | 81.0 min | 77.4 | 75.5 | -5.6% | | macOS JDK 21 | 77.9 min | 62.5 | 89.9 | -2.2% | | Windows JDK 25 shard 0 | 77.1 min | 95.4 | 96.6 | **+24.5%** | | Windows JDK 25 shard 1 | 43.1 min | 43.5 | 45.1 | +2.8% | | Windows JDK 25 shard 2 | 45.4 min | 36.6 | 44.2 | -11.0% | ### What this actually shows **Runner variance dominates.** macOS JDK 21 moved from 62.5 to 89.9 minutes on *identical code* - a 35 percentage point swing between two runs. Any single-sample CI comparison on this workflow, including my first one, is noise. The honest reading of two samples is that CI wall clock is roughly unchanged. **Windows JDK 25 shard 0 is the one consistent signal**, slower in both samples (+23.7%, +25.3%). Two samples is still thin, but it is the only job where both point the same way, so it deserves attention rather than dismissal. Worth noting the baseline for that job spans 66.8-93.7 minutes across three runs, so even this may be variance. ### Does this invalidate the change? Not the local result, which is the stronger evidence and was measured under controlled conditions - idle machine, daemons stopped between runs, and critically **run in both orderings** so filesystem-cache bias pointed against the change in one of them: ``` baseline first: baseline 3m25s, this branch 2m45s (19.5% faster) branch first: this branch 2m24s, baseline 3m05s (22.2% faster) ``` That reproducibility is what a shared GitHub runner cannot offer. It is also worth restating what CI is even exercising here: **fork counts are unchanged on CI by design** (`isCiBuild ? 4`), so the only CI-visible effect is the per-fork `ActiveProcessorCount` cap - 4 processors down to 2 on a runner, versus 20 down to 2 locally. A small or unmeasurable CI effect is the expected outcome, not a contradiction. The developer-machine problem this PR exists to fix - 23 JVMs, 12.4 GB, and roughly 465 threads on a 20-thread box - is unaffected by any of this. ### Suggested next step If CI timing is a merge criterion, this needs several more samples per job to say anything, particularly for Windows shard 0. If it is not, the local A/B plus unchanged CI fork counts should be sufficient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
