Vladimir Sitnikov created CALCITE-7892:
------------------------------------------
Summary: CI test jobs cost about 230 runner-minutes per commit and
test the same configurations on every run
Key: CALCITE-7892
URL: https://issues.apache.org/jira/browse/CALCITE-7892
Project: Calcite
Issue Type: Improvement
Components: ci
Reporter: Vladimir Sitnikov
h3. Problem
One commit of [PR #5350|https://github.com/apache/calcite/pull/5350] used about
230 runner-minutes, and its Linux jobs waited 2 hours in the ASF queue. Most of
that comes from the 11 test jobs in {{.github/workflows/main.yml}}:
* Windows: JDK 8, 17, 21
* Linux: JDK 8 with the oldest Guava, JDK 8 with the latest Guava, JDK 11, 17,
21, 24
* Linux: JDK 11 with Avatica main
* macOS: JDK 21
Each job is a copy of another with one or two values changed. The set has
drifted:
* It tests JDK 24, a non-LTS release that no longer gets updates, and does not
test JDK 25.
* It tests only two Guava versions, the oldest ({{21.0}}) and the latest
({{33.4.8-jre}}).
* Every job uses the Zulu JDK and an English locale.
* Every commit runs the same 11 combinations, so a combination outside them is
never tested.
Adding a JDK or a Guava version means copying a 30-line job and editing it by
hand.
h3. Proposal
Generate the test jobs with
[github-actions-random-matrix|https://github.com/vlsi/github-actions-random-matrix].
A short script lists the axes (OS, the LTS JDKs and the latest GA JDK, Guava
versions from {{21.0}} to the latest, time zone, locale, JDK vendor,
{{-XX:hashCode=2}}, {{-XX:ActiveProcessorCount=1}}) and the rows every run must
include: Windows, JDK 8, the latest JDK, the oldest Guava, and the latest
Guava. The library fills the remaining jobs with random rows, choosing rows
that cover value pairs no earlier row covers.
The static analysis jobs (ErrorProne, Checker Framework) and the Avatica job
stay as they are.
Apache JMeter, pgjdbc, and TestNG generate their CI matrix with this library.
h3. Acceptance criteria
* A commit to {{main}} runs 5 or 6 generated test jobs and the Avatica job
instead of today's 11 test jobs, and uses about half the runner-minutes of
today's set (estimate: 110–125).
* Every run includes Windows, JDK 8, the latest JDK, the oldest Guava, and the
latest Guava.
* Each job name lists its parameters, and the log prints the seed that
generated the matrix.
* A pull request keeps the same rows across pushes. Each commit to {{main}}
draws new rows.
* Adding a JDK or a Guava version is a one-line change.
h3. Out of scope
* The static analysis jobs and the Avatica job.
* The ASF-wide runner queue.
* Running CI only once for a pull request from a branch in apache/calcite; that
is CALCITE-7884.
h3. Prior discussion
* dev@ thread "Calcite CI pending delays" (October 2026):
https://lists.apache.org/thread/3vglxtojsqh2m03fhcdv76s3v1lsqzmf
* Proposed on dev@ in 2021:
https://lists.apache.org/thread.html/gw7klmm4000y6lnbh3yjqj1g1ljswgt5
* CALCITE-4790 (2021): Alessandro Solimando suggested adding time zones as a CI
axis.
* CALCITE-4259 (2020): Julian Hyde wrote that he tested each JDK against random
Guava versions every night.
* CALCITE-4816 (2021): Julian Hyde started running tests in various time zones
and Guava versions on Travis.
* CALCITE-6580 (2024): Julian Hyde: "the best way to test in multiple locales
is a CI run with {{-Duser.country=CA -Duser.language=fr}}".
I maintain the library and can prepare the pull request.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
