Vladimir Sitnikov created CALCITE-7942:
------------------------------------------
Summary: Report the lines a branch changes that no test executes
or checks
Key: CALCITE-7942
URL: https://issues.apache.org/jira/browse/CALCITE-7942
Project: Calcite
Issue Type: Improvement
Components: build
Reporter: Vladimir Sitnikov
When I change code in core, {{./gradlew test}} does not tell me which of the
lines I changed no test executed, or which of them no test would notice if they
were broken. A reviewer has the same question and has to answer it by reading
the tests. {{-PenableJacoco}} and the SonarCloud analysis from CALCITE-5427
report coverage of whole modules, not of the lines a branch changed.
h3. What it looks like on a real change
I ran the [cribro|https://github.com/cribro-dev/cribro] Gradle plugin through
an init script, with no change to the build, on [PR
#5419|https://github.com/apache/calcite/pull/5419] (CALCITE-7940):
{noformat}
./gradlew --init-script ~/.gradle/cribro/cribro.init.gradle :core:test --tests
org.apache.calcite.test.CoreQuidemTest --tests
org.apache.calcite.test.SqlValidatorTest
{noformat}
The run ends with this report (trimmed):
{noformat}
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java
20/24 changed lines executed, 9/12 branches covered
uncovered changed code:
779: 1 of 2 branches covered: if (expanded != id) {
782: not executed: children = ((ListScope) getJoinScope(input)).children;
797: not executed: throw new AssertionError("column " + name + " not
found in " + input);
{noformat}
{{:core:pitestDiff
-Pcribro.pitestTargetTests=org.apache.calcite.test.SqlValidatorTest,org.apache.calcite.test.CoreQuidemTest}}
runs PIT mutation testing on the changed lines only. It reported "8 mutants: 8
killed, 0 survived" in 462 seconds on an M1 Max.
h3. Proposal
One possible shape: apply {{io.github.cribro-dev.cribro}} 1.1.1 in
{{settings.gradle.kts}}, with the version in {{gradle.properties}} like the
other plugins, and add a paragraph to {{site/_docs/howto.md}} and
{{AGENTS.md}}. Then:
* {{test}} runs with the JaCoCo agent and prints the report above. It never
fails the build.
* {{pitestDiff}} runs only when asked for and is not part of {{check}}.
* {{-Pcribro.diffCoverageEnabled=false}} runs {{test}} without JaCoCo and skips
the report.
* {{./gradlew -q cribro}} prints the commands and properties.
Removing the plugin line reverts the change.
h3. Cost
JaCoCo makes {{:core:test}} noticeably slower. I timed {{./gradlew --no-daemon
--init-script ~/.gradle/cribro/cribro.init.gradle :core:test --rerun}} twice
with diff coverage on and twice with {{-Pcribro.diffCoverageEnabled=false}}, in
on-off-off-on order. The machine is an M1 Max with 10 cores, which was also
running background jobs at a lower priority, with a load average between 80 and
180:
||Diff coverage||Test run, s||Whole build, wall, s||Whole build, user+sys CPU,
s||
|on|292, 329|332, 369|775, 777|
|off|204, 202|234, 232|547, 555|
That is about 40% more CPU time and 50% more wall time, report generation
included. The load makes the wall times rough; the CPU times agree within 2%.
The slowdown is spread across the suite rather than caused by a few tests: test
classes that run 1.3 to 1.7 times slower account for 83% of the added time. The
parser tests slow down the most, 2.3 to 3.9 times, and account for 11%. To keep
CI time unchanged, CI could set {{-Pcribro.diffCoverageEnabled=false}} on every
job but one Linux job. Contributors who do not want the cost locally can set
the same property in {{~/.gradle/gradle.properties}}. Out of scope: coverage
thresholds, quality gates, and the Sonar job.
h3. Prior use
NullAway has had diff coverage and diff mutation testing in its build since
October 2026 ([uber/NullAway#1833|https://github.com/uber/NullAway/pull/1833],
[uber/NullAway#1975|https://github.com/uber/NullAway/pull/1975]). On November
13 the code moved out of NullAway's buildSrc into cribro
([uber/NullAway#2157|https://github.com/uber/NullAway/pull/2157]). On November
5, Manu Sridharan, NullAway's lead maintainer, proposed the same for the
Checker Framework
([typetools/checker-framework#8483|https://github.com/typetools/checker-framework/issues/8483]),
noting that it "has been quite helpful particularly in guiding agents towards
generating more thorough tests".
I wrote cribro; it is Apache-2.0 licensed and published on the Gradle Plugin
Portal. One known limitation affects Calcite: {{pitestDiff}} leaves out test
JVM arguments that contain a comma or a space, such as
{{-Djunit.jupiter.execution.timeout.default=5 m}}
([hcoles/pitest#1511|https://github.com/hcoles/pitest/issues/1511],
[cribro-dev/cribro#18|https://github.com/cribro-dev/cribro/issues/18]).
Done means: on a branch, {{./gradlew :core:test}} prints the changed lines no
test executed, and CI jobs that disable diff coverage take as long as before.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)