Vladimir Sitnikov created CALCITE-7942:
------------------------------------------

             Summary: Report the lines a branch changes that no test executes 
or checks
                 Key: CALCITE-7942
                 URL: https://issues.apache.org/jira/browse/CALCITE-7942
             Project: Calcite
          Issue Type: Improvement
          Components: build
            Reporter: Vladimir Sitnikov


When I change code in core, {{./gradlew test}} does not tell me which of the 
lines I changed no test executed, or which of them no test would notice if they 
were broken. A reviewer has the same question and has to answer it by reading 
the tests. {{-PenableJacoco}} and the SonarCloud analysis from CALCITE-5427 
report coverage of whole modules, not of the lines a branch changed.

h3. What it looks like on a real change

I ran the [cribro|https://github.com/cribro-dev/cribro] Gradle plugin through 
an init script, with no change to the build, on [PR 
#5419|https://github.com/apache/calcite/pull/5419] (CALCITE-7940):

{noformat}
./gradlew --init-script ~/.gradle/cribro/cribro.init.gradle :core:test --tests 
org.apache.calcite.test.CoreQuidemTest --tests 
org.apache.calcite.test.SqlValidatorTest
{noformat}

The run ends with this report (trimmed):

{noformat}
  core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java
    20/24 changed lines executed, 9/12 branches covered
    uncovered changed code:
      779: 1 of 2 branches covered: if (expanded != id) {
      782: not executed: children = ((ListScope) getJoinScope(input)).children;
      797: not executed: throw new AssertionError("column " + name + " not 
found in " + input);
{noformat}

{{:core:pitestDiff 
-Pcribro.pitestTargetTests=org.apache.calcite.test.SqlValidatorTest,org.apache.calcite.test.CoreQuidemTest}}
 runs PIT mutation testing on the changed lines only. It reported "8 mutants: 8 
killed, 0 survived" in 462 seconds on an M1 Max.

h3. Proposal

One possible shape: apply {{io.github.cribro-dev.cribro}} 1.1.1 in 
{{settings.gradle.kts}}, with the version in {{gradle.properties}} like the 
other plugins, and add a paragraph to {{site/_docs/howto.md}} and 
{{AGENTS.md}}. Then:
* {{test}} runs with the JaCoCo agent and prints the report above. It never 
fails the build.
* {{pitestDiff}} runs only when asked for and is not part of {{check}}.
* {{-Pcribro.diffCoverageEnabled=false}} runs {{test}} without JaCoCo and skips 
the report.
* {{./gradlew -q cribro}} prints the commands and properties.

Removing the plugin line reverts the change.

h3. Cost

JaCoCo makes {{:core:test}} noticeably slower. I timed {{./gradlew --no-daemon 
--init-script ~/.gradle/cribro/cribro.init.gradle :core:test --rerun}} twice 
with diff coverage on and twice with {{-Pcribro.diffCoverageEnabled=false}}, in 
on-off-off-on order. The machine is an M1 Max with 10 cores, which was also 
running background jobs at a lower priority, with a load average between 80 and 
180:

||Diff coverage||Test run, s||Whole build, wall, s||Whole build, user+sys CPU, 
s||
|on|292, 329|332, 369|775, 777|
|off|204, 202|234, 232|547, 555|

That is about 40% more CPU time and 50% more wall time, report generation 
included. The load makes the wall times rough; the CPU times agree within 2%. 
The slowdown is spread across the suite rather than caused by a few tests: test 
classes that run 1.3 to 1.7 times slower account for 83% of the added time. The 
parser tests slow down the most, 2.3 to 3.9 times, and account for 11%. To keep 
CI time unchanged, CI could set {{-Pcribro.diffCoverageEnabled=false}} on every 
job but one Linux job. Contributors who do not want the cost locally can set 
the same property in {{~/.gradle/gradle.properties}}. Out of scope: coverage 
thresholds, quality gates, and the Sonar job.

h3. Prior use

NullAway has had diff coverage and diff mutation testing in its build since 
October 2026 ([uber/NullAway#1833|https://github.com/uber/NullAway/pull/1833], 
[uber/NullAway#1975|https://github.com/uber/NullAway/pull/1975]). On November 
13 the code moved out of NullAway's buildSrc into cribro 
([uber/NullAway#2157|https://github.com/uber/NullAway/pull/2157]). On November 
5, Manu Sridharan, NullAway's lead maintainer, proposed the same for the 
Checker Framework 
([typetools/checker-framework#8483|https://github.com/typetools/checker-framework/issues/8483]),
 noting that it "has been quite helpful particularly in guiding agents towards 
generating more thorough tests".

I wrote cribro; it is Apache-2.0 licensed and published on the Gradle Plugin 
Portal. One known limitation affects Calcite: {{pitestDiff}} leaves out test 
JVM arguments that contain a comma or a space, such as 
{{-Djunit.jupiter.execution.timeout.default=5 m}} 
([hcoles/pitest#1511|https://github.com/hcoles/pitest/issues/1511], 
[cribro-dev/cribro#18|https://github.com/cribro-dev/cribro/issues/18]).

Done means: on a branch, {{./gradlew :core:test}} prints the changed lines no 
test executed, and CI jobs that disable diff coverage take as long as before.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to