edmondop opened a new pull request, #27480:
URL: https://github.com/apache/datafusion/pull/27480

   > [!IMPORTANT]
   >
   > **Fork CI setup:** For pull requests from a fork to `main`, enable 
workflows in your fork's **Actions** tab before you push.
   > See the [fork CI setup 
guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks).
   
   ## Which issue does this PR close?
   
   - Part of #14589.
   - Follows the discussion in #26865.
   
   ## Rationale for this change
   
   In #26865, @2010YOUY01 noted that the earlier attempt (#14590) failed 
because a full cargo-mutants run over the workspace took too long. They offered 
to review CI that only mutates code a PR changes. Running it on a PR's own code 
also addresses @alamb's concern that someone has to watch and fix the results: 
the author who wrote the code sees the missed mutants while the change is still 
in review.
   
   ## What changes are included in this PR?
   
   - `.github/workflows/mutants.yml`: an advisory `pull_request` workflow. It 
is not in `.asf.yaml` required checks, and it only runs when `.rs` files 
change. It runs `cargo mutants --in-diff`, which mutates only the lines the PR 
changes.
   - `.cargo/mutants.toml`: reuses the existing `ci` profile.
   - `docs/source/contributor-guide/testing.md`: a "Mutation testing" section 
with the commands to run the check locally.
   
   ### Why the workflow runs two passes
   
   A mutant is _caught_ when some test fails with it in place. It is _missed_ 
when all tests still pass. Only missed mutants are reported.
   
   By default, cargo-mutants runs only the tests of the crate the mutant is in. 
Much of DataFusion is tested from elsewhere. Optimizer rules, for example, are 
mostly covered by `datafusion/core/tests` and `.slt` files. A mutant can 
therefore be missed by its own crate's tests but caught by sqllogictest.
   
   Running every mutant against core and sqllogictest would fix that, but each 
mutant then needs a rebuild of `datafusion` and `datafusion-sqllogictest` and a 
full run of their tests: 2.5–6 minutes per mutant, against seconds for a 
crate's own tests. So the workflow uses the cheap run as a filter:
   
   1. **Pass 1:** every mutant against its own crate's tests.
   2. **Pass 2:** only the mutants pass 1 missed, via `--iterate`, against 
`datafusion` and `datafusion-sqllogictest`. Skipped if pass 1 missed nothing.
   
   Mutants still missed after pass 2 are listed in the job summary, and the job 
fails. `mutants.out` is uploaded as an artifact.
   
   For #26716 (23 mutants), pass 1 took 1.5 minutes and left 3 missed mutants. 
Pass 2 caught 1 of them. Running all 23 against core and sqllogictest would 
take about 1.5 hours. Pass 1 alone would have reported one false miss.
   
   A test in a third crate (not the mutated crate, `datafusion` or 
sqllogictest) is still not run, so the check can report a mutant as missed when 
that test covers it.
   
   ### Settings needed to avoid false passes
   
   - **The baseline is kept, and checkout includes submodules.** cargo-mutants 
first runs the tests on the unmutated code (the baseline). With `--baseline 
skip` and no test data, every core and sqllogictest test failed. cargo-mutants 
then counted every mutant as caught, so the check passed without testing 
anything.
   - **Pass 2 sets `--timeout 1800`.** In cargo-mutants 27.1.0, the pass 2 
baseline runs the mutated crate's tests instead of the `--test-package` ones, 
and derives a 20 s timeout from them. Every mutant then timed out instead of 
being reported as missed, and the check passed.
   
   ## What is the testing strategy for this PR?
   
   I ran the workflow on two PRs inside my fork. Both are based on the commit 
before #26716.
   
   | Fork PR | Change under test | Result | Duration |
   |---|---|---|---|
   | 
[edmondop/arrow-datafusion#14](https://github.com/edmondop/arrow-datafusion/pull/14)
 ([run](https://github.com/edmondop/arrow-datafusion/actions/runs/43540514006)) 
| An unused function, so no test can catch its mutants | Fails, 5/5 missed | 19 
min |
   | 
[edmondop/arrow-datafusion#13](https://github.com/edmondop/arrow-datafusion/pull/13)
 ([run](https://github.com/edmondop/arrow-datafusion/actions/runs/43540517964)) 
| The change from #26716 | Fails, 2 of 23 missed | 20 min |
   
   In the #26716 run, pass 1 tested 23 mutants: 16 caught, 4 unviable, 3 
missed. Pass 2 caught one more. No unit, integration or `.slt` test fails for 
these two:
   
   ```
   enforce_distribution.rs:1242:34: replace > with >= in 
ChildPolicy::needs_hash_repartition
   enforce_distribution.rs:1788:23: replace || with && in 
ensure_distribution_with_stats
   ```
   
   Cost on `ubuntu-latest`:
   - About 4 minutes of setup.
   - About 1.5–3 minutes for pass 1.
   - About 2.5–6 minutes in pass 2 for each surviving mutant.
   
   A PR whose mutants are all caught in pass 1 skips pass 2.
   
   actionlint, shellcheck, the license header check, prettier and typos pass 
locally.
   
   ## Are there any user-facing changes?
   
   No. This adds a non-required CI check and a contributor guide section.
   
   ## Open questions
   
   - A PR from a fork runs this workflow in the base repository, on 
GitHub-hosted runners. The `Rust` workflow instead runs fork PRs on the fork. 
Is the extra runner time acceptable for an advisory check, or should this 
follow the fork-CI pattern?
   - Should the check fail (red, but not required), or only post the summary?
   
   This PR was written with AI assistance (Claude Code). I have read the whole 
change and the fork runs.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_01TAeJdSfRAQtKMjuAtTDnuH
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to