HairstonE commented on issue #23839:
URL: https://github.com/apache/datafusion/issues/23839#issuecomment-5330041292
There is more nuance to using cargo mutants than I initially thought. The
constraint is runtime.
Against unit tests it's fine. The suite is fast, so a contributor can run
mutants on their diff and get through it. Against slt it isn't. slt is slow
enough that running it once per mutant would take hours to over a day to run.
A fix could be a coverage map from each mutant to the slt files that touch
its path, so you only run those. That cuts runtime, but now that map needs to
be maintained. I don't think that is worth the trouble.
Running mutants against unit tests only is much faster but some "uncaught"
mutants are actually caught by slt tests and others are actual gaps.
Differentiating them needs a certain amount of codebase familiarity.
A possible solution to this issue would be to have only the surviving
mutants run against the slt suite. Running cargo mutants would have two phases.
Phase 1 — fast filter (unit tests only, on the diff):
```
git diff <merge-base> > pr.diff
cargo mutants --in-diff pr.diff -j 4 --timeout-multiplier 3 -- --lib
# survivors are listed in mutants.out/missed.txt (file:line + the mutation)
```
--in-diff bounds it to your changed lines; -j parallelizes; -- --lib runs
only the fast unit tests.
Phase 2 — run slt on only the survivors:
The reason this needs a wrapper: if you just re-run --in-diff pr.diff with
an slt-inclusive test command, cargo-mutants re-tests all the diff's mutants
(all 93), not the 23 survivors — that's the overnight run again. To stay fast
you have to feed only the survivor lines back in, which isn't a built-in flag.
The shape:
```
# 1. pull the surviving file:line locations out of phase 1's output
# (from mutants.out/missed.txt or outcomes.json)
# 2. build a narrowed diff containing only those lines (survivors.diff)
# 3. re-run mutants on just those, with a test command that includes slt:
cargo mutants --in-diff survivors.diff -j 4 --timeout-multiplier 3 \
--test-workspace true -- --test sqllogictests
```
Phase 2 could still take quite a bit of time depending on how good the
existing unit suite is. If there are too many survivors then this falls back to
being too much to do.
This feels like a decent option for ensuring correctness coverage but could
be too heavyweight to have an agent/contributor/both run for every PR. Everyone
please let me know your thoughts!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
