TIL that `mix test --only async --exclude external_api` will say that the
external_api tag is skipped, but go ahead and run those cases anyway.
*Example code:*
```
defmodule OnlyExceptTest do use ExUnit.Case, async: true test "normal async
test" do assert true end @tag :skip test "this test should be excluded when
using --exclude skip_me" do flunk("This test should have been excluded!")
end end
```
# run `mix test --only async --except skip`
I would expect only the passing test to run, but in fact, both run, causing
an overall failure of the task. Jon R explained that this is due to the way
--only uses --include under the hood, which takes precedence over
--exclude, meaning additional exclude flags have no effect. The explanation
makes sense, as does the workaround of inverting the logic to pass an
unified --exclude async:false --exclude test --exclude skip.
However, it goes against the principle of least surprise, and creates a
subtle footgun, even for experienced developers.
*Real-World Use Case*
In CI, I have separate jobs for sync and async tests. A coworker added
tests for an external API and put an exclude in our `test_helper.exs`.
However, since `--exclude` is superseded by `--include`, and I was using
`--only async:true` in my CI job, the API tests ran and caused CI failures.
It took a message to the Elixir Slack, and a helpful reply to unblock me.
I thought we might be able to prevent the confusion by preventing `exclude`
from being combined with `only`, but I'm not sure how that would work with
`test_helper` usage.
However, it would still be really nice to have proper Ecto-style set
operations on mix test.
*Slack thread*
: https://elixir-lang.slack.com/archives/C03EPRA3B/p1765798909362919
*Illustrative Draft PR*: https://github.com/elixir-lang/elixir/pull/15014
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/elixir-lang-core/0ea6ac19-6ec6-441e-a8a3-e8e33b79c7d7n%40googlegroups.com.
