I like the idea of rethinking how these flags interact, though I do not like the proposed api. A string-based pseudo-query/natural language DSL feels overly complicated: hard to document, hard to maintain, not worth the potential benefits.
My intuition would be that we should be able to honor both includes and excludes in a sane way, but I'm unfamiliar with the trade-offs that led to the current design. However, note that any changes to this behaviour could be a breaking change to many test suites, so have to be released cautiously behind the right version number. On Monday, December 15, 2025 at 1:35:47 PM UTC-6 [email protected] wrote: > TIL that `mix test --only async --exclude external_api` will say that the > external_api tag is skipped, but go ahead and run those cases anyway. > > *Example code:* > ``` > defmodule OnlyExceptTest do use ExUnit.Case, async: true test "normal > async test" do assert true end @tag :skip test "this test should be > excluded when using --exclude skip_me" do flunk("This test should have been > excluded!") end end > ``` > # run `mix test --only async --except skip` > I would expect only the passing test to run, but in fact, both run, > causing an overall failure of the task. Jon R explained that this is due to > the way --only uses --include under the hood, which takes precedence over > --exclude, meaning additional exclude flags have no effect. The explanation > makes sense, as does the workaround of inverting the logic to pass an > unified --exclude async:false --exclude test --exclude skip. > > However, it goes against the principle of least surprise, and creates a > subtle footgun, even for experienced developers. > > *Real-World Use Case* > In CI, I have separate jobs for sync and async tests. A coworker added > tests for an external API and put an exclude in our `test_helper.exs`. > However, since `--exclude` is superseded by `--include`, and I was using > `--only async:true` in my CI job, the API tests ran and caused CI failures. > > It took a message to the Elixir Slack, and a helpful reply to unblock me. > > I thought we might be able to prevent the confusion by preventing > `exclude` from being combined with `only`, but I'm not sure how that would > work with `test_helper` usage. > > However, it would still be really nice to have proper Ecto-style set > operations on mix test. > > *Slack thread*: > https://elixir-lang.slack.com/archives/C03EPRA3B/p1765798909362919 > *Illustrative Draft PR*: https://github.com/elixir-lang/elixir/pull/15014 -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/elixir-lang-core/a71befef-cfb2-4cee-aa96-58ba2e56339an%40googlegroups.com.
