The reason I proposed a new flag, rather than "fixing" the way current flags work was to side-step a breaking change.
I would expect a limited set-based query API to be easy to document, since most developers will be familiar with AND/OR/NOT/bracket semantics from SQL/Ecto and general boolean logic. The proof-of-concept implementation is 600 LOC, 200 of which is tests, and another 150+ documentation, so not exactly big or unmaintainable. There is an extension where things might get more complex, and that is supporting arbitrary tag values e.g. --where (integration:UI or integration:API), but even that is manageable, as long as we don't get crazy and introduce value-level negation and other OTT stuff. Keep it simple, keep it familiar, avoid footguns. On Tuesday, 16 December 2025 at 00:52:58 UTC [email protected] wrote: > I like the idea of rethinking how these flags interact, though I do not > like the proposed api. A string-based pseudo-query/natural language DSL > feels overly complicated: hard to document, hard to maintain, not worth the > potential benefits. > > My intuition would be that we should be able to honor both includes and > excludes in a sane way, but I'm unfamiliar with the trade-offs that led to > the current design. However, note that any changes to this behaviour could > be a breaking change to many test suites, so have to be released cautiously > behind the right version number. > > On Monday, December 15, 2025 at 1:35:47 PM UTC-6 [email protected] > wrote: > >> TIL that `mix test --only async --exclude external_api` will say that the >> external_api tag is skipped, but go ahead and run those cases anyway. >> >> *Example code:* >> ``` >> defmodule OnlyExceptTest do use ExUnit.Case, async: true test "normal >> async test" do assert true end @tag :skip test "this test should be >> excluded when using --exclude skip_me" do flunk("This test should have been >> excluded!") end end >> ``` >> # run `mix test --only async --except skip` >> I would expect only the passing test to run, but in fact, both run, >> causing an overall failure of the task. Jon R explained that this is due to >> the way --only uses --include under the hood, which takes precedence over >> --exclude, meaning additional exclude flags have no effect. The explanation >> makes sense, as does the workaround of inverting the logic to pass an >> unified --exclude async:false --exclude test --exclude skip. >> >> However, it goes against the principle of least surprise, and creates a >> subtle footgun, even for experienced developers. >> >> *Real-World Use Case* >> In CI, I have separate jobs for sync and async tests. A coworker added >> tests for an external API and put an exclude in our `test_helper.exs`. >> However, since `--exclude` is superseded by `--include`, and I was using >> `--only async:true` in my CI job, the API tests ran and caused CI failures. >> >> It took a message to the Elixir Slack, and a helpful reply to unblock me. >> >> I thought we might be able to prevent the confusion by preventing >> `exclude` from being combined with `only`, but I'm not sure how that would >> work with `test_helper` usage. >> >> However, it would still be really nice to have proper Ecto-style set >> operations on mix test. >> >> *Slack thread*: >> https://elixir-lang.slack.com/archives/C03EPRA3B/p1765798909362919 >> *Illustrative Draft PR*: https://github.com/elixir-lang/elixir/pull/15014 > > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/elixir-lang-core/8c97c9de-5443-44ad-b3c1-833c707acc11n%40googlegroups.com.
