I believe this would be a good addition. My only question is where are the failed tests stored? In _build?
For RSpec we made users configure where this state is stored, via a config.example_status_persistence_file_path option. RSpec didn’t have an established place to write that state so we left it up to the user to decide where they wanted it to go. I think for ExUnit, storing it in _build make sense. However, note that we are not merely storing a list of failed tests. We store a list of *all* tests (including ones that were not included in the latest run) that looks like this: example_id | status | run_time | ---------------------------------------------------------------------- | ------- | --------------- | ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:1] | passed | 0.00115 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:2] | passed | 0.00052 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:3] | unknown | | ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:4] | passed | 0.00048 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:1:1] | passed | 0.00058 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:2:1] | failed | 0.00088 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:3:1] | passed | 0.00084 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:1] | passed | 0.00052 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:2] | failed | 0.00059 seconds | ./spec/rspec/core/backtrace_formatter_spec.rb[1:4:1] | pending | 0.00053 seconds | ./spec/rspec/core/bisect/coordinator_spec.rb[1:1] | passed | 0.00366 seconds | ./spec/rspec/core/bisect/coordinator_spec.rb[1:2] | passed | 0.00307 seconds | ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:1] | passed | 0.002 seconds | ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:2] | passed | 0.00231 seconds | ./spec/rspec/core/bisect/coordinator_spec.rb[1:4:1] | passed | 0.00293 seconds | ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:1] | passed | 0.00049 seconds | ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:2] | passed | 0.0006 seconds | # ... This is a custom serialization format we designed to be easily scannable by a human (as it’s useful information, particular the run_time). The example_id column uniquely identifies each test (since the other common ways to identify tests, such as description and file location, are not guaranteed to be unique). Every time a test run finishes, we merge the results with the existing contents of this file using a few simple rules <https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/example_status_persister.rb#L66-L72> . We then use this data to automatically add :last_run_status metadata to every test (with values of passed, failed, pending or unknown) when the spec files are loaded, which unlocks the generic ability to filter based on this via the RSpec CLI: $ rspec --tag last_run_status:failed This is the equivalent of --only failed like you asked about, José. Whether or not you add an explicit option like --only-failures is up to you, but the explicit option does provide a couple nice advantages for RSpec: - It surfaces this extremely useful option in the --help output. Without calling it out, it would not be clear to most users that failure filtering is possible. - Since we can easily tell from our persistence file which spec files have failures, when --only-failures is passed, we automatically load only those files. In contrast, --tag filtering doesn’t generally know anything in advance about which files might have specs matching the tag, so --tag last_run_status:failed will load *all* spec files, and then apply the filtering. This can be significantly slower, particularly if there are files without failures that load a heavyweight dependency (like rails). One other option we provide (which ExUnit may or may not want to provide) is --next-failure. This is the equivalent of --only-failures --fail-fast --order defined. The idea is that you often want to work through the failures systematically one-by-one. --fail-fast causes RSpec to abort as soon as the first failure is hit and --order defined disables the random ordering so you get the same failed example when you run rspec --next-failure over and over again to help you focus on a specific one. This option is also why we do the merging operation with the status from prior runs: it’s important that we preserve the failed status of tests that weren’t executed in the latest run. ExUnit certainly doesn’t have to go the same route RSpec went here, but the combination of the perf speed up from avoiding loading files with only passing tests and the usefulness of --next-failure is pretty awesome, IMO. Myron On Thu, Nov 23, 2017 at 4:03 AM, José Valim <[email protected]> wrote: > Thanks everyone! > > I believe this would be a good addition. My only question is where are the > failed tests stored? In _build? Also, maybe we can also implement it as a > special tag called "--only failed" or "--only failures"? > > > > > *José Valimwww.plataformatec.com.br > <http://www.plataformatec.com.br/>Founder and Director of R&D* > > On Thu, Nov 23, 2017 at 6:03 AM, Myron Marston <[email protected]> > wrote: > >> I too would love to see ExUnit support an `--only-failures` flag. It's >> one of my favorite features of RSpec and I wish every test framework had >> it. I find that it makes a huge difference to my workflow to be able to >> quickly and easily filter to the tests that failed the last time they ran. >> >> In fact, I love this feature of RSpec so much that I was the one who >> added it to the framework a couple years back :). I'd be happy to help see >> it get added to ExUnit if José and others were amenable. ExUnit already >> has most of the building blocks needed for it via tags and filtering. >> >> Myron >> >> On Wednesday, November 22, 2017 at 2:48:14 PM UTC-8, José Valim wrote: >>> >>> To clarify, --stale does not run previously failed tests. >>> >>> > I just changed the format of the message built within >>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests throughout >>> my app, as expected. But since the tests didn't directly reference >>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when the >>> message format changed; I had to run all tests to get them to fail. >>> >>> That feels like a bug. Maybe we are being conservative on how we compute >>> the dependencies. If you can provide a sample app that reproduces the >>> error, I would love to take a look at it. >>> >>> >>> >>> *José Valimwww.plataformatec.com.br >>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>> >>> On Wed, Nov 22, 2017 at 8:06 PM, Nathan Long <[email protected]> >>> wrote: >>> >>>> Sure. I have a module called `MyApp.Mixpanel` with functions like >>>> `track_event(:user_signup, data_map)`. These are called from various places >>>> throughout the codebase. There's a production adapter, which actually sends >>>> the event data to Mixpanel for analytics purposes, a dev adapter, which >>>> just logs it, and a test adapter, which sends it to `self()` as a message. >>>> >>>> Several of my tests say things like "if I POST the info required for a >>>> new user signup, I should get a message showing that the correct info would >>>> have been sent to Mixpanel." These use `assert_receive`. >>>> >>>> I just changed the format of the message built within `MyApp.Mixpanel`. >>>> This caused `assert_receive` to fail in tests throughout my app, as >>>> expected. But since the tests didn't directly reference `MyApp.Mixpanel`, >>>> `--stale` didn't know which ones should be run when the message format >>>> changed; I had to run all tests to get them to fail. >>>> >>>> This is no big deal, but it would be nice in such situations to run all >>>> tests once, then be able to whittle down the failing tests without >>>> re-running the whole suite. >>>> >>>> On Wednesday, November 22, 2017 at 4:54:51 PM UTC-5, Louis Pilfold >>>> wrote: >>>>> >>>>> Hi Nathan >>>>> >>>>> I feel ExUnit --stale should always be able to tell this. Could you >>>>> share your example please? >>>>> >>>>> Cheers, >>>>> Louis >>>>> >>>>> On Wed, 22 Nov 2017 at 20:43 Nathan Long <[email protected]> wrote: >>>>> >>>>>> Ruby's Rspec has a handy option, `--only-failures`, which "filters >>>>>> what examples are run so that only those that failed the last time they >>>>>> ran >>>>>> are executed". https://relishapp.com/rspec/rs >>>>>> pec-core/docs/command-line/only-failures >>>>>> >>>>>> I'd love to have this feature in ExUnit. The closest thing I see >>>>>> right now is `--stale`, but if ExUnit can't accurately determine which >>>>>> tests may have been broken by a change, it doesn't work. (I have such an >>>>>> example, but don't want to be long-winded; maybe the utility of this >>>>>> feature is clear enough?) >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elixir-lang-core" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3- >>>>>> ed51-44be-8f6b-81e5181fa449%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3-ed51-44be-8f6b-81e5181fa449%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/elixir-lang-core/2aa483e6-f63c-42d6-9e4b-84efb8adf9de%40 >>>> googlegroups.com >>>> <https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6-f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9% >> 40googlegroups.com >> <https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elixir-lang-core" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio% > 3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
