+1 for --next-failure functionality. My current approach with ExUnit is basically a manual version of that.
Allen Madsen http://www.allenmadsen.com On Thu, Nov 23, 2017 at 12:28 PM, Myron Marston <[email protected]> wrote: > I believe this would be a good addition. My only question is where are the > failed tests stored? In _build? > > For RSpec we made users configure where this state is stored, via a > config.example_status_persistence_file_path option. RSpec didn’t have an > established place to write that state so we left it up to the user to > decide where they wanted it to go. I think for ExUnit, storing it in > _build make sense. > > However, note that we are not merely storing a list of failed tests. We > store a list of *all* tests (including ones that were not included in the > latest run) that looks like this: > > example_id | > status | run_time | > ---------------------------------------------------------------------- | > ------- | --------------- | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:1] | > passed | 0.00115 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:2] | > passed | 0.00052 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:3] | > unknown | | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:4] | > passed | 0.00048 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:1:1] | > passed | 0.00058 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:2:1] | > failed | 0.00088 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:3:1] | > passed | 0.00084 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:1] | > passed | 0.00052 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:2] | > failed | 0.00059 seconds | > ./spec/rspec/core/backtrace_formatter_spec.rb[1:4:1] | > pending | 0.00053 seconds | > ./spec/rspec/core/bisect/coordinator_spec.rb[1:1] | > passed | 0.00366 seconds | > ./spec/rspec/core/bisect/coordinator_spec.rb[1:2] | > passed | 0.00307 seconds | > ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:1] | > passed | 0.002 seconds | > ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:2] | > passed | 0.00231 seconds | > ./spec/rspec/core/bisect/coordinator_spec.rb[1:4:1] | > passed | 0.00293 seconds | > ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:1] | > passed | 0.00049 seconds | > ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:2] | > passed | 0.0006 seconds | > > # ... > > This is a custom serialization format we designed to be easily scannable > by a human (as it’s useful information, particular the run_time). The > example_id column uniquely identifies each test (since the other common > ways to identify tests, such as description and file location, are not > guaranteed to be unique). Every time a test run finishes, we merge the > results with the existing contents of this file using a few simple rules > <https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/example_status_persister.rb#L66-L72> > . > > We then use this data to automatically add :last_run_status metadata to > every test (with values of passed, failed, pending or unknown) when the > spec files are loaded, which unlocks the generic ability to filter based on > this via the RSpec CLI: > > $ rspec --tag last_run_status:failed > > This is the equivalent of --only failed like you asked about, José. > Whether or not you add an explicit option like --only-failures is up to > you, but the explicit option does provide a couple nice advantages for > RSpec: > > - It surfaces this extremely useful option in the --help output. > Without calling it out, it would not be clear to most users that failure > filtering is possible. > - Since we can easily tell from our persistence file which spec files > have failures, when --only-failures is passed, we automatically load > only those files. In contrast, --tag filtering doesn’t generally know > anything in advance about which files might have specs matching the tag, > so --tag > last_run_status:failed will load *all* spec files, and then apply the > filtering. This can be significantly slower, particularly if there are > files without failures that load a heavyweight dependency (like rails). > > One other option we provide (which ExUnit may or may not want to provide) > is --next-failure. This is the equivalent of --only-failures --fail-fast > --order defined. The idea is that you often want to work through the > failures systematically one-by-one. --fail-fast causes RSpec to abort as > soon as the first failure is hit and --order defined disables the random > ordering so you get the same failed example when you run rspec > --next-failure over and over again to help you focus on a specific one. > This option is also why we do the merging operation with the status from > prior runs: it’s important that we preserve the failed status of tests > that weren’t executed in the latest run. > > ExUnit certainly doesn’t have to go the same route RSpec went here, but > the combination of the perf speed up from avoiding loading files with only > passing tests and the usefulness of --next-failure is pretty awesome, IMO. > > Myron > > > On Thu, Nov 23, 2017 at 4:03 AM, José Valim <[email protected]> wrote: > >> Thanks everyone! >> >> I believe this would be a good addition. My only question is where are >> the failed tests stored? In _build? Also, maybe we can also implement it as >> a special tag called "--only failed" or "--only failures"? >> >> >> >> >> *José Valimwww.plataformatec.com.br >> <http://www.plataformatec.com.br/>Founder and Director of R&D* >> >> On Thu, Nov 23, 2017 at 6:03 AM, Myron Marston <[email protected]> >> wrote: >> >>> I too would love to see ExUnit support an `--only-failures` flag. It's >>> one of my favorite features of RSpec and I wish every test framework had >>> it. I find that it makes a huge difference to my workflow to be able to >>> quickly and easily filter to the tests that failed the last time they ran. >>> >>> In fact, I love this feature of RSpec so much that I was the one who >>> added it to the framework a couple years back :). I'd be happy to help see >>> it get added to ExUnit if José and others were amenable. ExUnit already >>> has most of the building blocks needed for it via tags and filtering. >>> >>> Myron >>> >>> On Wednesday, November 22, 2017 at 2:48:14 PM UTC-8, José Valim wrote: >>>> >>>> To clarify, --stale does not run previously failed tests. >>>> >>>> > I just changed the format of the message built within >>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests throughout >>>> my app, as expected. But since the tests didn't directly reference >>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when the >>>> message format changed; I had to run all tests to get them to fail. >>>> >>>> That feels like a bug. Maybe we are being conservative on how we >>>> compute the dependencies. If you can provide a sample app that reproduces >>>> the error, I would love to take a look at it. >>>> >>>> >>>> >>>> *José Valimwww.plataformatec.com.br >>>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>>> >>>> On Wed, Nov 22, 2017 at 8:06 PM, Nathan Long <[email protected]> >>>> wrote: >>>> >>>>> Sure. I have a module called `MyApp.Mixpanel` with functions like >>>>> `track_event(:user_signup, data_map)`. These are called from various >>>>> places >>>>> throughout the codebase. There's a production adapter, which actually >>>>> sends >>>>> the event data to Mixpanel for analytics purposes, a dev adapter, which >>>>> just logs it, and a test adapter, which sends it to `self()` as a message. >>>>> >>>>> Several of my tests say things like "if I POST the info required for a >>>>> new user signup, I should get a message showing that the correct info >>>>> would >>>>> have been sent to Mixpanel." These use `assert_receive`. >>>>> >>>>> I just changed the format of the message built within >>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests throughout >>>>> my app, as expected. But since the tests didn't directly reference >>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when the >>>>> message format changed; I had to run all tests to get them to fail. >>>>> >>>>> This is no big deal, but it would be nice in such situations to run >>>>> all tests once, then be able to whittle down the failing tests without >>>>> re-running the whole suite. >>>>> >>>>> On Wednesday, November 22, 2017 at 4:54:51 PM UTC-5, Louis Pilfold >>>>> wrote: >>>>>> >>>>>> Hi Nathan >>>>>> >>>>>> I feel ExUnit --stale should always be able to tell this. Could you >>>>>> share your example please? >>>>>> >>>>>> Cheers, >>>>>> Louis >>>>>> >>>>>> On Wed, 22 Nov 2017 at 20:43 Nathan Long <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Ruby's Rspec has a handy option, `--only-failures`, which "filters >>>>>>> what examples are run so that only those that failed the last time they >>>>>>> ran >>>>>>> are executed". https://relishapp.com/rspec/rs >>>>>>> pec-core/docs/command-line/only-failures >>>>>>> >>>>>>> I'd love to have this feature in ExUnit. The closest thing I see >>>>>>> right now is `--stale`, but if ExUnit can't accurately determine which >>>>>>> tests may have been broken by a change, it doesn't work. (I have such an >>>>>>> example, but don't want to be long-winded; maybe the utility of this >>>>>>> feature is clear enough?) >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elixir-lang-core" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3- >>>>>>> ed51-44be-8f6b-81e5181fa449%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3-ed51-44be-8f6b-81e5181fa449%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elixir-lang-core" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6- >>>>> f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6-f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40 >>> googlegroups.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elixir-lang-core" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJ >> sP8ggzTngnGuxw%40mail.gmail.com >> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT- > W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej%3Db4vQEt6r0Cw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
