That's very helpful, thank you Myron. We already keep several manifests for compiled code with the function calls, files and modules. Therefore it should be relatively straight-forward to keep one for tests. I think the first step is to build the manifest itself which will give us the last_run_status information. Is that right?
Implementation-wise, we can probably even use a custom "formatter" to maintain this information. All we need is a path to store this manifest (which is opt-in but mix test can generate one by default in _build and pass to ExUnit). *José Valimwww.plataformatec.com.br <http://www.plataformatec.com.br/>Founder and Director of R&D* On Thu, Nov 23, 2017 at 4:27 PM, Allen Madsen <[email protected]> wrote: > +1 for --next-failure functionality. My current approach with ExUnit is > basically a manual version of that. > > Allen Madsen > http://www.allenmadsen.com > > On Thu, Nov 23, 2017 at 12:28 PM, Myron Marston <[email protected]> > wrote: > >> I believe this would be a good addition. My only question is where are >> the failed tests stored? In _build? >> >> For RSpec we made users configure where this state is stored, via a >> config.example_status_persistence_file_path option. RSpec didn’t have an >> established place to write that state so we left it up to the user to >> decide where they wanted it to go. I think for ExUnit, storing it in >> _build make sense. >> >> However, note that we are not merely storing a list of failed tests. We >> store a list of *all* tests (including ones that were not included in >> the latest run) that looks like this: >> >> example_id | >> status | run_time | >> ---------------------------------------------------------------------- | >> ------- | --------------- | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:1] | >> passed | 0.00115 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:2] | >> passed | 0.00052 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:3] | >> unknown | | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:4] | >> passed | 0.00048 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:1:1] | >> passed | 0.00058 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:2:1] | >> failed | 0.00088 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:3:1] | >> passed | 0.00084 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:1] | >> passed | 0.00052 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:2] | >> failed | 0.00059 seconds | >> ./spec/rspec/core/backtrace_formatter_spec.rb[1:4:1] | >> pending | 0.00053 seconds | >> ./spec/rspec/core/bisect/coordinator_spec.rb[1:1] | >> passed | 0.00366 seconds | >> ./spec/rspec/core/bisect/coordinator_spec.rb[1:2] | >> passed | 0.00307 seconds | >> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:1] | >> passed | 0.002 seconds | >> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:2] | >> passed | 0.00231 seconds | >> ./spec/rspec/core/bisect/coordinator_spec.rb[1:4:1] | >> passed | 0.00293 seconds | >> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:1] | >> passed | 0.00049 seconds | >> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:2] | >> passed | 0.0006 seconds | >> >> # ... >> >> This is a custom serialization format we designed to be easily scannable >> by a human (as it’s useful information, particular the run_time). The >> example_id column uniquely identifies each test (since the other common >> ways to identify tests, such as description and file location, are not >> guaranteed to be unique). Every time a test run finishes, we merge the >> results with the existing contents of this file using a few simple rules >> <https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/example_status_persister.rb#L66-L72> >> . >> >> We then use this data to automatically add :last_run_status metadata to >> every test (with values of passed, failed, pending or unknown) when the >> spec files are loaded, which unlocks the generic ability to filter based on >> this via the RSpec CLI: >> >> $ rspec --tag last_run_status:failed >> >> This is the equivalent of --only failed like you asked about, José. >> Whether or not you add an explicit option like --only-failures is up to >> you, but the explicit option does provide a couple nice advantages for >> RSpec: >> >> - It surfaces this extremely useful option in the --help output. >> Without calling it out, it would not be clear to most users that failure >> filtering is possible. >> - Since we can easily tell from our persistence file which spec files >> have failures, when --only-failures is passed, we automatically load >> only those files. In contrast, --tag filtering doesn’t generally know >> anything in advance about which files might have specs matching the tag, >> so --tag >> last_run_status:failed will load *all* spec files, and then apply the >> filtering. This can be significantly slower, particularly if there are >> files without failures that load a heavyweight dependency (like rails). >> >> One other option we provide (which ExUnit may or may not want to provide) >> is --next-failure. This is the equivalent of --only-failures --fail-fast >> --order defined. The idea is that you often want to work through the >> failures systematically one-by-one. --fail-fast causes RSpec to abort as >> soon as the first failure is hit and --order defined disables the random >> ordering so you get the same failed example when you run rspec >> --next-failure over and over again to help you focus on a specific one. >> This option is also why we do the merging operation with the status from >> prior runs: it’s important that we preserve the failed status of tests >> that weren’t executed in the latest run. >> >> ExUnit certainly doesn’t have to go the same route RSpec went here, but >> the combination of the perf speed up from avoiding loading files with only >> passing tests and the usefulness of --next-failure is pretty awesome, >> IMO. >> >> Myron >> >> >> On Thu, Nov 23, 2017 at 4:03 AM, José Valim <[email protected]> wrote: >> >>> Thanks everyone! >>> >>> I believe this would be a good addition. My only question is where are >>> the failed tests stored? In _build? Also, maybe we can also implement it as >>> a special tag called "--only failed" or "--only failures"? >>> >>> >>> >>> >>> *José Valimwww.plataformatec.com.br >>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>> >>> On Thu, Nov 23, 2017 at 6:03 AM, Myron Marston <[email protected]> >>> wrote: >>> >>>> I too would love to see ExUnit support an `--only-failures` flag. It's >>>> one of my favorite features of RSpec and I wish every test framework had >>>> it. I find that it makes a huge difference to my workflow to be able to >>>> quickly and easily filter to the tests that failed the last time they ran. >>>> >>>> In fact, I love this feature of RSpec so much that I was the one who >>>> added it to the framework a couple years back :). I'd be happy to help see >>>> it get added to ExUnit if José and others were amenable. ExUnit already >>>> has most of the building blocks needed for it via tags and filtering. >>>> >>>> Myron >>>> >>>> On Wednesday, November 22, 2017 at 2:48:14 PM UTC-8, José Valim wrote: >>>>> >>>>> To clarify, --stale does not run previously failed tests. >>>>> >>>>> > I just changed the format of the message built within >>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests throughout >>>>> my app, as expected. But since the tests didn't directly reference >>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when the >>>>> message format changed; I had to run all tests to get them to fail. >>>>> >>>>> That feels like a bug. Maybe we are being conservative on how we >>>>> compute the dependencies. If you can provide a sample app that reproduces >>>>> the error, I would love to take a look at it. >>>>> >>>>> >>>>> >>>>> *José Valimwww.plataformatec.com.br >>>>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>>>> >>>>> On Wed, Nov 22, 2017 at 8:06 PM, Nathan Long <[email protected]> >>>>> wrote: >>>>> >>>>>> Sure. I have a module called `MyApp.Mixpanel` with functions like >>>>>> `track_event(:user_signup, data_map)`. These are called from various >>>>>> places >>>>>> throughout the codebase. There's a production adapter, which actually >>>>>> sends >>>>>> the event data to Mixpanel for analytics purposes, a dev adapter, which >>>>>> just logs it, and a test adapter, which sends it to `self()` as a >>>>>> message. >>>>>> >>>>>> Several of my tests say things like "if I POST the info required for >>>>>> a new user signup, I should get a message showing that the correct info >>>>>> would have been sent to Mixpanel." These use `assert_receive`. >>>>>> >>>>>> I just changed the format of the message built within >>>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests >>>>>> throughout >>>>>> my app, as expected. But since the tests didn't directly reference >>>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when the >>>>>> message format changed; I had to run all tests to get them to fail. >>>>>> >>>>>> This is no big deal, but it would be nice in such situations to run >>>>>> all tests once, then be able to whittle down the failing tests without >>>>>> re-running the whole suite. >>>>>> >>>>>> On Wednesday, November 22, 2017 at 4:54:51 PM UTC-5, Louis Pilfold >>>>>> wrote: >>>>>>> >>>>>>> Hi Nathan >>>>>>> >>>>>>> I feel ExUnit --stale should always be able to tell this. Could you >>>>>>> share your example please? >>>>>>> >>>>>>> Cheers, >>>>>>> Louis >>>>>>> >>>>>>> On Wed, 22 Nov 2017 at 20:43 Nathan Long <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Ruby's Rspec has a handy option, `--only-failures`, which "filters >>>>>>>> what examples are run so that only those that failed the last time >>>>>>>> they ran >>>>>>>> are executed". https://relishapp.com/rspec/rs >>>>>>>> pec-core/docs/command-line/only-failures >>>>>>>> >>>>>>>> I'd love to have this feature in ExUnit. The closest thing I see >>>>>>>> right now is `--stale`, but if ExUnit can't accurately determine which >>>>>>>> tests may have been broken by a change, it doesn't work. (I have such >>>>>>>> an >>>>>>>> example, but don't want to be long-winded; maybe the utility of this >>>>>>>> feature is clear enough?) >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elixir-lang-core" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3- >>>>>>>> ed51-44be-8f6b-81e5181fa449%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3-ed51-44be-8f6b-81e5181fa449%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elixir-lang-core" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6- >>>>>> f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6-f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40 >>>> googlegroups.com >>>> <https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elixir-lang-core" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJ >>> sP8ggzTngnGuxw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz% >> 2BRGUx%2B7ACXs9nfQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLf > hQsej%3Db4vQEt6r0Cw%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej%3Db4vQEt6r0Cw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LE9NLxeSxkceQuw%2BHAGEtZ3gY6jUJ3WrLAw%3D9dREJY-Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
