Tests are uniquely identified by module+name. It is not quite powerful as an ID system but it does the job of identifying tests uniquely.
*José Valimwww.plataformatec.com.br <http://www.plataformatec.com.br/>Founder and Director of R&D* On Thu, Nov 23, 2017 at 7:14 PM, Myron Marston <[email protected]> wrote: > I think the first step is to build the manifest itself which will give us > the last_run_status information. Is that right? > > I think there’s a pre-requisite you need to get out of the way before you > can build the manifest: you need to decide how you plan to uniquely > identify each test. Does ExUnit already have something analogous to RSpec’s > example ids? If not, you could potentially use either the test name or the > test location (e.g. file_name:line_number) but those may not be > sufficient (for RSpec they weren’t). For RSpec, the file location is not > guaranteed unique, since you can dynamically define multiple tests in a > loop, which results in multiple tests sharing the same file location, and > this seems like a problem for ExUnit. Likewise, RSpec does not require that > each test description is unique (I think ExUnit might require this, > though…is that right?). Even if test descriptions are unique, it has some > properties that, IMO, make it undesirable for use here: > > - There’s no easy way to map a test description back to the file the > test is defined in, which means it limits the kind of cleanup you can do as > part of merging the current results and the old results. At the end of a > test run, RSpec cleans up the manifest by removing tests that cannot > possibly still exist due to their file no longer existing, which is only > possible since the example ids list what file the tests come from. > - Test descriptions often change when the contents of the test may > not. (Likewise, the location of a test can easily change just by the > introduction of a helper function, an import or alias at the top of > the module, etc). > > It’s easiest to explain how RSpec’s example ids work by showing an example: > > # foo_spec.rb > RSpec.describe "Group 1" do > it 'foos' do # foo_spec.rb[1:1] > # ... > end > > describe "a nested group" do > it 'bars' do # foo_spec.rb[1:2:1] > # ... > end > > it 'bars again' do # foo_spec.rb[1:2:2] > # ... > end > end > > it 'foos again' do # foo_spec.rb[1:3] > > endend > RSpec.describe "Group 2" do > it 'foos' do # foo_spec.rb[2:1] > # ... > endend > > Basically, we number each example and example group with a counter that > starts over at 1 within each new scope, and use colons to separate the > elements that form the “path” to the specific example. A nice thing about > the ids is that they are relatively stable even in the sense of further > development of the file. Users can change their test descriptions and > introduce new things that change the line numbers, and the ids still work > to correctly identify the tests. > > Would it make sense to introduce something like this for ExUnit? In RSpec > we have found these ids to be useful for several other things (including > --bisect, deterministic ordering when applying a seed to a subset, etc). > > BTW, this is something I’d be happy to take a stab at in ExUnit unless > someone else wanted to do it. > > Myron > > > > On Thu, Nov 23, 2017 at 10:45 AM, José Valim <[email protected]> wrote: > >> That's very helpful, thank you Myron. >> >> We already keep several manifests for compiled code with the function >> calls, files and modules. Therefore it should be relatively >> straight-forward to keep one for tests. I think the first step is to build >> the manifest itself which will give us the last_run_status information. Is >> that right? >> >> Implementation-wise, we can probably even use a custom "formatter" to >> maintain this information. All we need is a path to store this manifest >> (which is opt-in but mix test can generate one by default in _build and >> pass to ExUnit). >> >> >> >> >> *José Valimwww.plataformatec.com.br >> <http://www.plataformatec.com.br/>Founder and Director of R&D* >> >> On Thu, Nov 23, 2017 at 4:27 PM, Allen Madsen <[email protected]> >> wrote: >> >>> +1 for --next-failure functionality. My current approach with ExUnit is >>> basically a manual version of that. >>> >>> Allen Madsen >>> http://www.allenmadsen.com >>> >>> On Thu, Nov 23, 2017 at 12:28 PM, Myron Marston <[email protected] >>> > wrote: >>> >>>> I believe this would be a good addition. My only question is where are >>>> the failed tests stored? In _build? >>>> >>>> For RSpec we made users configure where this state is stored, via a >>>> config.example_status_persistence_file_path option. RSpec didn’t have >>>> an established place to write that state so we left it up to the user to >>>> decide where they wanted it to go. I think for ExUnit, storing it in >>>> _build make sense. >>>> >>>> However, note that we are not merely storing a list of failed tests. We >>>> store a list of *all* tests (including ones that were not included in >>>> the latest run) that looks like this: >>>> >>>> example_id | >>>> status | run_time | >>>> ---------------------------------------------------------------------- | >>>> ------- | --------------- | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:1] | >>>> passed | 0.00115 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:2] | >>>> passed | 0.00052 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:3] | >>>> unknown | | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:4] | >>>> passed | 0.00048 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:1:1] | >>>> passed | 0.00058 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:2:1] | >>>> failed | 0.00088 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:3:1] | >>>> passed | 0.00084 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:1] | >>>> passed | 0.00052 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:2] | >>>> failed | 0.00059 seconds | >>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:4:1] | >>>> pending | 0.00053 seconds | >>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:1] | >>>> passed | 0.00366 seconds | >>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:2] | >>>> passed | 0.00307 seconds | >>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:1] | >>>> passed | 0.002 seconds | >>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:2] | >>>> passed | 0.00231 seconds | >>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:4:1] | >>>> passed | 0.00293 seconds | >>>> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:1] | >>>> passed | 0.00049 seconds | >>>> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:2] | >>>> passed | 0.0006 seconds | >>>> >>>> # ... >>>> >>>> This is a custom serialization format we designed to be easily >>>> scannable by a human (as it’s useful information, particular the >>>> run_time). The example_id column uniquely identifies each test (since >>>> the other common ways to identify tests, such as description and file >>>> location, are not guaranteed to be unique). Every time a test run finishes, >>>> we merge the results with the existing contents of this file using a >>>> few simple rules >>>> <https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/example_status_persister.rb#L66-L72> >>>> . >>>> >>>> We then use this data to automatically add :last_run_status metadata >>>> to every test (with values of passed, failed, pending or unknown) when >>>> the spec files are loaded, which unlocks the generic ability to filter >>>> based on this via the RSpec CLI: >>>> >>>> $ rspec --tag last_run_status:failed >>>> >>>> This is the equivalent of --only failed like you asked about, José. >>>> Whether or not you add an explicit option like --only-failures is up >>>> to you, but the explicit option does provide a couple nice advantages for >>>> RSpec: >>>> >>>> - It surfaces this extremely useful option in the --help output. >>>> Without calling it out, it would not be clear to most users that failure >>>> filtering is possible. >>>> - Since we can easily tell from our persistence file which spec >>>> files have failures, when --only-failures is passed, we >>>> automatically load only those files. In contrast, --tag filtering >>>> doesn’t generally know anything in advance about which files might have >>>> specs matching the tag, so --tag last_run_status:failed will load >>>> *all* spec files, and then apply the filtering. This can be >>>> significantly slower, particularly if there are files without failures >>>> that >>>> load a heavyweight dependency (like rails). >>>> >>>> One other option we provide (which ExUnit may or may not want to >>>> provide) is --next-failure. This is the equivalent of --only-failures >>>> --fail-fast --order defined. The idea is that you often want to work >>>> through the failures systematically one-by-one. --fail-fast causes >>>> RSpec to abort as soon as the first failure is hit and --order defined >>>> disables the random ordering so you get the same failed example when you >>>> run rspec --next-failure over and over again to help you focus on a >>>> specific one. This option is also why we do the merging operation with the >>>> status from prior runs: it’s important that we preserve the failed >>>> status of tests that weren’t executed in the latest run. >>>> >>>> ExUnit certainly doesn’t have to go the same route RSpec went here, but >>>> the combination of the perf speed up from avoiding loading files with only >>>> passing tests and the usefulness of --next-failure is pretty awesome, >>>> IMO. >>>> >>>> Myron >>>> >>>> >>>> On Thu, Nov 23, 2017 at 4:03 AM, José Valim <[email protected]> >>>> wrote: >>>> >>>>> Thanks everyone! >>>>> >>>>> I believe this would be a good addition. My only question is where are >>>>> the failed tests stored? In _build? Also, maybe we can also implement it >>>>> as >>>>> a special tag called "--only failed" or "--only failures"? >>>>> >>>>> >>>>> >>>>> >>>>> *José Valimwww.plataformatec.com.br >>>>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>>>> >>>>> On Thu, Nov 23, 2017 at 6:03 AM, Myron Marston < >>>>> [email protected]> wrote: >>>>> >>>>>> I too would love to see ExUnit support an `--only-failures` flag. >>>>>> It's one of my favorite features of RSpec and I wish every test framework >>>>>> had it. I find that it makes a huge difference to my workflow to be able >>>>>> to quickly and easily filter to the tests that failed the last time they >>>>>> ran. >>>>>> >>>>>> In fact, I love this feature of RSpec so much that I was the one who >>>>>> added it to the framework a couple years back :). I'd be happy to help >>>>>> see >>>>>> it get added to ExUnit if José and others were amenable. ExUnit already >>>>>> has most of the building blocks needed for it via tags and filtering. >>>>>> >>>>>> Myron >>>>>> >>>>>> On Wednesday, November 22, 2017 at 2:48:14 PM UTC-8, José Valim wrote: >>>>>>> >>>>>>> To clarify, --stale does not run previously failed tests. >>>>>>> >>>>>>> > I just changed the format of the message built within >>>>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests >>>>>>> throughout >>>>>>> my app, as expected. But since the tests didn't directly reference >>>>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when >>>>>>> the >>>>>>> message format changed; I had to run all tests to get them to fail. >>>>>>> >>>>>>> That feels like a bug. Maybe we are being conservative on how we >>>>>>> compute the dependencies. If you can provide a sample app that >>>>>>> reproduces >>>>>>> the error, I would love to take a look at it. >>>>>>> >>>>>>> >>>>>>> >>>>>>> *José Valimwww.plataformatec.com.br >>>>>>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>>>>>> >>>>>>> On Wed, Nov 22, 2017 at 8:06 PM, Nathan Long <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Sure. I have a module called `MyApp.Mixpanel` with functions like >>>>>>>> `track_event(:user_signup, data_map)`. These are called from various >>>>>>>> places >>>>>>>> throughout the codebase. There's a production adapter, which actually >>>>>>>> sends >>>>>>>> the event data to Mixpanel for analytics purposes, a dev adapter, which >>>>>>>> just logs it, and a test adapter, which sends it to `self()` as a >>>>>>>> message. >>>>>>>> >>>>>>>> Several of my tests say things like "if I POST the info required >>>>>>>> for a new user signup, I should get a message showing that the correct >>>>>>>> info >>>>>>>> would have been sent to Mixpanel." These use `assert_receive`. >>>>>>>> >>>>>>>> I just changed the format of the message built within >>>>>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests >>>>>>>> throughout >>>>>>>> my app, as expected. But since the tests didn't directly reference >>>>>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when >>>>>>>> the >>>>>>>> message format changed; I had to run all tests to get them to fail. >>>>>>>> >>>>>>>> This is no big deal, but it would be nice in such situations to run >>>>>>>> all tests once, then be able to whittle down the failing tests without >>>>>>>> re-running the whole suite. >>>>>>>> >>>>>>>> On Wednesday, November 22, 2017 at 4:54:51 PM UTC-5, Louis Pilfold >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi Nathan >>>>>>>>> >>>>>>>>> I feel ExUnit --stale should always be able to tell this. Could >>>>>>>>> you share your example please? >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Louis >>>>>>>>> >>>>>>>>> On Wed, 22 Nov 2017 at 20:43 Nathan Long <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Ruby's Rspec has a handy option, `--only-failures`, which >>>>>>>>>> "filters what examples are run so that only those that failed the >>>>>>>>>> last time >>>>>>>>>> they ran are executed". https://relishapp.com/rspec/rs >>>>>>>>>> pec-core/docs/command-line/only-failures >>>>>>>>>> >>>>>>>>>> I'd love to have this feature in ExUnit. The closest thing I see >>>>>>>>>> right now is `--stale`, but if ExUnit can't accurately determine >>>>>>>>>> which >>>>>>>>>> tests may have been broken by a change, it doesn't work. (I have >>>>>>>>>> such an >>>>>>>>>> example, but don't want to be long-winded; maybe the utility of this >>>>>>>>>> feature is clear enough?) >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3- >>>>>>>>>> ed51-44be-8f6b-81e5181fa449%40googlegroups.com >>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3-ed51-44be-8f6b-81e5181fa449%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elixir-lang-core" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6- >>>>>>>> f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6-f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elixir-lang-core" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee- >>>>>> aa76-4e05-b7ad-c06427e748b9%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elixir-lang-core" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9 >>>>> wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2B >>>> RGUx%2B7ACXs9nfQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej >>> %3Db4vQEt6r0Cw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej%3Db4vQEt6r0Cw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elixir-lang-core" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/elixir-lang-core/CAGnRm4LE9NLxeSxkceQuw%2BHAGEtZ3gY6jUJ3 >> WrLAw%3D9dREJY-Q%40mail.gmail.com >> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LE9NLxeSxkceQuw%2BHAGEtZ3gY6jUJ3WrLAw%3D9dREJY-Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elixir-lang-core/CADUxQmskeA4VYJAGxEMF9j%2B4SkHWHqGU5D5J62H4QyE% > 3DT2DyeA%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmskeA4VYJAGxEMF9j%2B4SkHWHqGU5D5J62H4QyE%3DT2DyeA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L1o--ChgtkkOteOB9V11Teb1mAxuL64tS6G8rAeJZEEg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
