In that case, module + name should work just fine, so building the manifest is the first step :).
On Thu, Nov 23, 2017 at 1:55 PM, José Valim <[email protected]> wrote: > Tests are uniquely identified by module+name. It is not quite powerful as > an ID system but it does the job of identifying tests uniquely. > > > > *José Valimwww.plataformatec.com.br > <http://www.plataformatec.com.br/>Founder and Director of R&D* > > On Thu, Nov 23, 2017 at 7:14 PM, Myron Marston <[email protected]> > wrote: > >> I think the first step is to build the manifest itself which will give us >> the last_run_status information. Is that right? >> >> I think there’s a pre-requisite you need to get out of the way before you >> can build the manifest: you need to decide how you plan to uniquely >> identify each test. Does ExUnit already have something analogous to RSpec’s >> example ids? If not, you could potentially use either the test name or the >> test location (e.g. file_name:line_number) but those may not be >> sufficient (for RSpec they weren’t). For RSpec, the file location is not >> guaranteed unique, since you can dynamically define multiple tests in a >> loop, which results in multiple tests sharing the same file location, and >> this seems like a problem for ExUnit. Likewise, RSpec does not require that >> each test description is unique (I think ExUnit might require this, >> though…is that right?). Even if test descriptions are unique, it has some >> properties that, IMO, make it undesirable for use here: >> >> - There’s no easy way to map a test description back to the file the >> test is defined in, which means it limits the kind of cleanup you can do >> as >> part of merging the current results and the old results. At the end of a >> test run, RSpec cleans up the manifest by removing tests that cannot >> possibly still exist due to their file no longer existing, which is only >> possible since the example ids list what file the tests come from. >> - Test descriptions often change when the contents of the test may >> not. (Likewise, the location of a test can easily change just by the >> introduction of a helper function, an import or alias at the top of >> the module, etc). >> >> It’s easiest to explain how RSpec’s example ids work by showing an >> example: >> >> # foo_spec.rb >> RSpec.describe "Group 1" do >> it 'foos' do # foo_spec.rb[1:1] >> # ... >> end >> >> describe "a nested group" do >> it 'bars' do # foo_spec.rb[1:2:1] >> # ... >> end >> >> it 'bars again' do # foo_spec.rb[1:2:2] >> # ... >> end >> end >> >> it 'foos again' do # foo_spec.rb[1:3] >> >> endend >> RSpec.describe "Group 2" do >> it 'foos' do # foo_spec.rb[2:1] >> # ... >> endend >> >> Basically, we number each example and example group with a counter that >> starts over at 1 within each new scope, and use colons to separate the >> elements that form the “path” to the specific example. A nice thing about >> the ids is that they are relatively stable even in the sense of further >> development of the file. Users can change their test descriptions and >> introduce new things that change the line numbers, and the ids still work >> to correctly identify the tests. >> >> Would it make sense to introduce something like this for ExUnit? In RSpec >> we have found these ids to be useful for several other things (including >> --bisect, deterministic ordering when applying a seed to a subset, etc). >> >> BTW, this is something I’d be happy to take a stab at in ExUnit unless >> someone else wanted to do it. >> >> Myron >> >> >> >> On Thu, Nov 23, 2017 at 10:45 AM, José Valim <[email protected]> >> wrote: >> >>> That's very helpful, thank you Myron. >>> >>> We already keep several manifests for compiled code with the function >>> calls, files and modules. Therefore it should be relatively >>> straight-forward to keep one for tests. I think the first step is to build >>> the manifest itself which will give us the last_run_status information. Is >>> that right? >>> >>> Implementation-wise, we can probably even use a custom "formatter" to >>> maintain this information. All we need is a path to store this manifest >>> (which is opt-in but mix test can generate one by default in _build and >>> pass to ExUnit). >>> >>> >>> >>> >>> *José Valimwww.plataformatec.com.br >>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>> >>> On Thu, Nov 23, 2017 at 4:27 PM, Allen Madsen <[email protected]> >>> wrote: >>> >>>> +1 for --next-failure functionality. My current approach with ExUnit is >>>> basically a manual version of that. >>>> >>>> Allen Madsen >>>> http://www.allenmadsen.com >>>> >>>> On Thu, Nov 23, 2017 at 12:28 PM, Myron Marston < >>>> [email protected]> wrote: >>>> >>>>> I believe this would be a good addition. My only question is where are >>>>> the failed tests stored? In _build? >>>>> >>>>> For RSpec we made users configure where this state is stored, via a >>>>> config.example_status_persistence_file_path option. RSpec didn’t have >>>>> an established place to write that state so we left it up to the user to >>>>> decide where they wanted it to go. I think for ExUnit, storing it in >>>>> _build make sense. >>>>> >>>>> However, note that we are not merely storing a list of failed tests. >>>>> We store a list of *all* tests (including ones that were not included >>>>> in the latest run) that looks like this: >>>>> >>>>> example_id | >>>>> status | run_time | >>>>> ---------------------------------------------------------------------- | >>>>> ------- | --------------- | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:1] | >>>>> passed | 0.00115 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:2] | >>>>> passed | 0.00052 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:3] | >>>>> unknown | | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:4] | >>>>> passed | 0.00048 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:1:1] | >>>>> passed | 0.00058 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:2:1] | >>>>> failed | 0.00088 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:3:1] | >>>>> passed | 0.00084 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:1] | >>>>> passed | 0.00052 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:2] | >>>>> failed | 0.00059 seconds | >>>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:4:1] | >>>>> pending | 0.00053 seconds | >>>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:1] | >>>>> passed | 0.00366 seconds | >>>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:2] | >>>>> passed | 0.00307 seconds | >>>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:1] | >>>>> passed | 0.002 seconds | >>>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:2] | >>>>> passed | 0.00231 seconds | >>>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:4:1] | >>>>> passed | 0.00293 seconds | >>>>> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:1] | >>>>> passed | 0.00049 seconds | >>>>> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:2] | >>>>> passed | 0.0006 seconds | >>>>> >>>>> # ... >>>>> >>>>> This is a custom serialization format we designed to be easily >>>>> scannable by a human (as it’s useful information, particular the >>>>> run_time). The example_id column uniquely identifies each test (since >>>>> the other common ways to identify tests, such as description and file >>>>> location, are not guaranteed to be unique). Every time a test run >>>>> finishes, >>>>> we merge the results with the existing contents of this file using a >>>>> few simple rules >>>>> <https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/example_status_persister.rb#L66-L72> >>>>> . >>>>> >>>>> We then use this data to automatically add :last_run_status metadata >>>>> to every test (with values of passed, failed, pending or unknown) >>>>> when the spec files are loaded, which unlocks the generic ability to >>>>> filter >>>>> based on this via the RSpec CLI: >>>>> >>>>> $ rspec --tag last_run_status:failed >>>>> >>>>> This is the equivalent of --only failed like you asked about, José. >>>>> Whether or not you add an explicit option like --only-failures is up >>>>> to you, but the explicit option does provide a couple nice advantages for >>>>> RSpec: >>>>> >>>>> - It surfaces this extremely useful option in the --help output. >>>>> Without calling it out, it would not be clear to most users that >>>>> failure >>>>> filtering is possible. >>>>> - Since we can easily tell from our persistence file which spec >>>>> files have failures, when --only-failures is passed, we >>>>> automatically load only those files. In contrast, --tag filtering >>>>> doesn’t generally know anything in advance about which files might have >>>>> specs matching the tag, so --tag last_run_status:failed will load >>>>> *all* spec files, and then apply the filtering. This can be >>>>> significantly slower, particularly if there are files without failures >>>>> that >>>>> load a heavyweight dependency (like rails). >>>>> >>>>> One other option we provide (which ExUnit may or may not want to >>>>> provide) is --next-failure. This is the equivalent of --only-failures >>>>> --fail-fast --order defined. The idea is that you often want to work >>>>> through the failures systematically one-by-one. --fail-fast causes >>>>> RSpec to abort as soon as the first failure is hit and --order defined >>>>> disables the random ordering so you get the same failed example when you >>>>> run rspec --next-failure over and over again to help you focus on a >>>>> specific one. This option is also why we do the merging operation with the >>>>> status from prior runs: it’s important that we preserve the failed >>>>> status of tests that weren’t executed in the latest run. >>>>> >>>>> ExUnit certainly doesn’t have to go the same route RSpec went here, >>>>> but the combination of the perf speed up from avoiding loading files with >>>>> only passing tests and the usefulness of --next-failure is pretty >>>>> awesome, IMO. >>>>> >>>>> Myron >>>>> >>>>> >>>>> On Thu, Nov 23, 2017 at 4:03 AM, José Valim <[email protected]> >>>>> wrote: >>>>> >>>>>> Thanks everyone! >>>>>> >>>>>> I believe this would be a good addition. My only question is where >>>>>> are the failed tests stored? In _build? Also, maybe we can also implement >>>>>> it as a special tag called "--only failed" or "--only failures"? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *José Valimwww.plataformatec.com.br >>>>>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>>>>> >>>>>> On Thu, Nov 23, 2017 at 6:03 AM, Myron Marston < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I too would love to see ExUnit support an `--only-failures` flag. >>>>>>> It's one of my favorite features of RSpec and I wish every test >>>>>>> framework >>>>>>> had it. I find that it makes a huge difference to my workflow to be >>>>>>> able >>>>>>> to quickly and easily filter to the tests that failed the last time they >>>>>>> ran. >>>>>>> >>>>>>> In fact, I love this feature of RSpec so much that I was the one who >>>>>>> added it to the framework a couple years back :). I'd be happy to help >>>>>>> see >>>>>>> it get added to ExUnit if José and others were amenable. ExUnit already >>>>>>> has most of the building blocks needed for it via tags and filtering. >>>>>>> >>>>>>> Myron >>>>>>> >>>>>>> On Wednesday, November 22, 2017 at 2:48:14 PM UTC-8, José Valim >>>>>>> wrote: >>>>>>>> >>>>>>>> To clarify, --stale does not run previously failed tests. >>>>>>>> >>>>>>>> > I just changed the format of the message built within >>>>>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests >>>>>>>> throughout >>>>>>>> my app, as expected. But since the tests didn't directly reference >>>>>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when >>>>>>>> the >>>>>>>> message format changed; I had to run all tests to get them to fail. >>>>>>>> >>>>>>>> That feels like a bug. Maybe we are being conservative on how we >>>>>>>> compute the dependencies. If you can provide a sample app that >>>>>>>> reproduces >>>>>>>> the error, I would love to take a look at it. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *José Valimwww.plataformatec.com.br >>>>>>>> <http://www.plataformatec.com.br/>Founder and Director of R&D* >>>>>>>> >>>>>>>> On Wed, Nov 22, 2017 at 8:06 PM, Nathan Long <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Sure. I have a module called `MyApp.Mixpanel` with functions like >>>>>>>>> `track_event(:user_signup, data_map)`. These are called from various >>>>>>>>> places >>>>>>>>> throughout the codebase. There's a production adapter, which actually >>>>>>>>> sends >>>>>>>>> the event data to Mixpanel for analytics purposes, a dev adapter, >>>>>>>>> which >>>>>>>>> just logs it, and a test adapter, which sends it to `self()` as a >>>>>>>>> message. >>>>>>>>> >>>>>>>>> Several of my tests say things like "if I POST the info required >>>>>>>>> for a new user signup, I should get a message showing that the >>>>>>>>> correct info >>>>>>>>> would have been sent to Mixpanel." These use `assert_receive`. >>>>>>>>> >>>>>>>>> I just changed the format of the message built within >>>>>>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests >>>>>>>>> throughout >>>>>>>>> my app, as expected. But since the tests didn't directly reference >>>>>>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when >>>>>>>>> the >>>>>>>>> message format changed; I had to run all tests to get them to fail. >>>>>>>>> >>>>>>>>> This is no big deal, but it would be nice in such situations to >>>>>>>>> run all tests once, then be able to whittle down the failing tests >>>>>>>>> without >>>>>>>>> re-running the whole suite. >>>>>>>>> >>>>>>>>> On Wednesday, November 22, 2017 at 4:54:51 PM UTC-5, Louis Pilfold >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Nathan >>>>>>>>>> >>>>>>>>>> I feel ExUnit --stale should always be able to tell this. Could >>>>>>>>>> you share your example please? >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Louis >>>>>>>>>> >>>>>>>>>> On Wed, 22 Nov 2017 at 20:43 Nathan Long <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Ruby's Rspec has a handy option, `--only-failures`, which >>>>>>>>>>> "filters what examples are run so that only those that failed the >>>>>>>>>>> last time >>>>>>>>>>> they ran are executed". https://relishapp.com/rspec/rs >>>>>>>>>>> pec-core/docs/command-line/only-failures >>>>>>>>>>> >>>>>>>>>>> I'd love to have this feature in ExUnit. The closest thing I see >>>>>>>>>>> right now is `--stale`, but if ExUnit can't accurately determine >>>>>>>>>>> which >>>>>>>>>>> tests may have been broken by a change, it doesn't work. (I have >>>>>>>>>>> such an >>>>>>>>>>> example, but don't want to be long-winded; maybe the utility of this >>>>>>>>>>> feature is clear enough?) >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "elixir-lang-core" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3- >>>>>>>>>>> ed51-44be-8f6b-81e5181fa449%40googlegroups.com >>>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3-ed51-44be-8f6b-81e5181fa449%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elixir-lang-core" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6- >>>>>>>>> f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6-f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elixir-lang-core" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee- >>>>>>> aa76-4e05-b7ad-c06427e748b9%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "elixir-lang-core" group. >>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>>> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9 >>>>>> wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elixir-lang-core" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmvF >>>>> XN0hkrbOc39359DboqT-W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elixir-lang-core" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej >>>> %3Db4vQEt6r0Cw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej%3Db4vQEt6r0Cw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elixir-lang-core" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/elixir-lang-core/CAGnRm4LE9NLxeSxkceQuw%2BHAGEtZ3gY6jUJ3 >>> WrLAw%3D9dREJY-Q%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LE9NLxeSxkceQuw%2BHAGEtZ3gY6jUJ3WrLAw%3D9dREJY-Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/elixir-lang-core/CADUxQmskeA4VYJAGxEMF9j%2B4SkHWHqGU5D5J >> 62H4QyE%3DT2DyeA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmskeA4VYJAGxEMF9j%2B4SkHWHqGU5D5J62H4QyE%3DT2DyeA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elixir-lang-core" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/elixir-lang-core/_jbuzf4UvA4/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elixir-lang-core/CAGnRm4L1o--ChgtkkOteOB9V11Teb1mAxuL64tS6G > 8rAeJZEEg%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L1o--ChgtkkOteOB9V11Teb1mAxuL64tS6G8rAeJZEEg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmv3Ge9d4U1CctBFymQv554H%3DeWmETw1t9oT7jRgJ58WGw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
