I think the first step is to build the manifest itself which will give us
the last_run_status information. Is that right?
I think there’s a pre-requisite you need to get out of the way before you
can build the manifest: you need to decide how you plan to uniquely
identify each test. Does ExUnit already have something analogous to RSpec’s
example ids? If not, you could potentially use either the test name or the
test location (e.g. file_name:line_number) but those may not be sufficient
(for RSpec they weren’t). For RSpec, the file location is not guaranteed
unique, since you can dynamically define multiple tests in a loop, which
results in multiple tests sharing the same file location, and this seems
like a problem for ExUnit. Likewise, RSpec does not require that each test
description is unique (I think ExUnit might require this, though…is that
right?). Even if test descriptions are unique, it has some properties that,
IMO, make it undesirable for use here:
- There’s no easy way to map a test description back to the file the
test is defined in, which means it limits the kind of cleanup you can do as
part of merging the current results and the old results. At the end of a
test run, RSpec cleans up the manifest by removing tests that cannot
possibly still exist due to their file no longer existing, which is only
possible since the example ids list what file the tests come from.
- Test descriptions often change when the contents of the test may not.
(Likewise, the location of a test can easily change just by the
introduction of a helper function, an import or alias at the top of the
module, etc).
It’s easiest to explain how RSpec’s example ids work by showing an example:
# foo_spec.rb
RSpec.describe "Group 1" do
it 'foos' do # foo_spec.rb[1:1]
# ...
end
describe "a nested group" do
it 'bars' do # foo_spec.rb[1:2:1]
# ...
end
it 'bars again' do # foo_spec.rb[1:2:2]
# ...
end
end
it 'foos again' do # foo_spec.rb[1:3]
endend
RSpec.describe "Group 2" do
it 'foos' do # foo_spec.rb[2:1]
# ...
endend
Basically, we number each example and example group with a counter that
starts over at 1 within each new scope, and use colons to separate the
elements that form the “path” to the specific example. A nice thing about
the ids is that they are relatively stable even in the sense of further
development of the file. Users can change their test descriptions and
introduce new things that change the line numbers, and the ids still work
to correctly identify the tests.
Would it make sense to introduce something like this for ExUnit? In RSpec
we have found these ids to be useful for several other things (including
--bisect, deterministic ordering when applying a seed to a subset, etc).
BTW, this is something I’d be happy to take a stab at in ExUnit unless
someone else wanted to do it.
Myron
On Thu, Nov 23, 2017 at 10:45 AM, José Valim <[email protected]> wrote:
> That's very helpful, thank you Myron.
>
> We already keep several manifests for compiled code with the function
> calls, files and modules. Therefore it should be relatively
> straight-forward to keep one for tests. I think the first step is to build
> the manifest itself which will give us the last_run_status information. Is
> that right?
>
> Implementation-wise, we can probably even use a custom "formatter" to
> maintain this information. All we need is a path to store this manifest
> (which is opt-in but mix test can generate one by default in _build and
> pass to ExUnit).
>
>
>
>
> *José Valimwww.plataformatec.com.br
> <http://www.plataformatec.com.br/>Founder and Director of R&D*
>
> On Thu, Nov 23, 2017 at 4:27 PM, Allen Madsen <[email protected]>
> wrote:
>
>> +1 for --next-failure functionality. My current approach with ExUnit is
>> basically a manual version of that.
>>
>> Allen Madsen
>> http://www.allenmadsen.com
>>
>> On Thu, Nov 23, 2017 at 12:28 PM, Myron Marston <[email protected]>
>> wrote:
>>
>>> I believe this would be a good addition. My only question is where are
>>> the failed tests stored? In _build?
>>>
>>> For RSpec we made users configure where this state is stored, via a
>>> config.example_status_persistence_file_path option. RSpec didn’t have
>>> an established place to write that state so we left it up to the user to
>>> decide where they wanted it to go. I think for ExUnit, storing it in
>>> _build make sense.
>>>
>>> However, note that we are not merely storing a list of failed tests. We
>>> store a list of *all* tests (including ones that were not included in
>>> the latest run) that looks like this:
>>>
>>> example_id |
>>> status | run_time |
>>> ---------------------------------------------------------------------- |
>>> ------- | --------------- |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:1] |
>>> passed | 0.00115 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:2] |
>>> passed | 0.00052 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:3] |
>>> unknown | |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:1:4] |
>>> passed | 0.00048 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:1:1] |
>>> passed | 0.00058 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:2:1] |
>>> failed | 0.00088 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:2:3:1] |
>>> passed | 0.00084 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:1] |
>>> passed | 0.00052 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:3:2] |
>>> failed | 0.00059 seconds |
>>> ./spec/rspec/core/backtrace_formatter_spec.rb[1:4:1] |
>>> pending | 0.00053 seconds |
>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:1] |
>>> passed | 0.00366 seconds |
>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:2] |
>>> passed | 0.00307 seconds |
>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:1] |
>>> passed | 0.002 seconds |
>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:3:2] |
>>> passed | 0.00231 seconds |
>>> ./spec/rspec/core/bisect/coordinator_spec.rb[1:4:1] |
>>> passed | 0.00293 seconds |
>>> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:1] |
>>> passed | 0.00049 seconds |
>>> ./spec/rspec/core/bisect/example_minimizer_spec.rb[1:2] |
>>> passed | 0.0006 seconds |
>>>
>>> # ...
>>>
>>> This is a custom serialization format we designed to be easily scannable
>>> by a human (as it’s useful information, particular the run_time). The
>>> example_id column uniquely identifies each test (since the other common
>>> ways to identify tests, such as description and file location, are not
>>> guaranteed to be unique). Every time a test run finishes, we merge the
>>> results with the existing contents of this file using a few simple rules
>>> <https://github.com/rspec/rspec-core/blob/v3.7.0/lib/rspec/core/example_status_persister.rb#L66-L72>
>>> .
>>>
>>> We then use this data to automatically add :last_run_status metadata to
>>> every test (with values of passed, failed, pending or unknown) when the
>>> spec files are loaded, which unlocks the generic ability to filter based on
>>> this via the RSpec CLI:
>>>
>>> $ rspec --tag last_run_status:failed
>>>
>>> This is the equivalent of --only failed like you asked about, José.
>>> Whether or not you add an explicit option like --only-failures is up to
>>> you, but the explicit option does provide a couple nice advantages for
>>> RSpec:
>>>
>>> - It surfaces this extremely useful option in the --help output.
>>> Without calling it out, it would not be clear to most users that failure
>>> filtering is possible.
>>> - Since we can easily tell from our persistence file which spec
>>> files have failures, when --only-failures is passed, we
>>> automatically load only those files. In contrast, --tag filtering
>>> doesn’t generally know anything in advance about which files might have
>>> specs matching the tag, so --tag last_run_status:failed will load
>>> *all* spec files, and then apply the filtering. This can be
>>> significantly slower, particularly if there are files without failures
>>> that
>>> load a heavyweight dependency (like rails).
>>>
>>> One other option we provide (which ExUnit may or may not want to
>>> provide) is --next-failure. This is the equivalent of --only-failures
>>> --fail-fast --order defined. The idea is that you often want to work
>>> through the failures systematically one-by-one. --fail-fast causes
>>> RSpec to abort as soon as the first failure is hit and --order defined
>>> disables the random ordering so you get the same failed example when you
>>> run rspec --next-failure over and over again to help you focus on a
>>> specific one. This option is also why we do the merging operation with the
>>> status from prior runs: it’s important that we preserve the failed
>>> status of tests that weren’t executed in the latest run.
>>>
>>> ExUnit certainly doesn’t have to go the same route RSpec went here, but
>>> the combination of the perf speed up from avoiding loading files with only
>>> passing tests and the usefulness of --next-failure is pretty awesome,
>>> IMO.
>>>
>>> Myron
>>>
>>>
>>> On Thu, Nov 23, 2017 at 4:03 AM, José Valim <[email protected]>
>>> wrote:
>>>
>>>> Thanks everyone!
>>>>
>>>> I believe this would be a good addition. My only question is where are
>>>> the failed tests stored? In _build? Also, maybe we can also implement it as
>>>> a special tag called "--only failed" or "--only failures"?
>>>>
>>>>
>>>>
>>>>
>>>> *José Valimwww.plataformatec.com.br
>>>> <http://www.plataformatec.com.br/>Founder and Director of R&D*
>>>>
>>>> On Thu, Nov 23, 2017 at 6:03 AM, Myron Marston <[email protected]
>>>> > wrote:
>>>>
>>>>> I too would love to see ExUnit support an `--only-failures` flag.
>>>>> It's one of my favorite features of RSpec and I wish every test framework
>>>>> had it. I find that it makes a huge difference to my workflow to be able
>>>>> to quickly and easily filter to the tests that failed the last time they
>>>>> ran.
>>>>>
>>>>> In fact, I love this feature of RSpec so much that I was the one who
>>>>> added it to the framework a couple years back :). I'd be happy to help
>>>>> see
>>>>> it get added to ExUnit if José and others were amenable. ExUnit already
>>>>> has most of the building blocks needed for it via tags and filtering.
>>>>>
>>>>> Myron
>>>>>
>>>>> On Wednesday, November 22, 2017 at 2:48:14 PM UTC-8, José Valim wrote:
>>>>>>
>>>>>> To clarify, --stale does not run previously failed tests.
>>>>>>
>>>>>> > I just changed the format of the message built within
>>>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests
>>>>>> throughout
>>>>>> my app, as expected. But since the tests didn't directly reference
>>>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when the
>>>>>> message format changed; I had to run all tests to get them to fail.
>>>>>>
>>>>>> That feels like a bug. Maybe we are being conservative on how we
>>>>>> compute the dependencies. If you can provide a sample app that reproduces
>>>>>> the error, I would love to take a look at it.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *José Valimwww.plataformatec.com.br
>>>>>> <http://www.plataformatec.com.br/>Founder and Director of R&D*
>>>>>>
>>>>>> On Wed, Nov 22, 2017 at 8:06 PM, Nathan Long <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Sure. I have a module called `MyApp.Mixpanel` with functions like
>>>>>>> `track_event(:user_signup, data_map)`. These are called from various
>>>>>>> places
>>>>>>> throughout the codebase. There's a production adapter, which actually
>>>>>>> sends
>>>>>>> the event data to Mixpanel for analytics purposes, a dev adapter, which
>>>>>>> just logs it, and a test adapter, which sends it to `self()` as a
>>>>>>> message.
>>>>>>>
>>>>>>> Several of my tests say things like "if I POST the info required for
>>>>>>> a new user signup, I should get a message showing that the correct info
>>>>>>> would have been sent to Mixpanel." These use `assert_receive`.
>>>>>>>
>>>>>>> I just changed the format of the message built within
>>>>>>> `MyApp.Mixpanel`. This caused `assert_receive` to fail in tests
>>>>>>> throughout
>>>>>>> my app, as expected. But since the tests didn't directly reference
>>>>>>> `MyApp.Mixpanel`, `--stale` didn't know which ones should be run when
>>>>>>> the
>>>>>>> message format changed; I had to run all tests to get them to fail.
>>>>>>>
>>>>>>> This is no big deal, but it would be nice in such situations to run
>>>>>>> all tests once, then be able to whittle down the failing tests without
>>>>>>> re-running the whole suite.
>>>>>>>
>>>>>>> On Wednesday, November 22, 2017 at 4:54:51 PM UTC-5, Louis Pilfold
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Nathan
>>>>>>>>
>>>>>>>> I feel ExUnit --stale should always be able to tell this. Could you
>>>>>>>> share your example please?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Louis
>>>>>>>>
>>>>>>>> On Wed, 22 Nov 2017 at 20:43 Nathan Long <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ruby's Rspec has a handy option, `--only-failures`, which "filters
>>>>>>>>> what examples are run so that only those that failed the last time
>>>>>>>>> they ran
>>>>>>>>> are executed". https://relishapp.com/rspec/rs
>>>>>>>>> pec-core/docs/command-line/only-failures
>>>>>>>>>
>>>>>>>>> I'd love to have this feature in ExUnit. The closest thing I see
>>>>>>>>> right now is `--stale`, but if ExUnit can't accurately determine which
>>>>>>>>> tests may have been broken by a change, it doesn't work. (I have such
>>>>>>>>> an
>>>>>>>>> example, but don't want to be long-winded; maybe the utility of this
>>>>>>>>> feature is clear enough?)
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "elixir-lang-core" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to [email protected].
>>>>>>>>> To view this discussion on the web visit
>>>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3-
>>>>>>>>> ed51-44be-8f6b-81e5181fa449%40googlegroups.com
>>>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/f5881fa3-ed51-44be-8f6b-81e5181fa449%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>> .
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "elixir-lang-core" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6-
>>>>>>> f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com
>>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/2aa483e6-f63c-42d6-9e4b-84efb8adf9de%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elixir-lang-core" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee-
>>>>> aa76-4e05-b7ad-c06427e748b9%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/270ca4ee-aa76-4e05-b7ad-c06427e748b9%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "elixir-lang-core" group.
>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>>>> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> [email protected].
>>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>>> gid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJ
>>>> sP8ggzTngnGuxw%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J9wMEN4w3wZ4WPio%3DVvCSmgtpcdQJJsP8ggzTngnGuxw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elixir-lang-core" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2B
>>> RGUx%2B7ACXs9nfQ%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmvFXN0hkrbOc39359DboqT-W0Exxdz%2BRGUx%2B7ACXs9nfQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elixir-lang-core" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej
>> %3Db4vQEt6r0Cw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3Csn4Ka6e1Vu4njkmq2WZfv5QiRLfhQsej%3Db4vQEt6r0Cw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elixir-lang-core" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/to
> pic/elixir-lang-core/_jbuzf4UvA4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/elixir-lang-core/CAGnRm4LE9NLxeSxkceQuw%2BHAGEtZ3gY6jUJ3
> WrLAw%3D9dREJY-Q%40mail.gmail.com
> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4LE9NLxeSxkceQuw%2BHAGEtZ3gY6jUJ3WrLAw%3D9dREJY-Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/CADUxQmskeA4VYJAGxEMF9j%2B4SkHWHqGU5D5J62H4QyE%3DT2DyeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
