Modifying your code a little even makes it work with other `mix test`
arguments.
def run(args) do
for _ <- 0..3 do
Mix.Task.run(:test, args)
Code.unrequire_files(Code.required_files())
end
end
That seems like the simplest solution that doesn’t require changes to the
standard test runner task.
-a
On Wed, Oct 21, 2020 at 1:04 PM Devon Estes <[email protected]> wrote:
> You don’t need to modify the runner to re-run tests on the same VM. You
> can have a short mix task that looks something like this:
>
> def rerun() do
> for _ <- 0..3 do
> Mix.Task.run(:test)
> Code.unrequire_files(Code.required_files())
> end
> end
>
> However, I also can’t think of any race condition that couldn’t be
> reliably reproduced by running the tests on a different VM instance with
> the same seed but would be reliably reproduced by running the tests in the
> same VM, and setting the seed is already available behavior.
>
> Austin Ziegler <[email protected]> schrieb am Mi. 21. Okt. 2020 um
> 18:32:
>
>> I don’t really have a strong opinion on this feature (it feels like it
>> would be easy to do on a per-suite or per-test basis by modifying the test
>> code, if tedious), but I think that a lot of the counter-cases are built on
>> a fundamental misunderstanding of lee’s use case.
>>
>> Basically, if something isn’t going to help users of the tool in the long
>>> run, it shouldn’t be made easier to do. Yes, it is time intensive _now_ to
>>> find and fix things difficult situations like race conditions, but by
>>> ignoring them you’re both going to make it harder to find and fix later on,
>>> as well as decreasing the quality of your tests until they are properly
>>> fixed.
>>
>>
>> The use case presented here is *specifically* to permit the repeated
>> execution of a test, test suite, or all suites N number of times to see if
>> a failure due to a race condition that does not express itself in a single
>> run (which every separate invocation of `mix test` is) can be produced by a
>> repeated run. This could be done with a command-line parameter or some
>> other mechanism (although other mechanisms would need to wrap either a
>> `test` or `describe` block explicitly, and would have no way of repeating
>> all suites in the event of some sort of race condition between multiple
>> async tests across the suite.
>>
>> That is, lee’s looking for assistance from ExUnit (or more specifically
>> the `mix test` runner of ExUnit) to figure out what is causing a race
>> condition that is appearing occasionally in their tests or perhaps more
>> frequently in production.
>>
>> It’s not a bad request, because the shell script repeater approach
>> provides a clean VM every time and the in-line code repeater approach
>> doesn’t permit cross-suite checking in the event that it really *is* a
>> weird interaction issue. Repeating all tests N times in the same VM can
>> *only* be done by modifying the runner (and provides the benefit of
>> working for test-and-suite repeats when a suite file or test line are
>> provided); repeating suite or tests can be done by wrapping the `describe`
>> or `test` blocks in loops.
>>
>> -a
>>
>> On Wed, Oct 21, 2020 at 10:54 AM Devon Estes <[email protected]>
>> wrote:
>>
>>> Making tools that folks can easily misuse does indeed damage the greater
>>> community, in that it can harm the perception of that tool (or In this
>>> case, language), which in turn hinders its adoption.
>>>
>>> There are already existing examples of this sort of choice in ExUnit
>>> where restrictions are put on APIs to help guide users towards decisions
>>> that are more sustainable in the long run, like not allowing more than one
>>> level of nesting with the ‘describe/2’ macro.
>>>
>>> Basically, if something isn’t going to help users of the tool in the
>>> long run, it shouldn’t be made easier to do. Yes, it is time intensive
>>> _now_ to find and fix things difficult situations like race conditions, but
>>> by ignoring them you’re both going to make it harder to find and fix later
>>> on, as well as decreasing the quality of your tests until they are properly
>>> fixed.
>>>
>>> On Wed, Oct 21, 2020 at 4:11 PM lee eggebroten <[email protected]>
>>> wrote:
>>>
>>>> It's Austin's case that's I'm trying to address:
>>>> - It's often time-intensive uncovering where a race condition is
>>>> occurring. Not to "prove" that it IS fixed 'cause that's just dumb.,
>>>>
>>>> The counter cases feel very ... elitist and controlling
>>>> - "I think you might abuse the tool so I'm not going to let you have
>>>> it".
>>>> - Will someone else's "misuse" damage you in some way?
>>>>
>>>> -lee
>>>>
>>>>
>>>>
>>>> On Tuesday, October 20, 2020 at 3:08:34 PM UTC-6 [email protected] wrote:
>>>>
>>>>> I agree with Shane.
>>>>>
>>>>> If you want to do this in your test suite (without a clean VM), I'd
>>>>> recommend creating an Elixir library that abstracts the `1..rerun_count |>
>>>>> Enum.each(current_test_function)` functionality that you require.
>>>>> I think that would be a better way to allow people that *really* want
>>>>> it to use this technique,
>>>>> while not implicitly encouraging people to resort to this logic in the
>>>>> way it would do when it were added as a built-in ExUnit feature.
>>>>>
>>>>> ~Marten
>>>>> On 20-10-2020 22:45, Austin Ziegler wrote:
>>>>>
>>>>> I agree that this is fairly easy to do in shell script, but that
>>>>> results in a clean VM every time. This is intended to find that rare case
>>>>> where an error only shows up with repeated executions on the VM, not as a
>>>>> “run this repeatedly until it works”. It’s more of a “run this until it
>>>>> breaks, up to N times”.
>>>>>
>>>>> I’m not sure that this is substantially better than modifying the test
>>>>> suite in this case, but it’s not a bad idea.
>>>>>
>>>>> On Tue, Oct 20, 2020 at 12:25 PM [email protected] <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> I feel this is better addressed through simple shell automation,
>>>>>> which has probably a dozen different ways to solve for this. Blessing the
>>>>>> idea of "just try again" as an actual ExUnit feature implicitly
>>>>>> encourages
>>>>>> community/developer tolerance of this kind of circumstance when all
>>>>>> practical efforts should be made to eliminate it from the code first,
>>>>>> wherever the problem actually originates. If those efforts fail, then
>>>>>> fall
>>>>>> back on surrounding it with retry logic on a case-by-case basis. I would
>>>>>> personally discourage use of such a feature if it existed.
>>>>>>
>>>>>> Shane Sveller
>>>>>>
>>>>>> On Monday, October 19, 2020 at 9:30:34 PM UTC-5 [email protected]
>>>>>> wrote:
>>>>>>
>>>>>>> *Problem*:
>>>>>>> for heisnbugs and race conditions it is useful to rerun a set of
>>>>>>> tests a number of times.
>>>>>>> Currently you must alter the suite(s) to wrap the the test(s) in
>>>>>>> `Enum`s which is obvioiusly tedious
>>>>>>>
>>>>>>> *Naive Implementation Proposal*:
>>>>>>> Add a `--rerun-count` command line option that would default to 1
>>>>>>> but wrap the test run with `1..<rerun-count> |> Enum(
>>>>>>> current_test_runner_parent_function )`
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "elixir-lang-core" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/elixir-lang-core/3581cdf4-4d23-41d1-afaa-151995e450f8n%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/3581cdf4-4d23-41d1-afaa-151995e450f8n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Austin Ziegler • [email protected] • [email protected]
>>>>> http://www.halostatue.ca/ • http://twitter.com/halostatue
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elixir-lang-core" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>>
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQvmO_sO-QuqbxAbaOZc_Y8PVx7t64PFWU40YSk4ttordg%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQvmO_sO-QuqbxAbaOZc_Y8PVx7t64PFWU40YSk4ttordg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elixir-lang-core" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elixir-lang-core/00dff9ce-fa83-4d20-8b98-d2ef1cb66ae5n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/elixir-lang-core/00dff9ce-fa83-4d20-8b98-d2ef1cb66ae5n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>>>
>>> _________________
>>> Devon Estes
>>> +49 176 2356 4717
>>> www.devonestes.com
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elixir-lang-core" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgrLRT7_njz3673jqPM7a_nLQO4UPFesJenfq3S_zL-Mw%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgrLRT7_njz3673jqPM7a_nLQO4UPFesJenfq3S_zL-Mw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> --
>> Austin Ziegler • [email protected] • [email protected]
>> http://www.halostatue.ca/ • http://twitter.com/halostatue
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elixir-lang-core" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuj8g8Eb8eReKQS9gTKa8d%2BC8HLC%2B0H9k%3DzGBJxqZDGTA%40mail.gmail.com
>> <https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuj8g8Eb8eReKQS9gTKa8d%2BC8HLC%2B0H9k%3DzGBJxqZDGTA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
>
> _________________
> Devon Estes
> +49 176 2356 4717
> www.devonestes.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "elixir-lang-core" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgOoYTj8Y9kBapEo-ZcxAt0w3DbbVj0ZXzCs9WPGO9x_w%40mail.gmail.com
> <https://groups.google.com/d/msgid/elixir-lang-core/CAGowJcgOoYTj8Y9kBapEo-ZcxAt0w3DbbVj0ZXzCs9WPGO9x_w%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
--
Austin Ziegler • [email protected] • [email protected]
http://www.halostatue.ca/ • http://twitter.com/halostatue
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQtap%3D4dS_oGgc1z-FLFmHMe_bk%2B7s-Je7DyH-ypfYDyhw%40mail.gmail.com.
