Re: request to reopen ticket #24522: --random test option
Thank you for diving into how unittest discovery tests! On Friday, March 12, 2021 at 12:14:05 PM UTC+1 chris.j...@gmail.com wrote: > But, regardless of the default, to get the full benefit I think you'd want > at least one of the test runs in CI to be a random one. > This is were I somewhat disagree. We have to run it as often with as many different seeds as possible to have an effect. The space of tests is so big that I am not sure we'd find something otherwise (though I have to agree that is theoretical thinking I did not try in practice). Cheers, Florian -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4034f3db-e6ee-4d78-b140-553c73c1a170n%40googlegroups.com.
Re: request to reopen ticket #24522: --random test option
On Friday, March 12, 2021 at 10:47:56 AM UTC+1 Florian Apolloner wrote: > Is the current order defined in any way -- if not one should consider it > randoms anyways and as such actually randomizing should not make much of a > difference. FWIW I am fine with making it the default and outputting the > seed at the end of the run so it can reproduced. I am curious if it finds > many remaining issues. > I would characterize it as "mostly" sorted -- I think. I believe unittest's test discovery is used by default, in conjunction with a couple large test groups that Django configures (`DiscoverRunner.reorder_by = (TestCase, SimpleTestCase)`). The unittest package's test discovery is documented as first sorting by file path: > Paths are sorted before being imported so that execution order is the same > even if the underlying file system’s ordering is not dependent on file name. https://docs.python.org/3/library/unittest.html#unittest.TestLoader.discover And within test cases, test methods are sorted by name. For example, see-- https://docs.python.org/3/library/unittest.html#unittest.TestLoader.getTestCaseNames and-- > Note: The order in which the various tests will be run is determined by > sorting the test method names with respect to the built-in ordering for > strings. https://docs.python.org/3/library/unittest.html#organizing-test-code I don't know about classes within modules. On Fri, Mar 12, 2021 at 1:53 AM Florian Apolloner wrote: > ... Having randomization in CI might be interesting, but then we probably > need a way to derive the salt from the PR number or so, because I think the > number of tests inside a PR is small enough that you probably do not see > isolation failures, but you really want to see progress in a PR without > random failures due to isolation. > The question of how to use seeds in conjunction with CI if randomization were the default is an interesting one (e.g. how to tell CI to repeat a particular seed). Certainly if randomization were the default, CI could still be configured to be deterministic by passing --no-random / --sorted or whatever the opt-out option would be called. But, regardless of the default, to get the full benefit I think you'd want at least one of the test runs in CI to be a random one. --Chris > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAOTb1wfbbdFbauJXCtZwa_N7F-K5dMcB-Kw4EgLiwKgL_MRCsw%40mail.gmail.com.
Re: request to reopen ticket #24522: --random test option
Btw while I am fine with making it default (I really do not care much either way), we certainly need a way to disable it and keep the current algorithm (even though it might not be guaranteed stable). When I develop I often run one suite and fix the errors as they occur. Having a stable order there ensures that the tests I fixed before are still fixed when the next test fails (I quite often also run with failfast there). I think having a stable test output can be useful for development. Having randomization in CI might be interesting, but then we probably need a way to derive the salt from the PR number or so, because I think the number of tests inside a PR is small enough that you probably do not see isolation failures, but you really want to see progress in a PR without random failures due to isolation. On Friday, March 12, 2021 at 10:47:56 AM UTC+1 Florian Apolloner wrote: > Is the current order defined in any way -- if not one should consider it > randoms anyways and as such actually randomizing should not make much of a > difference. FWIW I am fine with making it the default and outputting the > seed at the end of the run so it can reproduced. I am curious if it finds > many remaining issues. > > On Thursday, March 11, 2021 at 12:15:44 AM UTC+1 chris.j...@gmail.com > wrote: > >> For those that would rather it be opt-out, remember that we can always >> make it opt-out later, once we have more experience with the feature. >> >> Regarding the suggestion to have `--random 0` mean not to use >> randomizing, I also think that could be confusing. 0 is also a valid >> integer seed, and it seems nice to have the option of passing 0 as a simple >> seed if one wants. We could use --no-random for disabling if we ever wanted >> to make the feature opt-out in the future. >> >> --Chris >> >> >> >> On Wednesday, March 10, 2021 at 4:42:08 AM UTC-8 Mariusz Felisiak wrote: >> >>> I usually agree with new features being opt-in, but perhaps this case is different? If I had tests that are breaking if executed randomly, I’d want to know about it yesterday. IOW, I’m having difficulty imagining a scenario where the user would be thankful for not activating this feature by default. So personally, I’d like to see an opt-out setting for this in settings.py. /$0.02 Fran >>> >>> `--reverse` will catch 95% of test isolation issues for you. It's >>> highly more likely that running tests in reverse order will catch isolation >>> issue for you than running them in a non-deterministic order. >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4ec1e24a-917b-4256-9949-6f5deded1a80n%40googlegroups.com.
Re: request to reopen ticket #24522: --random test option
Is the current order defined in any way -- if not one should consider it randoms anyways and as such actually randomizing should not make much of a difference. FWIW I am fine with making it the default and outputting the seed at the end of the run so it can reproduced. I am curious if it finds many remaining issues. On Thursday, March 11, 2021 at 12:15:44 AM UTC+1 chris.j...@gmail.com wrote: > For those that would rather it be opt-out, remember that we can always > make it opt-out later, once we have more experience with the feature. > > Regarding the suggestion to have `--random 0` mean not to use randomizing, > I also think that could be confusing. 0 is also a valid integer seed, and > it seems nice to have the option of passing 0 as a simple seed if one > wants. We could use --no-random for disabling if we ever wanted to make the > feature opt-out in the future. > > --Chris > > > > On Wednesday, March 10, 2021 at 4:42:08 AM UTC-8 Mariusz Felisiak wrote: > >> I usually agree with new features being opt-in, but perhaps this case is >>> different? >>> >>> If I had tests that are breaking if executed randomly, I’d want to know >>> about it yesterday. IOW, I’m having difficulty imagining a scenario where >>> the user would be thankful for not activating this feature by default. So >>> personally, I’d like to see an opt-out setting for this in settings.py. >>> >>> /$0.02 >>> Fran >>> >> >> `--reverse` will catch 95% of test isolation issues for you. It's highly >> more likely that running tests in reverse order will catch isolation issue >> for you than running them in a non-deterministic order. >> >> >> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/281504ae-18af-4d74-aca4-9d89de9b1eb9n%40googlegroups.com.
