New issue 667: Make tests using random numbers more reproducible https://bitbucket.org/hpk42/pytest/issue/667/make-tests-using-random-numbers-more
Nils Werner: I recently was thinking about what to do to make the use of random numbers more reproducible. Currently, when a test using a random number fails, there is no way of backtracking what random number seed was actually used so that the test can be reproduced. So... 1. While using `tox` I noticed it always sets `PYTHONHASHSEED` itself (to a random value, but printing that value) to ensure people can reproduce tests failing due to anything that has to do with that environment variable 2. While using `pytest-django` I noticed they always set the database back into its initial state and [require the tests to be marked (or have a `db` fixture)](http://pytest-django.readthedocs.org/en/latest/database.html#enabling-database-access) to touch the dabase. This brought me to the idea of controlling the `random` module a bit more to: - Set the seed - to a constant value, or - to a random value, printing that value in case of errors, or - reset the seed after each test, or - Allowing the test to set a seed without affecting other tests For example, currently you can do import pytest import my_module import random def test_something(): assert random.random() > 0.5 Which gives absolutely no information why and in what situations the (rather idiotic) test is failing. Or, when you have two tests import pytest import my_module import random def test_something(): assert random.random() > 0.5 def test_something_else(): assert random.random() > 0.5 The order of execution changes the values each test sees (particularily troubling with `xdist` I guess) ideally, the above code should fail and should be replaced with import pytest import my_module def test_something(random): assert random.random() > 0.5 That way pytest could always prepare `random` to be in the identical state for each tests and remember the seed value and print it, hinting at the fact that there are random numbers at play here. I haven't given too much thought about how to achieve this behaviour (monkeypatching the `random` module I guess) but first wanted to hear what you guys think. _______________________________________________ pytest-commit mailing list pytest-commit@python.org https://mail.python.org/mailman/listinfo/pytest-commit
