STINNER Victor added the comment: > The exact behavior depends on the order of files in directories, on string > hashes randomization, on address randomization, and on many other things out > of our control.
For hash randomization, maybe we need to generate a PYTHONHASHSEED, as tox test runner does. For the filesystem: right, it's not possible to get 100% reproductible tests, but IMHO it's worth it to make them more reliable. > Couldn't reseeding the PRNG just add a false promise? I'm trying to fix random failures on the Refleaks buildbots, not to promise anything :-) To be honest, at this point, I don't know if it would be enough since I'm unable to reproduce bugs... > The success in making tests deterministic can also narrow down the coverage > of the testing. Some branches that lead to failures can be never executed. > Our target not just making tests always success, but catch and fix even > pretty rare errors. I know that it's though question, and that's why I opened this issue, to discuss it :-) But I see more and more projects to get more reproductible softwares and tests: * https://reproducible-builds.org/ * systemd big project to get more "stateless" computers, or said differently: to isolate better services * containers which also want to isolate services, "stateless" containers * etc. Other test runners, like tox, also make efforts to get reproductible tests, like setting PYTHONHASHSEED. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31227> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
