Hi Bruno, Thank you for your time, and for the suggestions.
Unfortunately they don’t address the core pain I am trying to solve - namely that expensive_setup is, well, expensive, and that running it for every test will in the long run be unacceptable. The difference is having to run it exactly the amount of times I’ve specified, vs that amount, times the count of tests who are marked repeat. If it repeats 100 times, say, you can easily see how badly this would scale with just a few dozen tests. -x and—max-fail, similarly, if I understand them correctly, would stop the -rest- of the suite from running, which isn’t my intent. Rather, I want the rest of the tests to run, only wanting to consolidate the repeated tests into one result per test, with one failure counting as the whole chain of repetition. But I think I can solve this with pytest_runtest_logreport. pytest-repeat, as far as I could tell from the documentation, repeats every test. That isn’t my intent, as only those I will explicitly mark provide extra value being re-run. The next thing I’m going to try is to make some sort of shared resource, probably using some resources for threading, and making each test ‘wait’ for the resource to be freed. Then, maybe, from the “tell me when resource becomes free” function, I can wait for everyone to be waiting, then reboot, then release the lock. Hopefully that’ll work. Kind regards, -- Emil Petersen | Software Test Engineer <https://www.thalmic.com/> T: 1-888-777-2546 x692 <> E: [email protected] <mailto:[email protected]> > On Oct 3, 2016, at 6:37 PM, Bruno Oliveira <[email protected]> wrote: > > Hello Emil, > > On Fri, Sep 30, 2016 at 6:26 PM Emil Petersen [email protected] > <http://mailto:[email protected]/> wrote: > > > I am looking to repeat some of the tests in my test suite, and want to call a > certain action before each repetition. To be clear, I want to: > > 1) First, run all the tests marked with repeat, as-is > 2) Then, take a specific action, in this case rebooting a device > 3) Then, repeat each test marked with repeat > 4) Then, repeat steps 2-3 the amount of times specified. > > Using the above as an example, I want to run test_something and > test_something_else, then expensive_setup, then the two tests again, then > expensive_setup, etc. So only one call to expensive_setup for each run > including test_something() and test_something_else() > > What would be a sensible way to go about this? > > > I might not full fill all your requirements, but here are some ideas: > > You can use a parametrized fixture which calls the expensive setup after its > first instantiation: > > def test_something(device): > check_something == last_result_from_file() > > def test_something_else(device): > something_else() > > @pytest.fixture(params=[0, 1, 2, 3] > def expensive_setup(request): > device = Device('uri_to_device') > device.connect() > if request.param > 0: > device.reboot() > return device > > > As an addendum, it is ideal if each test is only reported as -one- test, > rather than one test being generated for each run. > > > That’s not so simple, you will probably have to override the terminal plugin > and implement your own pytest_runtest_logreport to accumulate reports. > > I suspect this is more trouble than it is worth. > > > > Ideally, if any one call to the functions ends up generating a failure or > error, it would be able to fail, and abort any attempts at subsequently > running (I.e. failures should be final). > > > For that you can use --max-fail or -x, which will cause pytest to stop at the > first failure. > > > > In case you’re curious about why the tests need to be repeated, it’s to hunt > for race conditions, and other failures which may occur sometimes. > > > You might want to take a look at pytest-repeat > <https://github.com/pytest-dev/pytest-repeat>, which was created to help find > intermittent failures. > > Hope that helps! > > Cheers, > Bruno; > -- ---- CONFIDENTIALITY NOTICE: This communication, and any attachments thereto, is confidential and intended solely for use by the recipient(s) named above.
_______________________________________________ pytest-dev mailing list [email protected] https://mail.python.org/mailman/listinfo/pytest-dev
