Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Victor, You're right, all the tests are currently passing the detector. There is still legacy code that does unmocking the 'old' way and there is a fair bit of it. Our concern if you remove the detector is that objects that have been mocked might not be unmocked correctly (due to copy/paste) and we might miss it in review. The detector won't miss it, and so we don't have to worry about it. We could fix all the old tests, but (as I said) it was agreed upon that that would not be productive and might even destabilize the code base, so it wasn't done. At the moment the detector is still doing a useful job and I don't think we can get rid of it just yet. Peter -Original Message- From: Victor Stinner [mailto:vstin...@redhat.com] Sent: May-16-16 12:10 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db Le 16/05/2016 16:12, Peter Stachowski a écrit : > We're aware that there are ways to mock (and un-mock) correctly. > We're trying to make sure that all our new test code follows those > patterns. We also decided that it wouldn't make sense to change all > the working tests to use the 'right' methods as that could have the > short-term effect of destabilizing the tests considerably (plus it > would be a fair bit of effort with very little actual gain). As a > compromise we added this code to check that things weren't left mocked > (just in case someone copy-pasted the old style, did it wrong and > no-one noticed - we're still trying to maintain consistency wherever > possible). Thanks for the explanation. But I don't understand something. The purpose of the code detecting dangling mocks is to find bugs in "legacy" tests which don't use correctly the mock module. I read the code and I saw that it makes the unit test failing in such case. All unit tests pass on Python 2. Does it mean that all the legacy code is gone and all unit tests use correctly the mock module? If I'm wrong, how can we detect remaining "dangling mocks"? Would it be possible to fix them at once to be able to remove the code detecting dangling mocks? I proposed a patch to remove the code detecting dangling mocks: it makes unit tests 13x faster on Python 3 (183.6 sec => 13.8 sec): https://review.openstack.org/#/c/316955/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
> -Original Message- > From: Victor Stinner [mailto:vstin...@redhat.com] > Sent: Monday, May 16, 2016 12:10 PM > To: OpenStack Development Mailing List (not for usage questions) > <openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db > > Le 16/05/2016 16:12, Peter Stachowski a écrit : > > We're aware that there are ways to mock (and un-mock) correctly. > > We're trying to make sure that all our new test code follows those > > patterns. We also decided that it wouldn't make sense to change all > > the working tests to use the 'right' methods as that could have the > > short-term effect of destabilizing the tests considerably (plus it > > would be a fair bit of effort with very little actual gain). As a > > compromise we added this code to check that things weren't left > > mocked (just in case someone copy-pasted the old style, did it wrong > > and no-one noticed - we're still trying to maintain consistency > > wherever possible). > > Thanks for the explanation. But I don't understand something. > > The purpose of the code detecting dangling mocks is to find bugs in > "legacy" tests which don't use correctly the mock module. I read the > code and I saw that it makes the unit test failing in such case. All > unit tests pass on Python 2. Does it mean that all the legacy code is > gone and all unit tests use correctly the mock module? > > If I'm wrong, how can we detect remaining "dangling mocks"? Would it be > possible to fix them at once to be able to remove the code detecting > dangling mocks? > [amrith] Not being flippant but this is analogous to saying that the tests in python27 now pass, so why not just remove the tests. That'd speed the thing up a lot! The reason this code remains is that we keep adding tests and from time to time someone adds a test that does something that leaves a dangling mock. And they can find it if they run tox. Even if they don't the gate will. That's the reason we have tests, to ensure that they continue to pass. > I proposed a patch to remove the code detecting dangling mocks: it makes > unit tests 13x faster on Python 3 (183.6 sec => 13.8 sec): > > https://review.openstack.org/#/c/316955/ > > Victor > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Victor, To correct the math, Python3 runs 686 tests in 15.971s or 42.95 tests per second. Python2 runs 1495 tests in 22.595s or 66.16 tests per second. Python2 is 54% faster (but even that's just an estimate, since it assumes all tests run at the same speed, which isn't true - plus it runs them on different hardware and not all VMs are created equal ;) ). We're aware that the 'dangling mock detector' makes the code run significantly slower (historically ~15 min instead of ~5 min) however it's helped us catch more than one bug with minimal effort. The question is why does it run even slower in python3? If it is just the infra issue of running on slower hardware, then we could probably address that. Peter -Original Message- From: Victor Stinner [mailto:vstin...@redhat.com] Sent: May-16-16 10:57 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db Le 16/05/2016 16:12, Peter Stachowski a écrit : > Amrith is right though - python34 is *much* slower running this check > (and somewhat in general) than python27. Maybe that doesn't translate > into real-world performance data, but it has made me a bit nervous > nonetheless. IMHO we should look more closely to the output. See my comments on the bug report: https://bugs.launchpad.net/trove/+bug/1582257 Currently, running unit tests on Python 3 runs 686 tests in 15.971s, whereas Python 2 runs 1495 tests in 22.595s. Python 3 is "faster" :-) (16 sec < 23 sec) The whole Python 3 job takes 10 minutes, but in these 10 minutes, almost 8 minutes are taken just to build binary wheel packages. On Python 2, prebuilt packages are used and so creating the virtual environment is much faster (around one minute). I proposed a change to prebuild also wheel packages on Python 3: https://review.openstack.org/#/c/316890/ Ok, but sometimes we get timeout, right? Again, if you look closely to the output, between two runs on Python 2, the timing also changes a lot: (a) Ran 1495 tests in 22.595s (b) Ran 1495 tests in 393.364s => 17x slower! The issue is not specific to Python 3. But it looks like the random slowdown is much larger on Python 3. Duration of Python 3 unit tests: (a) "Ran 686 tests in 15.971s" (b) "Ran 686 tests in 1581.090s" => 100x slower! I guess that the difference is that Python 3 unit tests are currently run sequentially (1 process), whereas Python 2 unit tests are run in parallel (8 processes). *Again* please check the code detecting dangling mocks. For example, "tox -e py34" takes 14 seconds without this code, 177 seconds with the code: 12x slower with the code. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Le 16/05/2016 16:12, Peter Stachowski a écrit : We're aware that there are ways to mock (and un-mock) correctly. We're trying to make sure that all our new test code follows those patterns. We also decided that it wouldn't make sense to change all the working tests to use the 'right' methods as that could have the short-term effect of destabilizing the tests considerably (plus it would be a fair bit of effort with very little actual gain). As a compromise we added this code to check that things weren't left mocked (just in case someone copy-pasted the old style, did it wrong and no-one noticed - we're still trying to maintain consistency wherever possible). Thanks for the explanation. But I don't understand something. The purpose of the code detecting dangling mocks is to find bugs in "legacy" tests which don't use correctly the mock module. I read the code and I saw that it makes the unit test failing in such case. All unit tests pass on Python 2. Does it mean that all the legacy code is gone and all unit tests use correctly the mock module? If I'm wrong, how can we detect remaining "dangling mocks"? Would it be possible to fix them at once to be able to remove the code detecting dangling mocks? I proposed a patch to remove the code detecting dangling mocks: it makes unit tests 13x faster on Python 3 (183.6 sec => 13.8 sec): https://review.openstack.org/#/c/316955/ Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Le 16/05/2016 16:12, Peter Stachowski a écrit : Amrith is right though - python34 is *much* slower running this check (and somewhat in general) than python27. Maybe that doesn't translate into real-world performance data, but it has made me a bit nervous nonetheless. IMHO we should look more closely to the output. See my comments on the bug report: https://bugs.launchpad.net/trove/+bug/1582257 Currently, running unit tests on Python 3 runs 686 tests in 15.971s, whereas Python 2 runs 1495 tests in 22.595s. Python 3 is "faster" :-) (16 sec < 23 sec) The whole Python 3 job takes 10 minutes, but in these 10 minutes, almost 8 minutes are taken just to build binary wheel packages. On Python 2, prebuilt packages are used and so creating the virtual environment is much faster (around one minute). I proposed a change to prebuild also wheel packages on Python 3: https://review.openstack.org/#/c/316890/ Ok, but sometimes we get timeout, right? Again, if you look closely to the output, between two runs on Python 2, the timing also changes a lot: (a) Ran 1495 tests in 22.595s (b) Ran 1495 tests in 393.364s => 17x slower! The issue is not specific to Python 3. But it looks like the random slowdown is much larger on Python 3. Duration of Python 3 unit tests: (a) "Ran 686 tests in 15.971s" (b) "Ran 686 tests in 1581.090s" => 100x slower! I guess that the difference is that Python 3 unit tests are currently run sequentially (1 process), whereas Python 2 unit tests are run in parallel (8 processes). *Again* please check the code detecting dangling mocks. For example, "tox -e py34" takes 14 seconds without this code, 177 seconds with the code: 12x slower with the code. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Le 09/05/2016 16:16, Peter Stachowski a écrit : Hi Amrith, We’ve had a lot of ‘new’ tests enabled for py34 in the last week – from your results you can see it’s running 6x the number of tests (although it’s taking 8.5x as long). It looks like python3 isn’t as fast as python2.7? If so, we may have to do something wrt the ‘dangling-mock’ detector code so that the tests run faster (I believe we can optimize it now that all the tests have moved over to the new paradigm). I opened a bug to collect information about this issue: https://bugs.launchpad.net/trove/+bug/1582257 Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Victor, We're aware that there are ways to mock (and un-mock) correctly. We're trying to make sure that all our new test code follows those patterns. We also decided that it wouldn't make sense to change all the working tests to use the 'right' methods as that could have the short-term effect of destabilizing the tests considerably (plus it would be a fair bit of effort with very little actual gain). As a compromise we added this code to check that things weren't left mocked (just in case someone copy-pasted the old style, did it wrong and no-one noticed - we're still trying to maintain consistency wherever possible). As the code base gets cleaned up we can eventually run this just once per test class, or even just once at the end of the test run. Petr has a changeset up now that will reduce it to twice per class (https://review.openstack.org/#/c/312781 ), and as I said we can drop that to just one probably by the end of the cycle. Amrith is right though - python34 is *much* slower running this check (and somewhat in general) than python27. Maybe that doesn't translate into real-world performance data, but it has made me a bit nervous nonetheless. Peter -Original Message- From: Victor Stinner [mailto:vstin...@redhat.com] Sent: May-16-16 9:38 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db Le 16/05/2016 13:52, Amrith Kumar a écrit : >> IMHO the strange mock detector code must be removed. It is very slow >> and I don't understand its purpose. > > [amrith] It serves and has served a very useful purpose and that is to > detect bad tests where code has (and we've had lots of trouble with > this) established a Mock() but failed to delete it. There are many options to disable (unregister, stop, call it as you want) mocks automatically. As I wrote, fixtures & oslotest give you tools to do that automatically. It's also a base feature of the mock module: "with mock.patch(...): ...". Sorry, I don't know enough the Trove code base (code of the unit tests) to say which option is the best. > I'd rather figure out why it is slower in Python3 because it may be > indicative of something that may impact other parts of the code as > well. We're having this whole discussion about Python performance and > the Go language, I think it is not a good idea to delete code which is > performing poorly because it is performing slowly. Sorry but Go doesn't solve badly designed functions :-) It's an algorithmic complexity issue: the function has to iterate on *all* alive Python objects: complexity of O(n). I propose to remove to code to simplify the complexity to O(1) :-) Ok, let's take an example with Python 2.7: the command "python -bb -m testtools.run trove/tests/unittests/common/test_exception.py" takes 972 ms. Remove the mock workaround, the command now taks 1 ms: it's now 972x faster. => removing the slow workaround makes test ~1000x faster! IMHO it's worth to start here, instead of starting to investigate why running tests on Python 3 seems slower. I'm not aware of such huge performance difference between Python 2 and Python 3 when running unit tests on other OpenStack services. The issue is specific to Trove, and IMHO it comes from the mock workaround. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Le 16/05/2016 13:52, Amrith Kumar a écrit : IMHO the strange mock detector code must be removed. It is very slow and I don't understand its purpose. [amrith] It serves and has served a very useful purpose and that is to detect bad tests where code has (and we've had lots of trouble with this) established a Mock() but failed to delete it. There are many options to disable (unregister, stop, call it as you want) mocks automatically. As I wrote, fixtures & oslotest give you tools to do that automatically. It's also a base feature of the mock module: "with mock.patch(...): ...". Sorry, I don't know enough the Trove code base (code of the unit tests) to say which option is the best. I'd rather figure out why it is slower in Python3 because it may be indicative of something that may impact other parts of the code as well. We're having this whole discussion about Python performance and the Go language, I think it is not a good idea to delete code which is performing poorly because it is performing slowly. Sorry but Go doesn't solve badly designed functions :-) It's an algorithmic complexity issue: the function has to iterate on *all* alive Python objects: complexity of O(n). I propose to remove to code to simplify the complexity to O(1) :-) Ok, let's take an example with Python 2.7: the command "python -bb -m testtools.run trove/tests/unittests/common/test_exception.py" takes 972 ms. Remove the mock workaround, the command now taks 1 ms: it's now 972x faster. => removing the slow workaround makes test ~1000x faster! IMHO it's worth to start here, instead of starting to investigate why running tests on Python 3 seems slower. I'm not aware of such huge performance difference between Python 2 and Python 3 when running unit tests on other OpenStack services. The issue is specific to Trove, and IMHO it comes from the mock workaround. Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
> -Original Message- > From: Victor Stinner [mailto:vstin...@redhat.com] > Sent: Monday, May 16, 2016 7:20 AM > To: OpenStack Development Mailing List (not for usage questions) > <openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db > > Hi, > > Le 09/05/2016 16:38, Amrith Kumar a écrit : > > So I'm able to run the py34 tests in about 10s if I run the dangling > > mock detector with a depth of 0 or 1. > > IMHO the strange mock detector code must be removed. It is very slow and > I don't understand its purpose. [amrith] It serves and has served a very useful purpose and that is to detect bad tests where code has (and we've had lots of trouble with this) established a Mock() but failed to delete it. Unfortunately, it is bad enough that Mock() is not thread specific (and the tests are multi-threaded) but even worse, if a mock persists later tests can go awfully wrong, or pass for no apparent reason. I'd rather figure out why it is slower in Python3 because it may be indicative of something that may impact other parts of the code as well. We're having this whole discussion about Python performance and the Go language, I think it is not a good idea to delete code which is performing poorly because it is performing slowly. > > The mock module has a nice stop_all() method, why not using it? > https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.patch > .stopall > [amrith] because stop_all isn't quite the same thing. > The fixtures and oslotest modules also help to stop mocks. > > (I don't know if the "mock detector" thing is related to the slow tests > on Python 3.) [amrith] It sure is slower in Python3. > > Victor > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Hi, Le 09/05/2016 16:38, Amrith Kumar a écrit : So I’m able to run the py34 tests in about 10s if I run the dangling mock detector with a depth of 0 or 1. IMHO the strange mock detector code must be removed. It is very slow and I don't understand its purpose. The mock module has a nice stop_all() method, why not using it? https://docs.python.org/dev/library/unittest.mock.html#unittest.mock.patch.stopall The fixtures and oslotest modules also help to stop mocks. (I don't know if the "mock detector" thing is related to the slow tests on Python 3.) Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
So I'm able to run the py34 tests in about 10s if I run the dangling mock detector with a depth of 0 or 1. I'd also like to figure out how we can get the py34 tests to show progress like py27 does. If a test fails, it is much harder to debug as it is now ... -amrith From: Peter Stachowski [mailto:pe...@tesora.com] Sent: Monday, May 09, 2016 10:17 AM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db Hi Amrith, We've had a lot of 'new' tests enabled for py34 in the last week - from your results you can see it's running 6x the number of tests (although it's taking 8.5x as long). It looks like python3 isn't as fast as python2.7? If so, we may have to do something wrt the 'dangling-mock' detector code so that the tests run faster (I believe we can optimize it now that all the tests have moved over to the new paradigm). Peter From: Amrith Kumar [mailto:amr...@tesora.com] Sent: May-08-16 8:42 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [trove] timeouts in gate-trove-python34-db I'm seeing python34's gate job running very long and sometimes it is timing out at 40m. There is something amiss with this and I've let Victor and Abhishek know about it. In https://review.openstack.org/#/c/313941/ which is a dummy commit (just changed README.rst) it took 37m. 2016-05-08 21:35:11.438 | Tests running... 2016-05-08 21:35:11.438 | 2016-05-08 21:35:11.438 | Ran 686 tests in 1581.090s 2016-05-08 21:35:11.439 | OK On the other hand, in other reviews (https://review.openstack.org/#/c/222752/) it takes barely 15m. 2016-05-02 19:26:06.018 | Tests running... 2016-05-02 19:26:06.019 | 2016-05-02 19:26:06.019 | Ran 115 tests in 185.864s 2016-05-02 19:26:06.019 | OK Running it on my desktop takes a long time as well. I guess we'll have to figure out why this is generating these failures. -amrith __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove] timeouts in gate-trove-python34-db
Hi Amrith, We've had a lot of 'new' tests enabled for py34 in the last week - from your results you can see it's running 6x the number of tests (although it's taking 8.5x as long). It looks like python3 isn't as fast as python2.7? If so, we may have to do something wrt the 'dangling-mock' detector code so that the tests run faster (I believe we can optimize it now that all the tests have moved over to the new paradigm). Peter From: Amrith Kumar [mailto:amr...@tesora.com] Sent: May-08-16 8:42 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [trove] timeouts in gate-trove-python34-db I'm seeing python34's gate job running very long and sometimes it is timing out at 40m. There is something amiss with this and I've let Victor and Abhishek know about it. In https://review.openstack.org/#/c/313941/ which is a dummy commit (just changed README.rst) it took 37m. 2016-05-08 21:35:11.438 | Tests running... 2016-05-08 21:35:11.438 | 2016-05-08 21:35:11.438 | Ran 686 tests in 1581.090s 2016-05-08 21:35:11.439 | OK On the other hand, in other reviews (https://review.openstack.org/#/c/222752/) it takes barely 15m. 2016-05-02 19:26:06.018 | Tests running... 2016-05-02 19:26:06.019 | 2016-05-02 19:26:06.019 | Ran 115 tests in 185.864s 2016-05-02 19:26:06.019 | OK Running it on my desktop takes a long time as well. I guess we'll have to figure out why this is generating these failures. -amrith __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
