Re: [openstack-dev] Proposal for including fake implementations in python-client packages
On Wed, Jul 3, 2013 at 6:19 PM, Christopher Armstrong chris.armstr...@rackspace.com wrote: On Tue, Jul 2, 2013 at 11:38 PM, Robert Collins robe...@robertcollins.net wrote: Radix points out I missed the naunce that you're targeting the users of python-novaclient, for instance, rather than python-novaclient's own tests. On 3 July 2013 16:29, Robert Collins robe...@robertcollins.net wrote: What I'd like is for each client library, in addition to the actual implementation, is that they ship a fake, in-memory, version of the API. The fake implementations should take the same arguments, have the same return values, raise the same exceptions, and otherwise be identical, besides the fact that they are entirely in memory and never make network requests. So, +1 on shipping a fake reference copy of the API. -1 on shipping it in the client. The server that defines the API should have two implementations - the production one, and a testing fake. The server tests should exercise *both* code paths [e.g. using testscenarios] to ensure there is no skew between them. Then the client tests can be fast and efficient but not subject to implementation skew between fake and prod implementations. Back on Launchpad I designed a similar thing, but with language neutrality as a goal : https://dev.launchpad.net/ArchitectureGuide/ServicesRequirements#Test_fake And in fact, I think that that design would work well here, because we have multiple language bindings - Python, Ruby, PHP, Java, Go etc, and all of them will benefit from a low(ms or less)-latency test fake. So taking the aspect I missed into account I'm much happier with the idea of shipping a fake in the client, but... AFAICT many of our client behaviours are only well defined in the presence of a server anyhow. So it seems to me that a fast server fake can be used in tests of python-novaclient, *and* in tests of code using python-novaclient (including for instance, heat itself), and we get to write it just once per server, rather than once per server per language binding. -Rob I want to make sure I understond you. Let's say I have a program named cool-cloud-tool, and it uses python-novaclient, python-keystoneclient, and three other clients for OpenStack services. You're suggesting that its test suite should start up instances of all those OpenStack services with in-memory or otherwise localized backends, and communicate with them using standard python-*client functionality? I can imagine that being a useful thing, if it's very easy to do, and won't increase my test execution time too much. The easiest way to do this would be to set up a devstack instance to run your tests against, but that would increase test execution time by at least a few minutes. So although something like this would not work well for unit tests it could be useful for integration tests. Also I do not know of anyone using this for this type of testing yet, but someone always has to be first. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for including fake implementations in python-client packages
On 4 July 2013 04:19, Christopher Armstrong chris.armstr...@rackspace.comwrote: On Tue, Jul 2, 2013 at 11:38 PM, Robert Collins robe...@robertcollins.net wrote: Radix points out I missed the naunce that you're targeting the users of python-novaclient, for instance, rather than python-novaclient's own tests. ... So it seems to me that a fast server fake can be used in tests of python-novaclient, *and* in tests of code using python-novaclient (including for instance, heat itself), and we get to write it just once per server, rather than once per server per language binding. -Rob I want to make sure I understond you. Let's say I have a program named cool-cloud-tool, and it uses python-novaclient, python-keystoneclient, and three other clients for OpenStack services. You're suggesting that its test suite should start up instances of all those OpenStack services with in-memory or otherwise localized backends, and communicate with them using standard python-*client functionality? Yes. I can imagine that being a useful thing, if it's very easy to do, and won't increase my test execution time too much. In bzr's test suite we found that such tests (using a non-encrypted plain socket to talk to servers, rather than mock objects) was plenty fast: slowness in the test suite occurred in tests that did a lot of actual vcs operations - it's the lower layers that needed chopping out. I think it's important to draw a distinction between what Joe Gordon is suggesting and what I'm suggesting: nova operations are 10's to 100's of ms today. I'm talking about a lightweight server that responds in single-digit ms. Consider a simple API call - say nova boot. Right now that has a good half dozen rabbit RPC calls: API hands off work which is then done done asynchronously: scheduler, neutron, compute, keystone once keypairs live there, cinder and glance. It's a *lot* of work which isn't useful for cool-cloud-tool, which just wants in it's test suite to: a) make a nova API call b) wait for the 'instance to boot' c) check that the expected API calls were made a and b should be no more than a couple ms apart in a test server; (c) is specific to a test server and not a use case the production server /should/ implement. So while today the fake virt backend for nova exists, its still too far down the stack to deliver the sort of story I'm talking about. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Cloud Services ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for including fake implementations in python-client packages
On Mon, Jul 1, 2013 at 11:08 PM, Alex Gaynor alex.gay...@gmail.com wrote: Hi all, I suspect many of you don't know me as I've only started to get involved in OpenStack recently, I work at Rackspace and I'm pretty involved in other Python open source stuff, notably Django and PyPy, I also serve on the board of the PSF. So hiwave /! I'd like to propose an addition to all of the python-client libraries going forwards (and perhaps a requirement for future ones). What I'd like is for each client library, in addition to the actual implementation, is that they ship a fake, in-memory, version of the API. The fake implementations should take the same arguments, have the same return values, raise the same exceptions, and otherwise be identical, besides the fact that they are entirely in memory and never make network requests. Why not ``mock.Mock(spec=...)``: First, for those not familiar with the distinction between fakes and mocks (and doubles and stubs and ...): http://mumak.net/test-doubles/ is a great resource. https://www.youtube.com/watch?v=Xu5EhKVZdV8 is also a great resource which explains much of what I'm about to say, but better. Fakes are better than Mocks, for this because: * Mocks tend to be brittle because they're testing for the implementation, and not the interface. * Each mock tends to grow its own divergent behaviors, which tend to not be correct. http://stackoverflow.com/questions/8943022/reactor-stops-between-tests-when-using-twisted-trial-unittest/8947354#8947354 explains how to avoid this with fakes * Mocks tend to encourage monkey patching, instead of just passing objects as parameters Again: https://www.youtube.com/watch?v=Xu5EhKVZdV8 is an amazing resource. This obviously adds a bit of a burden to development of client libraries, so there needs to be a good justification. Here are the advantages I see: Just as FYI, nova currently has a different way of addressing the two points that a fake API server would address. * Helps flesh out the API: by having a simple implementation it helps in designing a good API. Having a simple API implementation will not help us with having a good API, as by the time its in code almost too late to change (the window to change is between coded and released). Instead a fake API would help validate that the API follows the specs. Nova has two ways of doing this now, unit tests that exercise the APIs, and compare the results against recorded output (see doc/api_samples/ in nova). And we have tempest API tests. * Helps other projects tests: right now any project which uses an openstack client library has to do something manual in their tests, either add their own abstraction layer where they hand write an in-memory implementation, or they just monkey patch the socket, http, or client library to not make request. Either direction requires a bunch of work from each and every project using an openstack client. Having these in the core client libraries would allow downstream authors to simply swap out Connection classes. Instead of a fake API server with no scheduler, no db etc, nova has a fake backend. So the only code path that is different then a real deployment is which virt driver is used (https://review.openstack.org/#/c/24938/). I think these benefits out weigh the disadvantages. I'm not sure what the procedure for this is going forward. I think to demonstrate this concept it should start with a few (or even just one) client libraries, particularly ones which completely own the resources they serve (e.g. swift, marconi, ceilometer, trove), as compared to ones that interact more (e.g. neutrino, cinder, and nova). This is absolutely something I'm volunteering to work on, but I want to ensure this is an idea that has general buy in from the community and existing maintainers, so it doesn't wither. Thanks, Alex -- I disapprove of what you say, but I will defend to the death your right to say it. -- Evelyn Beatrice Hall (summarizing Voltaire) The people's good is the highest law. -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for including fake implementations in python-client packages
On Tue, Jul 2, 2013 at 11:38 PM, Robert Collins robe...@robertcollins.net wrote: Radix points out I missed the naunce that you're targeting the users of python-novaclient, for instance, rather than python-novaclient's own tests. On 3 July 2013 16:29, Robert Collins robe...@robertcollins.net wrote: What I'd like is for each client library, in addition to the actual implementation, is that they ship a fake, in-memory, version of the API. The fake implementations should take the same arguments, have the same return values, raise the same exceptions, and otherwise be identical, besides the fact that they are entirely in memory and never make network requests. So, +1 on shipping a fake reference copy of the API. -1 on shipping it in the client. The server that defines the API should have two implementations - the production one, and a testing fake. The server tests should exercise *both* code paths [e.g. using testscenarios] to ensure there is no skew between them. Then the client tests can be fast and efficient but not subject to implementation skew between fake and prod implementations. Back on Launchpad I designed a similar thing, but with language neutrality as a goal : https://dev.launchpad.net/ArchitectureGuide/ServicesRequirements#Test_fake And in fact, I think that that design would work well here, because we have multiple language bindings - Python, Ruby, PHP, Java, Go etc, and all of them will benefit from a low(ms or less)-latency test fake. So taking the aspect I missed into account I'm much happier with the idea of shipping a fake in the client, but... AFAICT many of our client behaviours are only well defined in the presence of a server anyhow. So it seems to me that a fast server fake can be used in tests of python-novaclient, *and* in tests of code using python-novaclient (including for instance, heat itself), and we get to write it just once per server, rather than once per server per language binding. -Rob I want to make sure I understond you. Let's say I have a program named cool-cloud-tool, and it uses python-novaclient, python-keystoneclient, and three other clients for OpenStack services. You're suggesting that its test suite should start up instances of all those OpenStack services with in-memory or otherwise localized backends, and communicate with them using standard python-*client functionality? I can imagine that being a useful thing, if it's very easy to do, and won't increase my test execution time too much. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal for including fake implementations in python-client packages
On Tue, Jul 2, 2013 at 11:10 AM, Kurt Griffiths kurt.griffi...@rackspace.com wrote: The idea has merit; my main concern is that we would be duplicating significant chunks of code/logic between the fakes and the real services. How can we do this in a DRY way? I've done it a few different ways for libraries I've worked on. Usually, the fakes don't actually duplicate much code from the real implementation. But in the cases they do, I've had situations like this: class RealImplementation(object): def do_network_stuff(self, stuff): ... def low_level_operation(self): return self.do_network_stuff(GET /integer) def high_level_operation(self): return self.low_level_operation() + 5 I'd just create a subclass like this: class FakeImplementation(RealImplementation): def do_network_stuff(self, stuff): raise NotImplementedError(This should never be called!) def low_level_operation(self): return self.integer # or however you implement your fake This has two interesting properties: 1. I don't have to reimplement the high_level_operation 2. If I forget to implement a fake version of some method that invokes do_network_stuff, then it will blow up with a NotImplementedError so my test doesn't accidentally do real network stuff. This is just an example from some recent work I did on a simple RPC client with an HTTP API (unrelated to OpenStack client libraries), but that just so happens to be the case that Alex is discussing, so I think it can work well. -- IRC: radix Christopher Armstrong Rackspace ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
