Hi Gavin! Comments inline... On Mon, Jan 16, 2012 at 10:18 AM, Brebner, Gavin <[email protected]> wrote: > I’m interested in running some “white-box” tests that check scalability and > limits of parts of a Nova system. I want to start from a > full working configuration however, as I get that from my dev team and it > includes all sorts of settings that would be tedious > and error prone to reproduce in other ways. My thought was that good use > could be made of some of the “fake” drivers > that are present – I’m hoping to be able to make some minimal code/config > changes to swap the driver on my host node(s) > for a fake one, restart the services and run tests.
Yes, this is an idea I've been playing around with as well. In performance testing, if you cannot isolate (to the best of your ability) other components in a multi-component system, it is virtually impossible to get trustworthy data. Everything boils down to the scientific principle of "all other things being equal". In other words, if I'm trying to produce regression test results for a particular component (say, Glance) from one revision to another, I cannot do so if the other components in the test system are changing or the system load itself is in flux. Fakes are useful when you are running repeated tests trying to identify when a particular patch or revision either improves or degrades performance of the system as a whole. By using fakes (which either toss data to /dev/null or store persistent data in an in-memory store), you can dramatically reduce the overall time to run a particular test. By reducing the time taken for a test, you can test multiple successive patches or revisions more easily, knowing that the only thing you are changing is the implementation of a particular component. > Any caveats on this that I should be aware of ? For that matter has anyone > tried this kind of thing before ? I’d be interested in hearing of your > experiences if so. There are a bunch of faked out drivers for various things in the core projects. But there's not really any consistency to them, AFAICT. What would be useful is to get a detailed description from you of the exact questions you are trying to answer. That would enable us to determine which fakes would be useful and/or need to be written in order to isolate your environment enough to retrieve trustworthy data from test results. Cheers! -jay _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : [email protected] Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
