Dedicated and isolated infrastructure is a must if we want consistent performance numbers. If we can come up with a reasonable plan, I'd be happy to ask for resources. Even with dedicated infrastructure we would still have to keep in mind that it's a data point from a single provider that hopefully highlights a general trend about performance.
Here is a list of focus points as I see them so far: - Dedicated hardware is a requirement in order to achieve somewhat consistent results - Tight loop micro benchmarks - Tests highlighting the performance cases we care about - The ability to determine a sane control - The ability to tests proposed patches, compare them to the control, and leave comments on reviews - Reproducible setup and test runner so that others can run these against a dedicated performance environment - Daily snapshots of performance published publicly (nice to have) On Fri, Jun 3, 2016 at 3:16 PM, Brant Knudson <b...@acm.org> wrote: > > > On Fri, Jun 3, 2016 at 2:35 PM, Lance Bragstad <lbrags...@gmail.com> > wrote: > >> Hey all, >> >> I have been curious about impact of providing performance feedback as >> part of the review process. From what I understand, keystone used to have a >> performance job that would run against proposed patches (I've only heard >> about it so someone else will have to keep me honest about its timeframe), >> but it sounds like it wasn't valued. >> >> > We had a job running rally for a year (I think) that nobody ever looked at > so we decided it was a waste and stopped running it. > > >> I think revisiting this topic is valuable, but it raises a series of >> questions. >> >> Initially it probably only makes sense to test a reasonable set of >> defaults. What do we want these defaults to be? Should they be determined >> by DevStack, openstack-ansible, or something else? >> >> > A performance test is going to depend on the environment (the machines, > disks, network, etc), the existing data (tokens, revocations, users, etc.), > and the config (fernet, uuid, caching, etc.). If these aren't consistent > between runs then the results are not going to be usable. (This is the > problem with running rally on infra hardware.) If the data isn't realistic > (1000s of tokens, etc.) then the results are going to be at best not useful > or at worst misleading. > > What does the performance test criteria look like and where does it live? >> Does it just consist of running tempest? >> >> > I don't think tempest is going to give us numbers that we're looking for > for performance. I've seen a few scripts and have my own for testing > performance of token validation, token creation, user creation, etc. which > I think will do the exact tests we want and we can get the results > formatted however we like. > > From a contributor and reviewer perspective, it would be nice to have the >> ability to compare performance results across patch sets. I understand that >> keeping all performance results for every patch for an extended period of >> time is unrealistic. Maybe we take a daily performance snapshot against >> master and use that to map performance patterns over time? >> >> > Where are you planning to store the results? > > >> Have any other projects implemented a similar workflow? >> >> I'm open to suggestions and discussions because I can't imagine there >> aren't other folks out there interested in this type of pre-merge data >> points. >> >> > Thanks! >> >> Lance >> >> > Since the performance numbers are going to be very dependent on the > machines I think the only way this is going to work is if somebody's > willing to set up dedicated hardware to run the tests on. If you're doing > that then set it up to mimic how you deploy keystone, deploy the patch > under test, run the performance tests, and report the results. I'd be fine > with something like this commenting on keystone changes. None of this has > to involve openstack infra. Gerrit has a REST API to get the current > patches. > > Everyone that's got performance requirements should do the same. Maybe I > can get the group I'm in to try it sometime. > > - Brant > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev