Thanks Matt. These are great! --- Ryan Ausanka-Crues CEO Palomino Labs, Inc. [email protected] (m) 805.242.2486
On Jun 21, 2012, at 12:36 PM, Matt Corgan wrote: > These are geared more towards development than regression testing, but here > are a few ideas that I would find useful: > > * Ability to run the performance tests (or at least a subset of them) on a > development machine would help people avoid committing regressions and > would speed development in general > * Ability to test a single region without heavier weight servers and > clusters > * Letting the test run with multiple combinations of input parameters > (block size, compression, blooms, encoding, flush size, etc, etc). > Possibly many combinations that could take a while to run > * Output results to a CSV file that's importable to a spreadsheet for > sorting/filtering/charting. > * Email the CSV file to the user notifying them the tests have finished. > * Getting fancier: ability to specify a list of branches or tags from git > or subversion as inputs, which would allow the developer to tag many > different performance changes and later figure out which combination is the > best (all before submitting a patch) > > > On Thu, Jun 21, 2012 at 12:13 PM, Elliott Clark <[email protected]>wrote: > >> I actually think that more measurements are needed than just per release. >> The best I could hope for would be a four node+ cluster(One master and >> three slaves) that for every check in on trunk run multiple different perf >> tests. >> >> >> - All Reads (Scans) >> - Large Writes (Should test compactions/flushes) >> - Read Dominated with 10% writes >> >> Then every checkin can be evaluated and large regressions can be treated as >> bugs. And with that we can see the difference between the different >> versions as well. http://arewefastyet.com/ is kind of the model that I >> would love to see. And I'm more than willing to help where ever needed. >> >> However in reality every night will probably be more feasible. And Four >> nodes is probably not going to happen either. >> >> On Thu, Jun 21, 2012 at 11:38 AM, Andrew Purtell <[email protected] >>> wrote: >> >>> On Wed, Jun 20, 2012 at 10:37 PM, Ryan Ausanka-Crues >>> <[email protected]> wrote: >>>> I think it makes sense to start by defining the goals for the >>> performance testing project and then deciding what we'd like to >> accomplish. >>> As such, I start by soliciting ideas from everyone on what they would >> like >>> to see from the project. We can then collate those thoughts and >> prioritize >>> the different features. Does that sound like a reasonable approach? >>> >>> In terms of defining a goal, the fundamental need I see for us as a >>> project is to quantify performance from one release to the next, thus >>> be able to avoid regressions by noting adverse changes in release >>> candidates. >>> >>> In terms of defining what "performance" means... well, that's an >>> involved and separate discussion I think. >>> >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet >>> Hein (via Tom White) >>> >>
