On Tue, Apr 20, 2010 at 2:56 PM, Jeremy Kemper <[email protected]> wrote: > Hey Federico, > > Have you tried the performance test suite built in to Rails apps? > > You write performance tests in the style of integration tests then > `rake test:benchmark` or `rake test:profile`. > > The benchmark task benches each performance test and saves the results > in a CSV file with info about your environment, the git revision, etc. > > So you can do simple CI with a git commit hook or a cron job, then > check the CSV for regressions and improvements. > > I think improving the tools here would make a great project, as would > sharing results from open-source apps tested on different platforms > and hardware. > > Good luck! > jeremy
Hi Jeremy, Yes, good points. I've already talked with Federico offline. I believe his original interest was in improving the rails test suite itself, and allowing collaboration and submission of results by users. I think it is mostly orthogonal to the problem I'm tackling - which is to have a performant, stable, easily reproducible "production" CI environment which runs the main build script (http://github.com/rails/rails/blob/master/ci/ci_build.rb) against the major interpreters. I don't want to make the "production" CI environment any more complicated than necessary - and adding the variable of user-submitted input from random environments would definitely complicate things. It's hard enough to get most people to care about the failing CI at all - we need to keep it simple, stable, and reproducible. That's why I'm taking the approach of a Rails CI AMI which anyone can run on EC2, and the generation of that AMI will also be completely automated. That hopefully eliminates any possibility of "non-reproducible" build failures, which are the bane of any CI system. I've said "reproducible" four times, so you get my point :) However, as long as any improvements/innovations in this area (rake tasks, additional/alternative test/benchmark scripts to ci_build.rb, etc) conform to basic conventions - stdout/stderr, generated csv/html flat file artifacts, and proper return codes - there's no reason we can't integrate it into the main CI environment (or any other CI environment) in the future. Even if it isn't integrated, the information gathered can drive improvements to the main test suite or build script. However, I'd prefer avoid calling this work "CI" or lumping it into the "CI" effort, for the reasons mentioned above, and also because it would be more "user driven" than "continuous". In fact, I've already updated the wiki to say "Crowdsourced Rails environment compatibility metrics" instead of continuous integration. Thanks, -- Chad -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rubyonrails-core?hl=en.
