On Jun 19, 2011, at 8:36 AM, Chuck Remes wrote: > 2. Number of runs (EC2 charges) > At this time I have it configured to do two runs for Jruby and two for > Rubinius on each commit to their respective repositories. The JRuby runs are > for '--client' and '--server' while for Rubinius they are '-Xint' and > JIT-enabled. Based on the number of benchmarks, each commit takes around 1 > hour to complete. Since EC2 charges by the instance hour consumed, this may > get expensive. > > Couple that with the fact that *every* commit causes a run and suddenly we > see a bunch of runs queue up due to documentation changes, fixing typos, etc. > Perhaps the commits need to be filtered so that those that do not contain any > source file changes (*.java, *.rb, *.hpp, *.cpp, etc) will be ignored. > > Overall, I'm concerned that this benchmark server is going to be expensive. > > > 5. Github post-receive hook
I suppose I should have written that we don't *need* to benchmark every commit. Like the pypy project, this could be a cron job that runs once per day and benchmarks the latest HEAD. That would certainly eliminate the concern over EC2 costs. cr -- --- !ruby/object:MailingList name: rubinius-dev view: http://groups.google.com/group/rubinius-dev?hl=en post: [email protected] unsubscribe: [email protected]
