I'm concerned that spawning an EC2 instance for each run may result in a lot of noise in the emerged benchmarks. I've found that the hardware that acquires each instance can differ noticeably in characteristics and shared load from run to run, even within the same AZ. Every now and then "bad," underperforming, instance launches may raise spurious alarm bells. On a large operation across multiple instances, we always expect something like a 20% difference between the first instances to report a result and the last ones ... and are never surprised by a few failures.
The noise would be less for a single long-lived instance, or best with a dedicated piece of hardware with no competing loads. Not that I have one I could reliably volunteer ... but for this very specific mission, it seems like run-to-run improvement or degradation only really has meaning if as many other variables as we can manage are taken away. Maybe some Ruby activist has a nice old server spinning away in their data center that feels lonely because all the fun has gone to the cloud? Of course, I could be fretting over nothing -- this could be in the "try it and see" category. On Sun, Jun 19, 2011 at 10:19 AM, Chuck Remes <[email protected]>wrote: > I suppose I should have written that we don't *need* to benchmark every > commit. Like the pypy project, this could be a cron job that runs once per > day and benchmarks the latest HEAD. That would certainly eliminate the > concern over EC2 costs. > -- --- !ruby/object:MailingList name: rubinius-dev view: http://groups.google.com/group/rubinius-dev?hl=en post: [email protected] unsubscribe: [email protected]
