(Cross-posted to JRuby ML)
I am nearly done with building out the benchmarking server for JRuby and
Rubinius. I am using the benchmarks from the Rubinius project
(rubinius/benchmark/core/*). There are around 90 benchmark files each with a
varying number of tests, so each run produces 522 reports right now.
I have several questions and concerns that I'll detail here. We can discuss
these either here or on irc; I have no preference.
Issues...
1. Benchmark names
The name given in the benchmark (x.report("this is the name")) acts as the
database key for the codespeed server. For continuity, we do *not* want these
names changing once they are in the system otherwise old results will become
decoupled from new results.
e.g. core/array/bench_append.rb, "append array" would be a different report
from "Append array"
So, I'd like to "lock down" the benchmark files so the names on old benchmarks
don't get changed without an extra sign-off.
Additionally, some of the names are a tad long (though very descriptive). The
codespeed server wants to limit names to 30 chars (I modified the source to get
beyond this) but we probably do want to cap the length at something reasonable
like 100 chars. I pulled that out of the air. The length limit increases
readability on the codespeed website.
2. Number of runs (EC2 charges)
At this time I have it configured to do two runs for Jruby and two for Rubinius
on each commit to their respective repositories. The JRuby runs are for
'--client' and '--server' while for Rubinius they are '-Xint' and JIT-enabled.
Based on the number of benchmarks, each commit takes around 1 hour to complete.
Since EC2 charges by the instance hour consumed, this may get expensive.
Couple that with the fact that *every* commit causes a run and suddenly we see
a bunch of runs queue up due to documentation changes, fixing typos, etc.
Perhaps the commits need to be filtered so that those that do not contain any
source file changes (*.java, *.rb, *.hpp, *.cpp, etc) will be ignored.
Overall, I'm concerned that this benchmark server is going to be expensive.
3. Benchmark repository
I think the benchmarks need to be moved out from under the Rubinius repository
and into a separate one. Also, they each need to be modified to work
out-of-the-box with the benchmark_suite gem (they don't work with it now). I
already know what changes need to be made, so this is relatively simple.
The concern becomes project ownership of these benchmarks. Should it be under
the JRuby or Rubinius organizations? What happens when we add MRI or Maglev or
IronRuby or ?? to the list of supported runtimes?
I suggest that the benchmarks be spun off to a new "ruby community"
organization. This change will ensure more open access to runtimes that are not
part of the Engine Yard ecosystem (if that's a concern to anyone) while also
providing a semblance of impartiality.
4. Database & webserver performance
I need to get back onto my "day job" now that this works. I don't have the time
or skills to replace sqlite3 with a more enterprisey DB nor do I know how to
reconfigure django to use apache/nginx/etc instead of the default python-based
web server.
So, if this site is popular and gets lots of hits, it may fall over.
I'd like to put a general call out to the Ruby community to ask for some
volunteer(s) to step in and do this optimization. It's probably okay to wait
and see if the site actually does get crushed before spending any more time on
it.
5. Github post-receive hook
The code that runs the benchmarks looks at commits in two ways. Upon startup it
updates each repository and looks up the last 25 commits. It enqueues the
commits it hasn't seen yet (stored in a small sqlite db). After this
initialization step, it starts listening for commits that are published from
github.
So we'll need each project to add a post-receive hook that points at this
server. It's super easy to do this through the admin control panel. Both
projects probably already have these setup for the CI servers.
The code I wrote to handle all of this bookkeeping will be pushed to github
early this week. The project name is currently 'benchmark_pipeline' so if you
hate the name please speak up now. The code ain't perfect but it is a decent
foundation to work from. With a little refactoring it could probably be adopted
by any project that wanted to setup their own codespeed server to track
performance.
cr
--
--- !ruby/object:MailingList
name: rubinius-dev
view: http://groups.google.com/group/rubinius-dev?hl=en
post: [email protected]
unsubscribe: [email protected]
