On Jun 19, 3:36 pm, Chuck Remes <[email protected]> wrote:
> 1. Benchmark names
> The name given in the benchmark (x.report("this is the name")) acts as the
> database key for the codespeed server. For continuity, we do *not* want these
> names changing once they are in the system otherwise old results will become
> decoupled from new results.
>
> e.g. core/array/bench_append.rb, "append array" would be a different report
> from "Append array"
>
> So, I'd like to "lock down" the benchmark files so the names on old
> benchmarks don't get changed without an extra sign-off.
>
> Additionally, some of the names are a tad long (though very descriptive). The
> codespeed server wants to limit names to 30 chars (I modified the source to
> get beyond this) but we probably do want to cap the length at something
> reasonable like 100 chars. I pulled that out of the air. The length limit
> increases readability on the codespeed website.
Could we use the syntax tree of do end block in x.report("name")
do ... end instead of its name ? Something along these lines:
Digest::MD5.hexdigest("proc {|i| i.to_i}".to_sexp.to_s)
will generate a usable key in the db. If the benchmark code changes
the id will change but that's good, because it's not the same
benchmark anymore, right (Renaming a variable will change the sexp
signature and the id but I guess this is less likely to happen
(renaming the variable for the sake of renaming it vs. renaming a
report) ? We'll be able to rename benchmarks (eg. if there are many
more benchmarks in the future, we get similar names and want to name
them slightly differently and won't be limited by the number of chars
etc.)
>
> 2. Number of runs (EC2 charges)
> At this time I have it configured to do two runs for Jruby and two for
> Rubinius on each commit to their respective repositories. The JRuby runs are
> for '--client' and '--server' while for Rubinius they are '-Xint' and
> JIT-enabled. Based on the number of benchmarks, each commit takes around 1
> hour to complete. Since EC2 charges by the instance hour consumed, this may
> get expensive.
>
> Couple that with the fact that *every* commit causes a run and suddenly we
> see a bunch of runs queue up due to documentation changes, fixing typos, etc.
> Perhaps the commits need to be filtered so that those that do not contain any
> source file changes (*.java, *.rb, *.hpp, *.cpp, etc) will be ignored.
>
> Overall, I'm concerned that this benchmark server is going to be expensive.
>
If the cost is going to be high for an EC2 instance and we're planning
things like post-commit hooks maybe it's worth to consider a dedicated
box ?
Prices in Europe start from 29 EUR/month.
Thanks
--
Karol Hosiawa
--
--- !ruby/object:MailingList
name: rubinius-dev
view: http://groups.google.com/group/rubinius-dev?hl=en
post: [email protected]
unsubscribe: [email protected]
