Hey Ajo,
Amazing work man! :D (I showed some of your graphs :P). As I told you
last week,
the AWS approach might work beautifully for the benchmarking. The AWS scripts
allow you to
bring up the instances on demand, even big instances with high IO rsources, run
the benchmarks
and kill them. Remote reboots are very easy an efortless. You can also test it
in several OS.
To some extend, the AWS approach is not perfect but it's close. A lot
of startups are using AWS infraestructure
for their startup and, as spoken with Stefan ;) most startups don't need a Sun
disk array to operate, so
IO performance in a high IO instance is as good as you can get with your own
data center metal :)
Alex
>
> Problems I found:
>
> * Client operating system, or just the client itself could be a
> problem: crashing the operating system, exceding limits, or... abperf,
> for example doesn't make use of keep alive.
>
> * Server machine sometimes crash with some servers ... (mostly
> because of the operating system...), we should find an stable
> architecture or at least, a way to reboot the machine, or
> maybe it isn't a problem if we're testing only cherokee.
>
> * The VM cpu/tick approach, that could be the cleanest, probably
> wouldn't be that realistic, because we're discarding IO times, that
> are also a part where servers can be faster (when they produce less
> IO).
>
> * if we don't go using an "exactly constrained" VM approach,
> then... the day we change the testing machines most tests will change.
> (we could live with it..)
>
>
>
> Miguel Angel Ajo Pelayo
> http://www.nbee.es
> +34 636 52 25 69
> skype: ajoajoajo
>
>
> 2010/5/10 Alvaro Lopez Ortega <[email protected]>
>
> > On 09/05/2010, at 22:24, Miguel Angel wrote:
> > > Here it's the benchmarking tool I worked on a "few" months ago.
> > >
> > > http://code.google.com/p/easybench/
> > >
> > > It could be used to track perfomance changes between differents
> > > versions of cherokee, which could be quite useful (it would need
> > > something like measuring cpu ticks in a virtual machine, as Stephan
> > > sugested).
> > >
> > > As can be readen in the project page, it's made of two part, the
> > > server, and the client (I tested it on standalone machines, and most
> > > of my problems came from operating system limits on the client machine
> > > -wow-).
> > >
> > > These are the settings for building/running/testing different servers
> > > from sources:
> > > http://code.google.com/p/easybench/source/browse/trunk/server-deployme
> > > nt/settings.py
> > >
> > > And those are the settings for the tested servers:
> > > http://code.google.com/p/easybench/source/browse/#svn/trunk/server-depl
> > > oyment/conf_templates/cherokee-0.99.20%3Fstate%3Dclosed
> > >
> > > It was prepared to get information about requests, failed requests and
> > > memory usage on server, but I think the last version (dumping the
> > > final results) was lost from my laptop disk crash... , I'll try to
> > > check the backups.
> >
> > Thanks a million of the effort, and for letting us know about it on the
> > Cherokee Summit.
> >
> > I'd love to figure how we could integrate this in some sort of building
> > system so Cherokee could be automatically tested for each new release.
> >
> > --
> > Octality
> > http://www.octality.com/
--
Founder @ Inkzee.com
http://www.twitter.com/abarrera
http://www.twitter.com/inkzee
--
http://www.neurosecurity.com
"We must be the change we wish to see in the world"
Mahatma Gandhi
_______________________________________________
Cherokee mailing list
[email protected]
http://lists.octality.com/listinfo/cherokee
