On Tue, 19 Mar 2002, Dalibor Topic wrote: > Speaking of benchmarking suites, there is Ashes from the SableVM people. > http://www.sable.mcgill.ca/ashes/ > I think it's open source, so I'd prefer that. :)
Okay, thanks, I'll take a look at that. There are also some open source conformance test-suites, I believe. > I like the idea, but that only gives us a rough estimate about performance, > not conformance. I think that running a performance benchmark on a > web/cvs/ftp/build server under load will produce rough results most of the > time. Good catch; no, obiviously you can't run the benchmark on a machine being used for something else at the time and expect to get reasonable results. However, since you'll have to run the benchmark on different architectures anyway, you might as well run them on separate machines that are idle overnight or the like. Also, it does give you a rough idea on conformance if the benchmarks break too badly :) I was thinking more in the terms of "breaking some optimization". The problem with optimizations is that they're quite platform specific and sometimes longer to run than simple conformance/ regression tests, so they're difficult for a single developer to run. In addition Kaffe has pretty good conformance, but it would appear performance could use some work. -Jukka Santala
