On 8/2/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote:
Vladimir Strigun wrote: > Our project is now in a good shape and we started encouraging people > to try real applications on top of Harmony. One of the important parts > of the feedback will be general impression of stability, reliability > and performance of the HDK. We are doing pretty good in fixing bugs > and developing new functionality in HDK, and it might be a good time > for us to start thinking of performance as well. It might be a little > bit preliminary, but do we consider having any performance targets for > us? If yes, we may need to focus at least some of our efforts on > benchmarking and tuning overall performance of HDK. Well, I wouldn't say the HDK, as much as Harmony. > > One of the main questions here is what should be the targets for us > and how should we measure our performance. There are several ways for > measuring performance, such us commercial benchmarks, free benchmarks, > application startup, small micro suites, etc. Some of free benchmarks > have been mentioned in JIRA issues and dev list, nevertheless at the > moment we don't have any goals for performance. In spite of > application enabling initiative it might be good to consider publicly > available benchmarks as the additional list of the software > applications which we would enable on Harmony. Yep. > > So I suggest to start discussing performance techniques and methods > that can be used for comparing performance between RI and HDK. I think s/HDK/Harmony > in case we do not consider performance issues, we can get negative > feedback from users even if application starts without any errors, > exceptions, etc. > > One of the benchmarks that was mentioned is DaCapo[1]. It's a free > open-source benchmark suite and I believe it can be used for regular > performance measurement of HDK. I've tried to find other free suites > and got the following list: > > Telco - this one mostly stresses BigInteger/BigDecimal functionality > GcOld - the purpose of this one is clear from the name :) > SciMark - java benchmark for scientific and numerical computing > Linpack java - well-known benchmark solving linear equations The > Plasma Benchmark - creates an animated display by continuously summing > four sine waves in an applet > JavaWorld Benchmark - benchmark for low-level operations: loops, > accessing variables, method invocation, arithmetic operators, casting, > instantiation, exception handling, thread creation and switching. > CaffeineMark 3.0 - low-level benchmark suite, including sieve of > Eratosthenes, sorting, logic ops, method invocation, floating point, > simple graphics and GUI ops > JavaGrande benchmark suite - a set of benchmarks stressing different > areas of java. These are good. I it would be nice to just hook them into the build-test framework as optional parts.
I'll try to hook them into build-test framework. Do you think that all of them should be added to the framework? I suppose benchmarks should be downloaded the same way as other dependencies in Harmony?
> > Having in mind that the list of publicly available benchmarks is not > too big, sometimes it will be necessary to create micro benches for > some of patches (for instance, Harmony-935). IMO micro should be > started in case we change some code that the bench covers. Sure > > Other interesting and possibly more productive way for comparing > performance between different implementations are to use non-free > benchmarks. For instance, we can use benchmarks from Spec[n], like > SpecJVM, SpecJBB, SpecJAppserver. Unfortunately first we should get > license for it, but I believe this issue can be solved within the help > of companies participating in Harmony :) Maybe. What we want to have is those benchmarks available to anyone in the project, although we'll take what we can get for now... geir > > Thoughts? Comments? > > [1] http://dacapobench.org > [2] http://www.spec.org > > -- > Vladimir Strigun, > Intel Middleware Products Division > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
