benchmarks are statistics.
statistics are numbers.
numbers can be made to lie.
I make no judgement of the validity of Raja's numbers - I only
wish to supply the grain of salt that I feel must accompany any
benchmarks for them to be meaningfull. (and to prevent the Ziff-
Davis reporters on the list from coming away with an easy "OSS
can't perform, Linux can't perform unless a corporate entity
makes it" style piece)
At 13:52 10/12/99 -0400, Raja Vallee-Rai wrote:
>Hello,
>
>We have formally evaluated the different virtual machines available
>for Linux and thought it would be worthwhile to share the results with
>the Linux community.
Note that there is a difference between a virtual machine and a JIT.
You say you're looking at one, but your numbers and presentation say
different.
>The following tests were conducted on an unloaded dual processor
>Pentium II/400mhz running Debian GNU/Linux (kernel 2.2.8). Each
how much RAM? Swap?
>benchmark execution was repeated ten times. We discarded the maximum
>and minimum results, and averaged the remaining 8 execution times.
very good methodology... sure wish more people would do that.
>base(s): time in seconds to run under blackdown jdk 1.2, pre-release
>2, with jit.
>
>sunint: speedup (base time/this time) of the blackdown jdk1.2,
>pre-release 2, with no jit.
>
>borjit: speedup of blackdown jdk1.2, pre-release 2, with the Borland
>jit installed (http://www.borland.com)
>
>ibmjit: speedup of the AlphaWorks IBM 1.1.8 JIT
>(http://www.alphaworks.ibm.com)
this is the fish out of water. The others are 1.2 JVM/JDKs, this is a
1.1 JVM and JDK. It really isn't a fair comparison.
>A # indicates that the run failed validity checks.
poor bor...er...inprise....
> base(s) sunint borjit ibmjit
> check .84 - 1.33 - 1.25 # 1.75 -
> compress 65.61 - .15 - 1.07 - 2.42 -
> db 148.43 - .57 - .98 - 2.98 -
> jack 64.50 - .43 - 1.35 - 3.65 -
> javac 75.67 - .54 - 1.21 - 2.51 -
> jess 50.86 - .47 - 1.44 - 2.67 -
> mpegaudio 54.61 - .15 - 1.19 # 2.32 -
> mtrt 40.32 - .41 - 1.78 - 2.79 -
> raytrace 55.56 - .45 - 1.92 - 3.04 -
> sablecc-w 42.57 - .58 - 1.06 - 2.32 -
> soot-j 132.93 - .69 - 1.25 - 2.26 -
^^^^^^^ This is in seconds?
with benchmarks this short the memory/swap size and vm settings for
heap size and management become fairly important.
your focus seems to be jits more than jvms... in which case the base and
sunint should be swapped, such that the "speedup" values are the actual
speed improvement provided by each JIT. Similarly the IBMJITC should either
not be involved until it works with blackdown *and* is on Java2. (in other
words the only variable between columns should be the object being compared,
the JIT.) If you want to include the IBMJITC then it should also have an
independent baseline present and the speedup should be relative to that.
This would lead to a comparison between IBM 1.1 and Blackdown 2 which again
leads back to the statement that there should only be one variable between
columns... so wait till IBM has Java2 for Linux, or move back to Blackdown
1.1.
>To everyone working on Java for Linux: keep up the great work! :)
Agreed! The members of the Blackdown effort continue to show incredible
progress and achivement for such a small volunteer development group.
-=Chris
cabbey at home dot net <*> http://members.home.net/cabbey
I want a binary interface to the brain!
Today's opto-mechanical digital interfaces are just too slow!
----------------------------------------------------------------------
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
