Sergey Nesterov wrote:
I had tested SPECjbb2005 on Harmony and submitted the results for reviewing
to SPEC.
SPEC committee has some questions on my submission concerning Harmony
support and testing of command-line options.
We had this discussion already on the private list!?
I believe that SPEC only want to publish results for code that people
can and would run their applications on, not experimental or custom builds.
However, it would seem that SPEC are not used to dealing with open
source projects on the same level as commercial implementations, so
their evaluation criteria seem a bit strange.
1) Support of Harmony:
The issue is section (1) of the SPECjbb2005 Run Rules [1], citing:
"The implementation is generally available, documented and supported
by the providing vendor."
Though the support requirement for SPECjbb2005 is loosely defined, all
current publications are on Java implementations that provide various
levels of support. SPEC argues that this level of support cannot be compared
to forum support provided by Harmony developers. SPEC argues that current
production JVMs on the market today are fully supported by the companies
that develop them and they need to clarify these issues before any result
using Harmony can be accepted.
What can you comment on this?
Our code is certainly generally available (more so than commercial
builds), is documented (including design documentation on our website),
and is supported by the Apache community who participate here.
However, it seems to me that SPEC want somewhere people can go and hand
over money to support a particular build of the runtime. They want to
pay for somebody to fix the code in preference to the ability to fix the
code themselves through open source. I'm not sure why exchanging money
is the key point for them.
There are people who will accept payment to 'support' Harmony by fixing
bugs. Independent consultants, like Andrew, will take paid work on
Harmony. Of course, their support terms are varied, like everyone else
supporting a product.
2) Testing:
SPEC also concerned about testing required for performance
optimizations to be considered as generally available, broad and
suggested for production use. They want to know what checks are in
place to prevent unsafe changes that benefit SPECjbb2005 only?
This is a joke right? If SPEC try to tell you that commercial vendors
do not test and optimize their code to benefit the SPEC scores
specifically then they are deluding you.
We have no special command-line options that can only be used in running
the SPEC benchmarks. The command-line options that are typically used
to get good performance runs are also valid (though not all appropriate)
for running users' applications.
I will be glad to see the input on these questions from Harmony developers.
SPEC run the risk of devaluing their benchmark and results if they are
going to be selective about the body of people who contribute to the
runtime's development. I seriously doubt that potential adopters
evaluate support levels based on the run rules.
Other performance benchmark suites are available...
Regards,
Tim
Thanks,
Sergey Nesterov,
CS Department of MSU
[1] http://www.spec.org/jbb2005/docs/RunRules.html
