Rana, To cover most of the performance regressions ideally we might have three types of tests:
1. High-level benchmarks (like SPECs or DaCapo benchmarks) - cover significant performance issues or issues that are not related to a particular optimization 2. Performance regression tests or micro-benchmarks - a bytecode-based tests covering one optimization or a small group of them 3. Low-level tests (like VM or JIT unit tests) - tests for particular code transformations. To my mind the second type of tests is the most sensitive to the environment. In many cases it's difficult to create such test in a reasonable amount of time. I admit that this type of tests might be a short-term solution to check if a fix has been integrated properly but in the long-term we need also 1 and 3 from the list. Thanks, Pavel On 9/14/06, Rana Dasgupta <[EMAIL PROTECTED]> wrote:
Hi Pavel, Platform specific optimizations can be accomodated in the scheme described by doing a cpuid check in the test and automatically passing it or disabling on all other platforms. That shouldn't be too hard. I understand that some jit optimizations are deeper and more abstract, but ultimately the value of the optimization cannot just be the morphing of an IR, and the gain cannot be invisible to the user, or the regression undetectable. If it needs to be part of a sequence to be effective, the scenario in the test needs to be set up accordingly. It is a little uncomfortable if a framework does some magic and then comes back and says "everything is OK". Sorry to sound difficult. Thanks, Rana > On 9/14/06, Pavel Ozhdikhin <[EMAIL PROTECTED]> wrote: > > > > Hello Rana, > > > > When I think of an optimization which gives 1% improvement on some > > simple > > workload or 3% improvement on EM64T platforms only I doubt this can be > > easily detected with a general-purpose test suite. IMO the performance > > regression testing should have a specialized framework and a stable > > environment which guarantees no user application can spoil the results. > > > > The right solution might also be a JIT testing framework which would > > understand the JIT IRs and check if some code patterns have been > > optimized > > as expected. Such way we can guarantee necessary optimizations are done > > independently of the user environment. > > > > Thanks, > > Pavel > > > > > > > > > >