On the 0x227 day of Apache Harmony Alexey Varlamov wrote: > Folks, > > I've just did a little step in improving pre-commit testing for DRLVM; > most important change is a move from "perTest" forking mode to "once" > (aka sameVM mode). > This reduces testing time drastically (~50%), but may introduce some > extra instability (like new intermittent failures or timeouts). > Nevertheless I believe we are ready to accept this challenge and it is > important to take it earlier. > Other enhancements, are described below; hopefully will make us more > productive. Have fun!
Although, that was a kind of surprize... :) I like the changes because: * we do not have to spend 50% of pre-commit testing on loading/compiling same bootstrap many times. * a good opportunity opened to make DRLVM more stable with relatively small tests That makes me think we have testing time spent more effectively. +1 to have fun > > 2006/11/20, [EMAIL PROTECTED] <[EMAIL PROTECTED]>: > > Author: varlax > > Date: Mon Nov 20 03:27:39 2006 > > New Revision: 477149 > > > > URL: http://svn.apache.org/viewvc?view=rev&rev=477149 > > Log: > > Refactored kernel.test.xml, made it more "ant-minded" and added some > > features: > > [+] "kernel.test.forkmode" support, "once" by default; > > [+] single "test.case" support; > > [+] "kernel.test.failfast" behaviour support; > > [!] Improved results reporting; > > [+] easy customization via property file; > > > > Some intermittent failures appeared on Win32 due to forking only "once", > > but we must meet them sooner or later - why not now... > > > > Added: > > harmony/enhanced/drlvm/trunk/build/make/test.properties > > Modified: > > harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml > > > > Modified: harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml > > URL: > > http://svn.apache.org/viewvc/harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml?view=diff&rev=477149&r1=477148&r2=477149 > > ============================================================================== > > --- harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml > > (original) > > +++ harmony/enhanced/drlvm/trunk/build/make/targets/kernel.test.xml Mon Nov > > 20 03:27:39 2006 > > @@ -1,17 +1,18 @@ > > <!-- > > - Copyright 2006 The Apache Software Foundation or its licensors, as > > applicable. > > - > > - Licensed under the Apache License, Version 2.0 (the "License"); > > - you may not use this file except in compliance with the License. > > - You may obtain a copy of the License at > > - > > - http://www.apache.org/licenses/LICENSE-2.0 > > - > > - Unless required by applicable law or agreed to in writing, software > > - distributed under the License is distributed on an "AS IS" BASIS, > > - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > > implied. > > - See the License for the specific language governing permissions and > > - limitations under the License. > > + Licensed to the Apache Software Foundation (ASF) under one or more > > + contributor license agreements. See the NOTICE file distributed with > > + this work for additional information regarding copyright ownership. > > + The ASF licenses this file to You under the Apache License, Version > > 2.0 > > + (the "License"); you may not use this file except in compliance with > > + the License. You may obtain a copy of the License at > > + > > + http://www.apache.org/licenses/LICENSE-2.0 > > + > > + Unless required by applicable law or agreed to in writing, software > > + distributed under the License is distributed on an "AS IS" BASIS, > > + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > > implied. > > + See the License for the specific language governing permissions and > > + limitations under the License. > > --> > > > > <project name="KERNEL TESTS"> > > @@ -20,7 +21,9 @@ > > <property name="kernel.test.dir" > > location="${build.semi.dir}/kernel.tests"/> > > <property name="test.resource.path" > > location="${kernel.test.dir}/resources"/> > > <property name="test.class.path" location="${kernel.test.dir}/classes"/> > > - > > + > > + <property name="test.jvm.exe" value="${build.deploy.dir}/bin/java"/> > > + <property file="test.properties"/> > > > > <patternset id="kernel.test.java.pattern"> > > <include name="org/apache/harmony/**/*Test.java" /> > > @@ -36,68 +39,58 @@ > > <exclude name="java/lang/RuntimeAdditionalTest40.java" /> > > <exclude name="java/lang/RuntimeAdditionalTest41.java" /> > > <exclude name="java/lang/RuntimeAdditionalTest42.java" /> > > - > > - <!-- This test fails --> > > - <!--exclude name="java/lang/ThreadTest.java" /--> > > </patternset> > > - > > - <property name="vm.use_verifier" value="1" /> > > - <property name="kernel.test.timeout" value="360000" /> > > > > - <target name="kernel.test" depends="-pre-kernel-test"> > > - <property name="test.mode" value="jet+opt+int" /> > > - <if> > > - <contains string="${test.mode}" substring="batch" /> > > - <then> > > - <antcallback target="-run-kernel-test-batch" > > - return="batch.status"> > > - </antcallback> > > - </then> > > - </if> > > - <if> > > - <contains string="${test.mode}" substring="jet" /> > > - <then> > > - <antcallback target="-run-kernel-test" > > - return="jit.jet.status"> > > - <param name="jit.or.interpreter" value="jitrino.jet" /> > > - <param name="int.or.jit.arg" > > value="-Dem.properties=jet" /> > > - </antcallback> > > - </then> > > - </if> > > - <if> > > - <contains string="${test.mode}" substring="opt" /> > > - <then> > > - <antcallback target="-run-kernel-test" > > - return="jit.opt.status"> > > - <param name="jit.or.interpreter" value="jitrino.opt" /> > > - <param name="int.or.jit.arg" > > value="-Dem.properties=opt" /> > > - </antcallback> > > - </then> > > - </if> > > - <if> > > - <contains string="${test.mode}" substring="int" /> > > - <then> > > - <antcallback target="-run-kernel-test" > > - return="interpreter.status"> > > - <param name="jit.or.interpreter" value="interpreter" /> > > - <param name="int.or.jit.arg" value="-Xint" /> > > - </antcallback> > > - </then> > > - </if> > > - <if> > > - <or> > > - <contains string="${jit.jet.status}" substring="FAILED" /> > > - <contains string="${jit.opt.status}" substring="FAILED" /> > > - <contains string="${interpreter.status}" > > substring="FAILED" /> > > - <contains string="${batch.status}" substring="FAILED" /> > > - </or> > > - <then> > > - <fail message="Kernel tests FAILED" /> > > - </then> > > - </if> > > - </target> > > + <target name="kernel.test" > > + depends="compile-kernel-test, run-kernel-test, report-kernel-test, > > check-kernel-test"/> > > > > - <target name="compile-kernel-tests" depends=""> > > + <target name="run-kernel-test" depends="-prepare-kernel-test"> > > + <condition property="kernel.test.continue" value="true" > > else="false"> > > + <isfalse value="${kernel.test.failfast}"/> > > + </condition> > > + <for list="${kernel.test.mode}" param="mode" > > + trim="true" keepgoing="${kernel.test.continue}"> > > + <sequential> > > + <antcall target="-run-kernel-test-batch"> > > + <param name="kernel.mode" value="@{mode}" /> > > + </antcall> > > + </sequential> > > + </for> > > + </target> > > + > > + <target name="report-kernel-test"> > > + <for list="${kernel.test.mode}" param="mode" > > + trim="true" keepgoing="true"> > > + <sequential> > > + <junitreport > > todir="${kernel.test.dir}/reports/@{mode}.mode"> > > + <fileset dir="${kernel.test.dir}/reports/@{mode}.mode"> > > + <include name="*.xml" /> > > + </fileset> > > + <report format="frames" > > + > > todir="${kernel.test.dir}/reports/@{mode}.mode/html" /> > > + </junitreport> > > + </sequential> > > + </for> > > + </target> > > + > > + <target name="check-kernel-test"> > > + <loadfile property="kernel.test.summary" > > + srcFile="${kernel.test.dir}/reports/summary"/> > > + <echo message=" ***************************${line.separator}"/> > > + <echo message=" *** Kernel tests status ***${line.separator}"/> > > + <echo message=" ***************************${line.separator}"/> > > + <echo message="${line.separator}"/> > > + <echo message="${kernel.test.summary}"/> > > + <echo message="${line.separator}"/> > > + <echo>Please find detailed results under > > ${kernel.test.dir}/reports directory.</echo> > > + > > + <condition property="kernel.failures"> > > + <contains string="${kernel.test.summary}" substring="FAILED" /> > > + </condition> > > + <fail if="kernel.failures" message="There were test failures." /> > > + </target> > > + > > + <target name="compile-kernel-test"> > > <mkdir dir="${test.resource.path}" /> > > <javac srcdir="${kernel.test.javasrc}/../kernel_resources" > > destdir="${test.resource.path}" > > @@ -111,177 +104,61 @@ > > <delete quiet="true" dir="${test.class.path}/notfound" /> > > </target> > > > > - <target name="-pre-kernel-test" depends="compile-kernel-tests"> > > + <target name="-prepare-kernel-test"> > > + <delete quiet="true" dir="${kernel.test.dir}/reports" /> > > + <mkdir dir="${kernel.test.dir}/reports" /> > > + > > <condition property="junit.jar" value="${junit.home}/junit.jar"> > > <isset property="junit.home" /> > > </condition> > > - <property name="junit.jar" > > location="${external.dep.CLASSLIB}/depends/jars/junit_3.8.2/junit.jar"/> > > - <if> > > - <not> > > - <available file="${junit.jar}"/> > > - </not> > > - <then> > > - <echo> > > + <property name="junit.jar" > > + > > location="${external.dep.CLASSLIB}/depends/jars/junit_3.8.2/junit.jar"/> > > + <available file="${junit.jar}" property="junit.found"/> > > + <fail unless="junit.found"> > > ============================================== > > Please set the classpath of junit as follows: > > build.bat -Djunit.jar=%JUNIT_HOME% test > > ============================================== > > - </echo> > > - <fail message="Kernel tests misconfigured" /> > > - </then> > > - </if> > > - > > - <delete quiet="true" dir="${kernel.test.dir}/reports" /> > > - <delete quiet="true" dir="${kernel.test.dir}/ref-reports" /> > > - <mkdir dir="${kernel.test.dir}/reports" /> > > - <mkdir dir="${kernel.test.dir}/ref-reports" /> > > - > > + </fail> > > </target> > > > > <target name="-run-kernel-test-batch"> > > + <propertycopy property="kernel.mode.name" > > from="${kernel.mode}.name" override="on"/> > > + <propertycopy property="kernel.mode.switch" > > from="${kernel.mode}.switch" override="on"/> > > + <echo> > > + ================================== > > + Run kernel tests using ${kernel.mode.name} > > + ================================== > > + </echo> > > <property name="report.dir" > > - location="${kernel.test.dir}/reports/batch.mode"/> > > + location="${kernel.test.dir}/reports/${kernel.mode}.mode"/> > > <mkdir dir="${report.dir}" /> > > > > - <junit fork="yes" forkmode="once" timeout="${kernel.test.timeout}" > > - haltonfailure="no" failureproperty="batch.failed" > > - filtertrace="no" printsummary="on" > > - jvm="${build.deploy.dir}/bin/java"> > > - > > - <env key="JAVA_HOME" value="${build.deploy.dir}"/> > > + <junit fork="yes" failureproperty="kernel.failed" > > + filtertrace="no" printsummary="on" showoutput="off" > > + haltonfailure="${kernel.test.failfast}" > > + forkmode="${kernel.test.forkmode}" > > + timeout="${kernel.test.timeout}" > > + jvm="${test.jvm.exe}"> > > > > - <jvmarg value="-Dtest.resource.path=${test.resource.path}"/> > > <jvmarg value="-Dvm.assert_dialog=0" /> > > + <jvmarg value="-Dtest.resource.path=${test.resource.path}"/> > > + <jvmarg value="${kernel.mode.switch}" /> > > <jvmarg > > value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}" /> > > - <formatter type="brief"/> > > - <batchtest todir="${report.dir}"> > > + <formatter type="xml"/> > > + <batchtest todir="${report.dir}" unless="test.case"> > > <fileset dir="${kernel.test.javasrc}"> > > <patternset refid="kernel.test.java.pattern"/> > > </fileset> > > </batchtest> > > + <test name="${test.case}" todir="${report.dir}" if="test.case" > > /> > > </junit> > > > > - <condition property="batch.status" value="FAILED" else="PASSED"> > > - <isset property="batch.failed" /> > > + <condition property="kernel.status" value="FAILED" else="PASSED"> > > + <isset property="kernel.failed" /> > > </condition> > > - </target> > > - > > - > > - <target name="-run-kernel-test"> > > - <echo> > > - ================================== > > - Run kernel tests using ${jit.or.interpreter} > > - ================================== > > - </echo> > > - > > - <property name="report.dir" > > - location="${kernel.test.dir}/reports/${jit.or.interpreter}"/> > > - <mkdir dir="${report.dir}" /> > > - > > - <for param="test"> > > - <fileset dir="${kernel.test.javasrc}"> > > - <patternset refid="kernel.test.java.pattern" /> > > - </fileset> > > - <sequential> > > - > > - <basename property="last.dir" > > file="${kernel.test.javasrc}" /> > > - <propertyregex property="class.name" override="true" > > input="@{test}" regexp="(.)+${last.dir}\${file.separator}" replace="" > > defaultValue="@{test}" /> > > - <propertyregex property="class.name" override="true" > > input="${class.name}" regexp="\.java" replace="" > > defaultValue="${class.name}" /> > > - <propertyregex property="class.name" override="true" > > input="${class.name}" regexp="\${file.separator}" replace="." global="true" > > defaultValue="${class.name}" /> > > - <echo message="RUNNING : ${class.name}" /> > > - > > - <junit fork="yes" failureproperty="@{test}.failed" > > - timeout="${kernel.test.timeout}" newenvironment="yes" > > filtertrace="no" > > - jvm="${build.deploy.dir}/bin/java"> > > - > > - <env key="JAVA_HOME" value="${build.deploy.dir}"/> > > - > > - <!--env key="LD_LIBRARY_PATH" > > value="${build.deploy.dir}/bin:${env.LD_LIBRARY_PATH}" /--> > > - <env key="SYSTEMDRIVE" value="${env.SystemDrive}" /> > > - <env key="HOME" value="${env.HOME}" /> > > - <env key="PATH" value="${env.PATH}" /> > > - <jvmarg > > value="-Dtest.resource.path=${test.resource.path}"/> > > - <jvmarg > > value="-Demma.coverage.out.file=${emma.coverage.out}" /> > > - <jvmarg value="-Demma.coverage.out.merge=true" /> > > - <jvmarg > > value="-Xbootclasspath/p:${emma.coverage.ip}${path.separator}${emma.jar}" /> > > - <jvmarg value="-Dvm.use_verifier=${vm.use_verifier}" /> > > - <jvmarg value="-Dvm.assert_dialog=0" /> > > - <jvmarg > > value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}" /> > > - <jvmarg value="${int.or.jit.arg}" /> > > - <jvmarg value="-ea:java.lang.reflect..." /> > > - <formatter type="brief"/> > > - <formatter type="xml" /> > > - <test name="${class.name}" todir="${report.dir}"/> > > - </junit> > > - <condition property="${jit.or.interpreter}.status" > > value="FAILED"> > > - <isset property="@{test}.failed" /> > > - </condition> > > - > > - <!-- > > - Gregory > > - It appears that some kernel unit tests for drlvm classes > > are > > - implementation specific and may fail on RI. So it makes no > > - sense to check tests on RI. Also running test on RI > > - right after it failed on drlvm creates wrong test run > > report. > > - > > - I am commenting this block for now in case someone wants it > > - back, possibly under some condition. My POV is that this > > - block should be removed eventually. > > - <if> > > - <or> > > - <isset property="@{test}.failed" /> > > - <isset property="kernel.test.reference" /> > > - </or> > > - <then> > > - <junit fork="yes" > > failureproperty="@{test}.ref.failed" > > - timeout="${kernel.test.timeout}" > > - newenvironment="yes" filtertrace="no"> > > - > > - > > - <env key="JAVA_HOME" > > value="${build.deploy.dir}"/> > > - > > - <!- -env key="LD_LIBRARY_PATH" > > value="${build.deploy.dir}/bin:${env.LD_LIBRARY_PATH}" /- -> > > - <env key="SYSTEMDRIVE" > > value="${env.SystemDrive}" /> > > - <env key="HOME" value="${env.HOME}" /> > > - <env key="PATH" value="${env.PATH}" /> > > - <jvmarg > > value="-Dtest.resource.path=${test.resource.path}"/> > > - <jvmarg > > value="-Xbootclasspath/a:${junit.jar}${path.separator}${test.class.path}" /> > > - <formatter type="brief"/> > > - <formatter type="xml" /> > > - <test name="${class.name}" > > todir="${kernel.test.dir}/ref-reports"/> > > - </junit> > > - > > - <if> > > - <isset property="@{test}.ref.failed" /> > > - <then> > > - <echo message=" **** FAILED on reference > > JRE ****" /> > > - </then> > > - </if> > > - </then> > > - </if> > > - /--> > > - </sequential> > > - </for> > > - > > - <junitreport todir="${report.dir}"> > > - <fileset dir="${report.dir}"> > > - <include name="TEST-*.xml" /> > > - </fileset> > > - <report format="frames" todir="${report.dir}/html" /> > > - </junitreport> > > - <junitreport todir="${kernel.test.dir}/ref-reports"> > > - <fileset dir="${kernel.test.dir}/ref-reports"> > > - <include name="TEST-*.xml" /> > > - </fileset> > > - <report format="frames" > > todir="${kernel.test.dir}/ref-reports/html" /> > > - </junitreport> > > - <property name="${jit.or.interpreter}.status" value="PASSED"/> > > - <propertycopy name="status" from="${jit.or.interpreter}.status"/> > > - <echo message="" /> > > - <echo message="==============================================" /> > > - <echo message="Kernel tests ${status} using ${jit.or.interpreter}. > > Please find the detailed results here:" /> > > - <echo > > message="${report.dir}${file.separator}html${file.separator}index.html" /> > > - <echo message="==============================================" /> > > > > + <concat append="on" destfile="${kernel.test.dir}/reports/summary" > > + > ${kernel.mode.name} > > ${kernel.status}${line.separator}</concat> > > </target> > > </project> > > > > Added: harmony/enhanced/drlvm/trunk/build/make/test.properties > > URL: > > http://svn.apache.org/viewvc/harmony/enhanced/drlvm/trunk/build/make/test.properties?view=auto&rev=477149 > > ============================================================================== > > --- harmony/enhanced/drlvm/trunk/build/make/test.properties (added) > > +++ harmony/enhanced/drlvm/trunk/build/make/test.properties Mon Nov 20 > > 03:27:39 2006 > > @@ -0,0 +1,19 @@ > > +kernel.test.mode=jet,opt,int > > +kernel.test.failfast=off > > +kernel.test.forkmode=once > > +kernel.test.timeout=360000 > > + > > + > > +jit.name=Client mode JIT (default) > > +jit.switch=-Dbogus > > + > > +jet.name=jitrino.JET > > +jet.switch=-Xem:jet > > + > > +opt.name=jitrino.OPT > > +opt.switch=-Xem:opt > > + > > +int.name=interpreter > > +int.switch=-Xint > > + > > + > > > > > > > -- Egor Pasko
