Hi George, Your example looks good for me and I think everybody agreed that we should organize testing to avoid running all tests for each update: if you fix bug in 'net' module you don't have to run tests say for 'awt' module but if you update 'luni' then you have to run tests for all modules.
Thanks, Stepan. On 4/11/06, George Harley wrote: > > Stepan Mishura wrote: > > On 4/11/06, Mark Hindess wrote: > > > > > >> Personally, obviously, I'd expect people to run the tests before > >> committing. > >> > >> However, I notice that since enabling the security tests - which fork > >> for every test - that the tests take over half an hour to run now on > >> our Linux build machine. So I can see why enthusiasm might lead to > >> people not running all the tests but instead perhaps just running > >> those from the module you are changing. > >> > > > > > > I think that this message[1] relates to the problem with running big > test > > suites. So should we do the similar thing i.e. define which test suites > > should be run for each module? > > > > Thanks, > > Stepan. > > > > [1] > > > http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/[EMAIL > PROTECTED] > > > > > > > > Hi Stepan, > > We should both define and then automate which test suites get run for > changes made to a given module. The dependency information is already > there for us in the contents of each module's MANIFEST.MF file. So, to > take a recent example which we are all familiar with ;-) , we can see > that the java.text package exported from the text module is imported by > modules logging, LUNI, security and SQL. > > Folks wishing to verify that changes made to a given module do not break > other modules should ideally be able to kick off a JUnit run that > incorporates all of the tests from the other modules into one test > suite. This could either be folded into the existing "run.tests" target > of the module's build.xml or else be a separate target like > "run.depends.tests" (or similar). > i.e. something like ... > > <target name="run.depends.tests"> > > <mkdir dir="${hy.tests.reports}" /> > > <junit fork="yes" > forkmode="once" > printsummary="withOutAndErr" > errorproperty="test.error" > showoutput="on" > dir="${hy.text.bin.test}" > jvm="${hy.target}/jre/bin/java"> > > <jvmarg value="-showversion"/> > > <!-- Required by various tests that set security manager etc --> > <jvmarg > value="- > Djava.security.policy=../../../../support/src/test/resources/config/testing.policy > " > /> > > <!-- Required for running the java.net unit tests --> > <jvmarg > value="- > Dtest.ini.file=../../../../support/src/test/resources/config/localhosttest.ini > " > /> > > <env key="JAVA_HOME" value="${hy.target}/jre"/> > > <!-- Adding all dependent test bins to the classpath --> > <classpath> > <pathelement path="${hy.text.bin.test}"/> > <pathelement path="${hy.logging.bin.test}"/> > <pathelement path="${hy.luni.bin.test}"/> > <pathelement path="${hy.security.bin.test}"/> > <pathelement path="${hy.sql.bin.test}"/> > <pathelement path="${hy.tests.support.bin}" /> > </classpath> > > <!-- The DependencyTests suite logically groups the other dependent test > suites --> > <test name="tests.text.DependencyTests" > todir="${hy.tests.reports}" > haltonfailure="no" > > <formatter type="xml"/> > </test> > </junit> > </target> > > > In this scenario the class tests.text.DependencyTests would be a simple > logical grouping of the dependent test suites using standard JUnit > techniques. > i.e. something like ... > > public class DependencyTests { > public static Test suite() { > TestSuite suite = new TestSuite("Dependents on java.text module"); > suite.addTestSuite(org.apache.harmony.logging.AllTests.suite()); > suite.addTestSuite(org.apache.harmony.luni.AllTests.suite()); > suite.addTestSuite(org.apache.harmony.security.AllTests.suite()); > suite.addTestSuite(org.apache.harmony.sql.AllTests.suite()); > suite.addTestSuite(org.apache.harmony.logging.AllTests.suite()); > return suite; > } > } > > ...and each of the AllTests classes referred to were just a logical > grouping of the test cases in that particular module. > > > I think that such a scheme will be a tremendous convenience to > developers as it will increase our confidence that breakages introduced > into other modules by changes in a particular module will be caught soon > without recourse to running the entire test suite - just the necessary > ones. > > If we see merit in setting up something like this to run alongside the > existing test targets then I will be happy to open the JIRA and create > the new Ant targets and test suites. > > Best regards, > George > > > > >> George just commented in a commit message backing out some changes in > >> text that all the text tests passed even though the changed caused > >> failures elsewhere. > >> > >> Regards, > >> Mark. > >> > >> On 4/11/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote: > >> > >>> Just curious (and this isn't a criticism - I'm just as guilty of not > >>> doing this)... > >>> > >>> Don't you run the tests before committing? > >>> > >>> geir > >>> > >>> George Harley wrote: > >>> > >>>> Hi, > >>>> > >>>> It *seems* like things started failing after I committed the changes > >>>> > >> for > >> > >>>> HARMONY-205 last night. I'm looking into this now. If the > >>>> > >> investigation > >> > >>>> begins to take up too much time I will back the changes out. > >>>> > >>>> Best regards, > >>>> George > >>>> > >>>> > >>>> Stepan Mishura wrote: > >>>> > >>>>> The same for me + DatagramChannelTest > >>>>> > >>>>> Thanks, > >>>>> Stepan. > >>>>> > >>>>> On 4/11/06, Mark Hindess wrote: > >>>>> > >>>>> > >>>>>> No. These: > >>>>>> > >>>>>> F > >>>>>> > >>>>>> > >> > org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder > >> > >>>>>> E > >>>>>> > >>>>>> > >> > org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01 > >> > >>>>>> E > >>>>>> > >>>>>> > >> > org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02 > >> > >>>>>> F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt > >>>>>> E > >>>>>> > >>>>>> > >> > org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode > >> > >>>>>> F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid > >>>>>> F java.security.cert.X509CertSelectorTest.testMatch > >>>>>> F java.security.cert.X509CertSelectorTest.testClone > >>>>>> F tests.api.java.sql.DateTest.testSetTimelong > >>>>>> F tests.api.java.sql.DateTest.testToString > >>>>>> F tests.api.java.sql.DateTest.testValueOf > >>>>>> F tests.api.java.sql.TimestampTest.testSetNanosint > >>>>>> F tests.api.java.sql.TimestampTest.testToString > >>>>>> F tests.api.java.util.DateTest.test_toGMTString > >>>>>> F tests.api.java.util.DateTest.test_toString > >>>>>> > >>>>>> Regards, > >>>>>> Mark. > >>>>>> > >>>>>> On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote: > >>>>>> > >>>>>> > >>>>>>> On 4/11/06, Mark Hindess wrote: > >>>>>>> > >>>>>>> > >>>>>>>> Yes. I was using the failureproperty mechanism. Trying to get > >>>>>>>> > >> this > >> > >>>>>>>> property propogated back to the top level ant file was what I was > >>>>>>>> having trouble with. > >>>>>>>> > >>>>>>>> Using a file as you suggest might help. I'll give that a try > >>>>>>>> > >>>>>>>> > >>>>>> shortly... > >>>>>> > >>>>>> > >>>>>>>> Incidentally, I'm seeing 12 failures and 3 errors on r393111. > >>>>>>>> > >>>>>>>> > >>>>>>> I guess that you have 9 tests from DatagramChannelTest passed. > And > >>>>>>> 12 + > >>>>>>> > >>>>>>> > >>>>>> 3 = > >>>>>> > >>>>>> > >>>>>>> 15 :-) > >>>>>>> > >>>>>>> (And > >>>>>>> > >>>>>>> > >>>>>>>> there are typos - "mathc" should be "match" - in the failure > >>>>>>>> > >> messages > >> > >>>>>>>> for java.security.cert testMatch and testClone.) > >>>>>>>> > >>>>>>>> > >>>>>>> I've fixed typo. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Stepan. > >>>>>>> > >>>>>>> Regards, > >>>>>>> > >>>>>>> > >>>>>>>> Mark. > >>>>>>>> > >>>>>>>> On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>>> On 4/11/06, Mark Hindess wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Stepan, > >>>>>>>>>> > >>>>>>>>>> I have another build running (but without notifications going > to > >>>>>>>>>> > >>>>>>>>>> > >>>>>> the > >>>>>> > >>>>>> > >>>>>>>>>> list) that runs: > >>>>>>>>>> > >>>>>>>>>> 1) build (with reference jdk) > >>>>>>>>>> 2) build with what we created with 1) > >>>>>>>>>> 3) test > >>>>>>>>>> 4) create classlists and compare with class load data for > >>>>>>>>>> > >>>>>>>>>> > >>>>>> applications > >>>>>> > >>>>>> > >>>>>>>>>> (tomcat, geronimo, continuum, etc.) > >>>>>>>>>> 5) generate JAPI results > >>>>>>>>>> > >>>>>>>>>> I'd like to fail this build at step 3, but I can't see an easy > >>>>>>>>>> > >> way > >> > >>>>>> to > >>>>>> > >>>>>> > >>>>>>>>>> get 'ant -f make/build.xml test' to run all tests and then fail > >>>>>>>>>> > >> if > >> > >>>>>> any > >>>>>> > >>>>>> > >>>>>>>>>> of the module test sub-targets had test failures. I could > parse > >>>>>>>>>> > >>>>>>>>>> > >>>>>> the > >>>>>> > >>>>>> > >>>>>>>>>> output I suppose, but I'd rather get ant to propagate the junit > >>>>>>>>>> > >>>>>>>>>> > >>>>>> tasks > >>>>>> > >>>>>> > >>>>>>>>>> failure property back up to the top level. I've tried a couple > >>>>>>>>>> > >> of > >> > >>>>>>>>>> things and none seem to work. Any suggestions welcome. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> Mark, did you try failureproperty parameter for junit task? > >>>>>>>>> We may add it to ant sub-targets to raise a flag, for example, > >>>>>>>>> > >>>>>>>>> > >>>>>> create > >>>>>> > >>>>>> > >>>>>>>> file " > >>>>>>>> > >>>>>>>> > >>>>>>>>> TESTS.FAILED" in the root, when tests for some module fail. And > >>>>>>>>> > >> in > >> > >>>>>> the > >>>>>> > >>>>>> > >>>>>>>> end > >>>>>>>> > >>>>>>>> > >>>>>>>>> of tests suite run check whether this file exists on not. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Stepan. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Mark. > >>>>>>>>>> > >>>>>>>>>> On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> I've checked out at morning last updates, built the code base > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>> and > >>>>>> > >>>>>> > >>>>>>>> run > >>>>>>>> > >>>>>>>> > >>>>>>>>>> the > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> tests …and there are 24 tests failures! > >>>>>>>>>>> > >>>>>>>>>>> There are 9 tests failures in > >>>>>>>>>>> > org.apache.harmony.tests.java.nio.channels.DatagramChannelTest– > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>> I > >>>>>> > >>>>>> > >>>>>>>> saw > >>>>>>>> > >>>>>>>> > >>>>>>>>>> these > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> failures before from time to time. It seems that tests depend > >>>>>>>>>>> > >> on > >> > >>>>>>>> some > >>>>>>>> > >>>>>>>> > >>>>>>>>>> race > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> conditions because they may pass if I rerun them. Paulex, are > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>> these > >>>>>> > >>>>>> > >>>>>>>>>> tests > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> passing for you? > >>>>>>>>>>> > >>>>>>>>>>> And there are new 15 test failures. So now if I'll make a > code > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> update > >>>>>>>> > >>>>>>>> > >>>>>>>>>> or a > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> bug fix how I can be 100% sure that I don't do any regression? > >>>>>>>>>>> > >>>>>>>>>>> Currently if a commit breaks the Harmony classlib build a > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> notification > >>>>>>>> > >>>>>>>> > >>>>>>>>>> with > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> subject: "[continuum] BUILD FAILURE: Classlib/linux.ia32" will > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>> be > >>>>>> > >>>>>> > >>>>>>>> send > >>>>>>>> > >>>>>>>> > >>>>>>>>>> to > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> harmony-commits mailing list. Is it possible to have the same > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>> for > >>>>>> > >>>>>> > >>>>>>>> tests? > >>>>>>>> > >>>>>>>> > >>>>>>>>>> I > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> mean that after completing automatic build the existing > >>>>>>>>>>> > >> classlib > >> > >>>>>>>> tests > >>>>>>>> > >>>>>>>> > >>>>>>>>>> suite > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> should be run. If there are failing tests then a report > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>> notification > >>>>>> > >>>>>> > >>>>>>>>>> with > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> corresponding subject will be send. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Stepan Mishura > >>>>>>>>>>> Intel Middleware Products Division > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Mark Hindess <[EMAIL PROTECTED]> > >>>>>>>>>> IBM Java Technology Centre, UK. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >> --------------------------------------------------------------------- > >> > >>>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>>>>> To unsubscribe, e-mail: > >>>>>>>>>> > >>>>>>>>>> > >>>>>> [EMAIL PROTECTED] > >>>>>> > >>>>>> > >>>>>>>>>> For additional commands, e-mail: > >>>>>>>>>> > >>>>>>>>>> > >>>>>> [EMAIL PROTECTED] > >>>>>> > >>>>>> > >>>>>>>>> -- > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >> --------------------------------------------------------------------- > >> > >>>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>>>> To unsubscribe, e-mail: > >>>>>>>>> > >> [EMAIL PROTECTED] > >> > >>>>>>>>> For additional commands, e-mail: > >>>>>>>>> > >>>>>>>>> > >>>>>> [EMAIL PROTECTED] > >>>>>> > >>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Stepan Mishura > >>>>>>>>> Intel Middleware Products Division > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> -- > >>>>>>>> Mark Hindess <[EMAIL PROTECTED]> > >>>>>>>> IBM Java Technology Centre, UK. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >> --------------------------------------------------------------------- > >> > >>>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>>> To unsubscribe, e-mail: > >>>>>>>> > >> [EMAIL PROTECTED] > >> > >>>>>>>> For additional commands, e-mail: > >>>>>>>> > >> [EMAIL PROTECTED] > >> > >>>>>>>> > >>>>>>>> > >>>>>>> -- > >>>>>>> > >>>>>>> > >> --------------------------------------------------------------------- > >> > >>>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>>> To unsubscribe, e-mail: > >>>>>>> > >> [EMAIL PROTECTED] > >> > >>>>>>> For additional commands, e-mail: > >>>>>>> > >> [EMAIL PROTECTED] > >> > >>>>>>> Thanks, > >>>>>>> Stepan Mishura > >>>>>>> Intel Middleware Products Division > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> -- > >>>>>> Mark Hindess <[EMAIL PROTECTED]> > >>>>>> IBM Java Technology Centre, UK. > >>>>>> > >>>>>> > >>>>>> > >> --------------------------------------------------------------------- > >> > >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>> To unsubscribe, e-mail: > [EMAIL PROTECTED] > >>>>>> For additional commands, e-mail: > >>>>>> > >> [EMAIL PROTECTED] > >> > >>>>>> > >>>>>> > >>>>> -- > >>>>> > --------------------------------------------------------------------- > >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>>> For additional commands, e-mail: > >>>>> > >> [EMAIL PROTECTED] > >> > >>>>> Thanks, > >>>>> Stepan Mishura > >>>>> Intel Middleware Products Division > >>>>> > >>>>> > >>>>> > >>>> --------------------------------------------------------------------- > >>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>> For additional commands, e-mail: > [EMAIL PROTECTED] > >>>> > >>>> > >>>> > >>> --------------------------------------------------------------------- > >>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>> For additional commands, e-mail: [EMAIL PROTECTED] > >>> > >>> > >>> > >> -- > >> Mark Hindess <[EMAIL PROTECTED]> > >> IBM Java Technology Centre, UK. > >> > >> --------------------------------------------------------------------- > >> Terms of use : http://incubator.apache.org/harmony/mailing.html > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: [EMAIL PROTECTED] > >> > >> > >> > > > > > > -- > > --------------------------------------------------------------------- > > Terms of use : http://incubator.apache.org/harmony/mailing.html > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > Thanks, > > Stepan Mishura > > Intel Middleware Products Division > > > > > > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Thanks, Stepan Mishura Intel Middleware Products Division
