Stepan Mishura wrote:
On 4/11/06, Mark Hindess wrote:
Personally, obviously, I'd expect people to run the tests before
committing.
However, I notice that since enabling the security tests - which fork
for every test - that the tests take over half an hour to run now on
our Linux build machine. So I can see why enthusiasm might lead to
people not running all the tests but instead perhaps just running
those from the module you are changing.
I think that this message[1] relates to the problem with running big test
suites. So should we do the similar thing i.e. define which test suites
should be run for each module?
Thanks,
Stepan.
[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200603.mbox/[EMAIL
PROTECTED]
Hi Stepan,
We should both define and then automate which test suites get run for
changes made to a given module. The dependency information is already
there for us in the contents of each module's MANIFEST.MF file. So, to
take a recent example which we are all familiar with ;-) , we can see
that the java.text package exported from the text module is imported by
modules logging, LUNI, security and SQL.
Folks wishing to verify that changes made to a given module do not break
other modules should ideally be able to kick off a JUnit run that
incorporates all of the tests from the other modules into one test
suite. This could either be folded into the existing "run.tests" target
of the module's build.xml or else be a separate target like
"run.depends.tests" (or similar).
i.e. something like ...
<target name="run.depends.tests">
<mkdir dir="${hy.tests.reports}" />
<junit fork="yes"
forkmode="once"
printsummary="withOutAndErr"
errorproperty="test.error"
showoutput="on"
dir="${hy.text.bin.test}"
jvm="${hy.target}/jre/bin/java">
<jvmarg value="-showversion"/>
<!-- Required by various tests that set security manager etc -->
<jvmarg
value="-Djava.security.policy=../../../../support/src/test/resources/config/testing.policy"
/>
<!-- Required for running the java.net unit tests -->
<jvmarg
value="-Dtest.ini.file=../../../../support/src/test/resources/config/localhosttest.ini"
/>
<env key="JAVA_HOME" value="${hy.target}/jre"/>
<!-- Adding all dependent test bins to the classpath -->
<classpath>
<pathelement path="${hy.text.bin.test}"/>
<pathelement path="${hy.logging.bin.test}"/>
<pathelement path="${hy.luni.bin.test}"/>
<pathelement path="${hy.security.bin.test}"/>
<pathelement path="${hy.sql.bin.test}"/>
<pathelement path="${hy.tests.support.bin}" />
</classpath>
<!-- The DependencyTests suite logically groups the other dependent test
suites -->
<test name="tests.text.DependencyTests"
todir="${hy.tests.reports}"
haltonfailure="no" >
<formatter type="xml"/>
</test>
</junit>
</target>
In this scenario the class tests.text.DependencyTests would be a simple
logical grouping of the dependent test suites using standard JUnit
techniques.
i.e. something like ...
public class DependencyTests {
public static Test suite() {
TestSuite suite = new TestSuite("Dependents on java.text module");
suite.addTestSuite(org.apache.harmony.logging.AllTests.suite());
suite.addTestSuite(org.apache.harmony.luni.AllTests.suite());
suite.addTestSuite(org.apache.harmony.security.AllTests.suite());
suite.addTestSuite(org.apache.harmony.sql.AllTests.suite());
suite.addTestSuite(org.apache.harmony.logging.AllTests.suite());
return suite;
}
}
...and each of the AllTests classes referred to were just a logical
grouping of the test cases in that particular module.
I think that such a scheme will be a tremendous convenience to
developers as it will increase our confidence that breakages introduced
into other modules by changes in a particular module will be caught soon
without recourse to running the entire test suite - just the necessary
ones.
If we see merit in setting up something like this to run alongside the
existing test targets then I will be happy to open the JIRA and create
the new Ant targets and test suites.
Best regards,
George
George just commented in a commit message backing out some changes in
text that all the text tests passed even though the changed caused
failures elsewhere.
Regards,
Mark.
On 4/11/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote:
Just curious (and this isn't a criticism - I'm just as guilty of not
doing this)...
Don't you run the tests before committing?
geir
George Harley wrote:
Hi,
It *seems* like things started failing after I committed the changes
for
HARMONY-205 last night. I'm looking into this now. If the
investigation
begins to take up too much time I will back the changes out.
Best regards,
George
Stepan Mishura wrote:
The same for me + DatagramChannelTest
Thanks,
Stepan.
On 4/11/06, Mark Hindess wrote:
No. These:
F
org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoder
E
org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder01
E
org.apache.harmony.security.asn1.der.DerGeneralizedTimeEDTest.testGeneralizedEncoderDecoder02
F org.apache.harmony.security.asn1.der.DerUTCTimeEDTest.testMt
E
org.apache.harmony.security.x509.PrivateKeyUsagePeriodTest.testEncodeDecode
F java.security.cert.X509CertSelectorTest.testSetPrivateKeyValid
F java.security.cert.X509CertSelectorTest.testMatch
F java.security.cert.X509CertSelectorTest.testClone
F tests.api.java.sql.DateTest.testSetTimelong
F tests.api.java.sql.DateTest.testToString
F tests.api.java.sql.DateTest.testValueOf
F tests.api.java.sql.TimestampTest.testSetNanosint
F tests.api.java.sql.TimestampTest.testToString
F tests.api.java.util.DateTest.test_toGMTString
F tests.api.java.util.DateTest.test_toString
Regards,
Mark.
On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote:
On 4/11/06, Mark Hindess wrote:
Yes. I was using the failureproperty mechanism. Trying to get
this
property propogated back to the top level ant file was what I was
having trouble with.
Using a file as you suggest might help. I'll give that a try
shortly...
Incidentally, I'm seeing 12 failures and 3 errors on r393111.
I guess that you have 9 tests from DatagramChannelTest passed. And
12 +
3 =
15 :-)
(And
there are typos - "mathc" should be "match" - in the failure
messages
for java.security.cert testMatch and testClone.)
I've fixed typo.
Thanks,
Stepan.
Regards,
Mark.
On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote:
On 4/11/06, Mark Hindess wrote:
Stepan,
I have another build running (but without notifications going to
the
list) that runs:
1) build (with reference jdk)
2) build with what we created with 1)
3) test
4) create classlists and compare with class load data for
applications
(tomcat, geronimo, continuum, etc.)
5) generate JAPI results
I'd like to fail this build at step 3, but I can't see an easy
way
to
get 'ant -f make/build.xml test' to run all tests and then fail
if
any
of the module test sub-targets had test failures. I could parse
the
output I suppose, but I'd rather get ant to propagate the junit
tasks
failure property back up to the top level. I've tried a couple
of
things and none seem to work. Any suggestions welcome.
Mark, did you try failureproperty parameter for junit task?
We may add it to ant sub-targets to raise a flag, for example,
create
file "
TESTS.FAILED" in the root, when tests for some module fail. And
in
the
end
of tests suite run check whether this file exists on not.
Thanks,
Stepan.
Regards,
Mark.
On 4/11/06, Stepan Mishura <[EMAIL PROTECTED]> wrote:
Hi,
I've checked out at morning last updates, built the code base
and
run
the
tests …and there are 24 tests failures!
There are 9 tests failures in
org.apache.harmony.tests.java.nio.channels.DatagramChannelTest–
I
saw
these
failures before from time to time. It seems that tests depend
on
some
race
conditions because they may pass if I rerun them. Paulex, are
these
tests
passing for you?
And there are new 15 test failures. So now if I'll make a code
update
or a
bug fix how I can be 100% sure that I don't do any regression?
Currently if a commit breaks the Harmony classlib build a
notification
with
subject: "[continuum] BUILD FAILURE: Classlib/linux.ia32" will
be
send
to
harmony-commits mailing list. Is it possible to have the same
for
tests?
I
mean that after completing automatic build the existing
classlib
tests
suite
should be run. If there are failing tests then a report
notification
with
corresponding subject will be send.
Thanks,
Stepan Mishura
Intel Middleware Products Division
--
Mark Hindess <[EMAIL PROTECTED]>
IBM Java Technology Centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
--
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
Thanks,
Stepan Mishura
Intel Middleware Products Division
--
Mark Hindess <[EMAIL PROTECTED]>
IBM Java Technology Centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
--
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
Thanks,
Stepan Mishura
Intel Middleware Products Division
--
Mark Hindess <[EMAIL PROTECTED]>
IBM Java Technology Centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
--
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
Thanks,
Stepan Mishura
Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Mark Hindess <[EMAIL PROTECTED]>
IBM Java Technology Centre, UK.
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Thanks,
Stepan Mishura
Intel Middleware Products Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
