On 5/16/11 3:44 AM, Dr. Dietmar Wolz wrote: > Nikolaus Hansen, Luc and me discussed this issue in Toulouse. > We have two options to handle this kind of failure in tests of stochastic > optimization algorithms: > 1) fixed random seed - but this reduces the value of the test > 2) Using the RetryRunner - preferred solution > > @Retry(3) should be sufficient for all tests. > The problem with that is that it is really equivalent to just reducing the sensitivity of the test to sensitivity^3 (if, e.g, the test will pick up anomalies with stochastic probability of less than alpha as is, making it retry three times really just reduces that sensitivity to alpha^3). I think the right answer here is to find out why the test is failing with higher than, say .001 probability and fix the underlying problem. If the test itself is too sensitive, then we should fix that. Then switch to a fixed seed for the released code, reverting to random seeding when the code is under development.
Phil > Shall I prepare a patch? > >> @RunWith(RetryRunner.class) before the corresponding test class and by > adding the annotation @Retry(n) with any some > small integer, say 3 to 5 > > -----Ursprüngliche Nachricht----- > Von: Luc Maisonobe [mailto:luc.maison...@free.fr] > Gesendet: Donnerstag, 28. April 2011 10:40 > An: Commons Developers List > Betreff: [math] [GUMP@vmgump]: Project commons-math (in module > apache-commons) failed > > Le 28/04/2011 00:43, Gump a écrit : >> To whom it may engage... >> >> This is an automated request, but not an unsolicited one. For more >> information please visit http://gump.apache.org/nagged.html, >> and/or contact the folk at gene...@gump.apache.org. >> >> Project commons-math has an issue affecting its community integration. >> This issue affects 1 projects, >> and has been outstanding for 7 runs. >> The current state of this project is 'Failed', with reason 'Build Failed'. >> For reference only, the following projects are affected by this: >> - commons-math : The Jakarta Mathematics Library >> >> >> Full details are available at: >> >> http://vmgump.apache.org/gump/public/apache-commons/commons-math/index >> .html > Hi all, > > Once again, the failure is due to some tests in CMA-ES. > The output of the failed test is here: > <http://vmgump.apache.org/gump/public/apache-commons/commons-math/gump_file/ > TEST-org.apache.commons.math.optimization.direct.CMAESOptimizerTest.txt.html >> . > Dietmar, could you check this and tell if the test failure is expected or > not ? If the test should fail once every few runs (typically due to some > random drawings within the test), we have a way to handle such failures in > the automatic tests. We run the tests that may fail a few times to make sure > we can separate random failures from systematic failures. > > The way we do this is by adding the annotation > @RunWith(RetryRunner.class) before the corresponding test class and by > adding the annotation @Retry(n) with any some small integer, say 3 to 5, > before the test method (in addition to the classical JUnit @Test > annotation). The number n is the number of retries we allow before declaring > the test systematically failed. > > Could you have a look at this ? > > thanks > Luc > >> That said, some information snippets are provided here. >> >> The following annotations (debug/informational/warning/error messages) > were provided: >> -DEBUG- Sole jar output [commons-math-28042011.jar] identifier set to >> project name >> -DEBUG- Dependency on junit exists, no need to add for property > junit.jar. >> -INFO- Failed with reason build failed >> -INFO- Project Reports in: >> /srv/gump/public/workspace/apache-commons/math/target/surefire-reports >> -WARNING- No directory >> [/srv/gump/public/workspace/apache-commons/math/target/surefire-report >> s] >> -INFO- Project Reports in: >> /srv/gump/public/workspace/apache-commons/math/target/test-reports >> -INFO- Failed to extract fallback artifacts from Gump Repository >> >> >> >> The following work was performed: >> http://vmgump.apache.org/gump/public/apache-commons/commons-math/gump_ >> work/build_apache-commons_commons-math.html >> Work Name: build_apache-commons_commons-math (Type: Build) Work ended >> in a state of : Failed >> Elapsed: 5 mins 18 secs >> Command Line: /usr/lib/jvm/java-6-openjdk/bin/java >> -Djava.awt.headless=true -Dbuild.sysclasspath=only >> -Xbootclasspath/p:/srv/gump/public/workspace/xml-xerces2/build/xercesI >> mpl.jar:/srv/gump/public/workspace/xml-commons/java/external/build/xml >> -apis.jar org.apache.tools.ant.Main >> -Dgump.merge=/srv/gump/public/gump/work/merge.xml >> -Djunit.jar=/srv/gump/public/workspace/junit/dist/junit-28042011.jar >> -Dfinal.name=commons-math-28042011 jar [Working Directory: >> /srv/gump/public/workspace/apache-commons/math] >> CLASSPATH: >> /usr/lib/jvm/java-6-openjdk/lib/tools.jar:/srv/gump/public/workspace/a >> pache-commons/math/target/classes:/srv/gump/public/workspace/apache-co >> mmons/math/target/test-classes:/srv/gump/public/workspace/ant/dist/lib >> /ant.jar:/srv/gump/public/workspace/ant/dist/lib/ant-launcher.jar:/srv >> /gump/public/workspace/ant/dist/lib/ant-jmf.jar:/srv/gump/public/works >> pace/ant/dist/lib/ant-junit.jar:/srv/gump/public/workspace/ant/dist/li >> b/ant-swing.jar:/srv/gump/public/workspace/ant/dist/lib/ant-apache-res >> olver.jar:/srv/gump/public/workspace/ant/dist/lib/ant-apache-xalan2.ja >> r:/srv/gump/packages/junit3.8.1/junit.jar:/srv/gump/public/workspace/x >> ml-commons/java/build/resolver.jar:/srv/gump/public/workspace/xml-comm >> ons/java/external/build/xml-apis-ext.jar:/srv/gump/public/workspace/ju >> nit/dist/junit-28042011.jar:/srv/gump/public/workspace/junit/dist/juni >> t-dep-28042011.jar >> --------------------------------------------- >> [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.17 sec >> [junit] Running > org.apache.commons.math.stat.ranking.NaturalRankingTest >> [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.157 sec >> [junit] Running > org.apache.commons.math.stat.regression.GLSMultipleLinearRegressionTest >> [junit] Tests run: 21, Failures: 0, Errors: 0, Time elapsed: 1.064 sec >> [junit] Running > org.apache.commons.math.stat.regression.OLSMultipleLinearRegressionTest >> [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 0.295 sec >> [junit] Running > org.apache.commons.math.stat.regression.SimpleRegressionTest >> [junit] Tests run: 16, Failures: 0, Errors: 0, Time elapsed: 0.202 sec >> [junit] Running > org.apache.commons.math.transform.FastCosineTransformerTest >> [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.155 sec >> [junit] Running > org.apache.commons.math.transform.FastFourierTransformerTest >> [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.178 sec >> [junit] Running > org.apache.commons.math.transform.FastHadamardTransformerTest >> [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.153 sec >> [junit] Running > org.apache.commons.math.transform.FastSineTransformerTest >> [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.173 sec >> [junit] Running org.apache.commons.math.util.BigRealFieldTest >> [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.03 sec >> [junit] Running org.apache.commons.math.util.BigRealTest >> [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 0.064 sec >> [junit] Running org.apache.commons.math.util.ContinuedFractionTest >> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.136 sec >> [junit] Running org.apache.commons.math.util.DefaultTransformerTest >> [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.041 sec >> [junit] Running > org.apache.commons.math.util.FastMathStrictComparisonTest >> [junit] Tests run: 53, Failures: 0, Errors: 0, Time elapsed: 0.481 sec >> [junit] Running org.apache.commons.math.util.FastMathTest >> [junit] Tests run: 35, Failures: 0, Errors: 0, Time elapsed: 15.605 > sec >> [junit] Running org.apache.commons.math.util.IncrementorTest >> [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.026 sec >> [junit] Running org.apache.commons.math.util.MathUtilsTest >> [junit] Tests run: 73, Failures: 0, Errors: 0, Time elapsed: 1.918 sec >> [junit] Running > org.apache.commons.math.util.MultidimensionalCounterTest >> [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.17 sec >> [junit] Running > org.apache.commons.math.util.OpenIntToDoubleHashMapTest >> [junit] Tests run: 17, Failures: 0, Errors: 0, Time elapsed: 0.423 sec >> [junit] Running org.apache.commons.math.util.OpenIntToFieldTest >> [junit] Tests run: 17, Failures: 0, Errors: 0, Time elapsed: 0.274 sec >> [junit] Running org.apache.commons.math.util.PairTest >> [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.016 sec >> [junit] Running org.apache.commons.math.util.ResizableDoubleArrayTest >> [junit] Tests run: 14, Failures: 0, Errors: 0, Time elapsed: 0.202 sec >> [junit] Running org.apache.commons.math.util.TransformerMapTest >> [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 0.039 >> sec >> >> BUILD FAILED >> /srv/gump/public/workspace/apache-commons/math/build.xml:214: There were > test failures. >> Total time: 5 minutes 18 seconds >> --------------------------------------------- >> >> To subscribe to this information via syndicated feeds: >> - RSS: >> http://vmgump.apache.org/gump/public/apache-commons/commons-math/rss.x >> ml >> - Atom: >> http://vmgump.apache.org/gump/public/apache-commons/commons-math/atom. >> xml >> >> ============================== Gump Tracking Only === Produced by >> Apache Gump(TM) version 2.3. >> Gump Run 10000028042011, vmgump.apache.org:vmgump:10000028042011 >> Gump E-mail Identifier (unique within run) #3. >> >> -- >> Apache Gump >> http://gump.apache.org/ [Instance: vmgump] >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >> For additional commands, e-mail: dev-h...@commons.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
