Arcturus Bootes created NETBEANS-4418:
-----------------------------------------
Summary: Unit tests silently failing
Key: NETBEANS-4418
URL: https://issues.apache.org/jira/browse/NETBEANS-4418
Project: NetBeans
Issue Type: Bug
Components: apisupport - Harness, platform - Module System
Affects Versions: 11.3
Reporter: Arcturus Bootes
I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.
I'm working on a NetBeans platform application and we recently noticed that our
unit tests are silently failing. Unit tests fail but `ant test` completes with
exit code 0. We are running this through Travis which marks the PR as if it
passed.
Based on debugging the issue broadly is that the
{{continue.after.failing.tests}} property (as defined in
${NETBEANS}/harness/README) does not work.
The reason it doesn't work is that the ant target used for ({{-do-testng}} in
${NETBEANS}/harness/common.xml) isn't processing it correctly
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests
failed; see details above.</fail> {code}
The {{unless}} keyword will pass if the parameter exists (even if it has no
value or a false value), so it doesn't matter if you set it true or false, the
fail state will never happen.
Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always
exists even if you don't set it.
{code:java}
584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback
--> {code}
You can re-produce this issue simply by creating a new module suite containing
one module and one unit test which is set to always fail.
{code:java}
import static org.testng.Assert.fail;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class AntTestNGTest {
/**
* Test of main method, of class AntTest.
*/
@Test
public void testMain() throws InterruptedException {
System.out.println("main");
String[] args = null;
AntTest.main(args);
// TODO review the generated test code and remove the default call to
fail.
fail("The test case is a prototype.");
}
}
{code}
Then run:
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
-Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
-Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
-Dcontinue.after.failing.tests=false test{code}
or
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
-Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
-Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
-Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}
or add {{continue.after.failing.tests=false}} to the {{project.properties
}}file for your module suite and test through Netbeans.
Either way, the test will fail but the build will succeed.
If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml
to point to a property which does not exist
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some
tests failed; see details above.</fail> {code}
The build will fail with the message "Some tests failed..." as expected.
A suggested fix would be to add some logic prior to this {{fail }}condition to
check "continue.after.failing.tests" and create a new property if it is set to
true - that way you could check for that property in the {{unless}} condition
and it would either exist (fail does not happen, all tests are executed) or
not-exist (fail state will happen if tests failed).
Also, while we're at it - I also confirmed the {{test.timeout}} property (also
defined in ${NETBEANS}/harness/README) also does not work - probably for
similar reasons.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
