[
https://issues.apache.org/jira/browse/NETBEANS-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arcturus Bootes updated NETBEANS-4418:
--------------------------------------
Description:
I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.
I'm working on a NetBeans platform application and we recently noticed that our
unit tests are silently failing. Unit tests fail but `ant test` completes with
exit code 0. We are running this through Travis which marks the PR as if it
passed.
Based on debugging the issue broadly is that the
{{continue.after.failing.tests}} property (as defined in
${NETBEANS}/harness/README) does not work.
The reason it doesn't work is that the ant target used for ({{-do-testng}} in
${NETBEANS}/harness/common.xml) isn't processing it correctly
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests
failed; see details above.</fail> {code}
The {{unless}} keyword will pass if the parameter exists (even if it has no
value or a false value), so it doesn't matter if you set it true or false, the
fail state will never happen.
Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always
exists even if you don't set it.
{code:java}
584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback
--> {code}
You can re-produce this issue simply by creating a new module suite containing
one module and one unit test which is set to always fail.
{code:java}
import static org.testng.Assert.fail;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class AntTestNGTest {
/**
* Test of main method, of class AntTest.
*/
@Test
public void testMain() throws InterruptedException {
System.out.println("main");
String[] args = null;
AntTest.main(args);
// TODO review the generated test code and remove the default call to
fail.
fail("The test case is a prototype.");
}
}
{code}
Then run:
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
-Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
-Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
-Dcontinue.after.failing.tests=false test{code}
or
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
-Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
-Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
-Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}
or add {{continue.after.failing.tests=false}} to the {{project.properties
}}file for your module suite and test through Netbeans.
Either way, the test will fail but the build will succeed.
If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml
to point to a property which does not exist
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some
tests failed; see details above.</fail> {code}
The build will fail with the message "Some tests failed..." as expected.
A suggested fix would be to add some logic prior to this {{fail}} condition to
check "continue.after.failing.tests" and create a new property if it is set to
true - that way you could check for that property in the {{unless}} condition
and it would either exist (fail does not happen, all tests are executed) or
not-exist (fail state will happen if tests failed).
Also, while we're at it - I also confirmed the {{test.timeout}} property (also
defined in ${NETBEANS}/harness/README) also does not work - probably for
similar reasons.
was:
I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.
I'm working on a NetBeans platform application and we recently noticed that our
unit tests are silently failing. Unit tests fail but `ant test` completes with
exit code 0. We are running this through Travis which marks the PR as if it
passed.
Based on debugging the issue broadly is that the
{{continue.after.failing.tests}} property (as defined in
${NETBEANS}/harness/README) does not work.
The reason it doesn't work is that the ant target used for ({{-do-testng}} in
${NETBEANS}/harness/common.xml) isn't processing it correctly
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests
failed; see details above.</fail> {code}
The {{unless}} keyword will pass if the parameter exists (even if it has no
value or a false value), so it doesn't matter if you set it true or false, the
fail state will never happen.
Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always
exists even if you don't set it.
{code:java}
584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback
--> {code}
You can re-produce this issue simply by creating a new module suite containing
one module and one unit test which is set to always fail.
{code:java}
import static org.testng.Assert.fail;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class AntTestNGTest {
/**
* Test of main method, of class AntTest.
*/
@Test
public void testMain() throws InterruptedException {
System.out.println("main");
String[] args = null;
AntTest.main(args);
// TODO review the generated test code and remove the default call to
fail.
fail("The test case is a prototype.");
}
}
{code}
Then run:
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
-Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
-Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
-Dcontinue.after.failing.tests=false test{code}
or
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
-Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
-Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
-Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}
or add {{continue.after.failing.tests=false}} to the {{project.properties
}}file for your module suite and test through Netbeans.
Either way, the test will fail but the build will succeed.
If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml
to point to a property which does not exist
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some
tests failed; see details above.</fail> {code}
The build will fail with the message "Some tests failed..." as expected.
A suggested fix would be to add some logic prior to this {{fail }}condition to
check "continue.after.failing.tests" and create a new property if it is set to
true - that way you could check for that property in the {{unless}} condition
and it would either exist (fail does not happen, all tests are executed) or
not-exist (fail state will happen if tests failed).
Also, while we're at it - I also confirmed the {{test.timeout}} property (also
defined in ${NETBEANS}/harness/README) also does not work - probably for
similar reasons.
> Unit tests silently failing
> ---------------------------
>
> Key: NETBEANS-4418
> URL: https://issues.apache.org/jira/browse/NETBEANS-4418
> Project: NetBeans
> Issue Type: Bug
> Components: apisupport - Harness, platform - Module System
> Affects Versions: 11.3
> Reporter: Arcturus Bootes
> Priority: Major
>
> I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.
>
> I'm working on a NetBeans platform application and we recently noticed that
> our unit tests are silently failing. Unit tests fail but `ant test` completes
> with exit code 0. We are running this through Travis which marks the PR as if
> it passed.
>
> Based on debugging the issue broadly is that the
> {{continue.after.failing.tests}} property (as defined in
> ${NETBEANS}/harness/README) does not work.
>
> The reason it doesn't work is that the ant target used for ({{-do-testng}} in
> ${NETBEANS}/harness/common.xml) isn't processing it correctly
>
> {code:java}
> 773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests
> failed; see details above.</fail> {code}
>
> The {{unless}} keyword will pass if the parameter exists (even if it has no
> value or a false value), so it doesn't matter if you set it true or false,
> the fail state will never happen.
>
> Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always
> exists even if you don't set it.
>
> {code:java}
> 584: <property name="continue.after.failing.tests" value="true"/> <!--
> fallback --> {code}
>
> You can re-produce this issue simply by creating a new module suite
> containing one module and one unit test which is set to always fail.
>
>
> {code:java}
> import static org.testng.Assert.fail;
> import org.testng.annotations.AfterClass;
> import org.testng.annotations.AfterMethod;
> import org.testng.annotations.BeforeClass;
> import org.testng.annotations.BeforeMethod;
> import org.testng.annotations.Test;
> public class AntTestNGTest {
>
> /**
> * Test of main method, of class AntTest.
> */
> @Test
> public void testMain() throws InterruptedException {
> System.out.println("main");
> String[] args = null;
> AntTest.main(args);
> // TODO review the generated test code and remove the default call to
> fail.
> fail("The test case is a prototype.");
> }
> }
> {code}
>
> Then run:
>
> {code:java}
> ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
> -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
> -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
> -Dcontinue.after.failing.tests=false test{code}
>
> or
> {code:java}
> ant -Dnbplatform.active.dir="${NETBEANS_HOME}"
> -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}"
> -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness
> -Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}
> or add {{continue.after.failing.tests=false}} to the {{project.properties
> }}file for your module suite and test through Netbeans.
>
> Either way, the test will fail but the build will succeed.
>
> If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml
> to point to a property which does not exist
>
> {code:java}
> 773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some
> tests failed; see details above.</fail> {code}
>
> The build will fail with the message "Some tests failed..." as expected.
> A suggested fix would be to add some logic prior to this {{fail}} condition
> to check "continue.after.failing.tests" and create a new property if it is
> set to true - that way you could check for that property in the {{unless}}
> condition and it would either exist (fail does not happen, all tests are
> executed) or not-exist (fail state will happen if tests failed).
>
> Also, while we're at it - I also confirmed the {{test.timeout}} property
> (also defined in ${NETBEANS}/harness/README) also does not work - probably
> for similar reasons.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
