Andreas Pabst created SUREFIRE-1914:
---------------------------------------
Summary: XML report omits method signature / display name of Junit
5 parameterized tests if testset reporter is configured to use phrased naming
Key: SUREFIRE-1914
URL: https://issues.apache.org/jira/browse/SUREFIRE-1914
Project: Maven Surefire
Issue Type: Bug
Components: JUnit 5.x support, xml generation
Affects Versions: 3.0.0-M5
Reporter: Andreas Pabst
h3. Description of the issue
Given a test class with two parameterized tests as follows:
{code:java}
@ParameterizedTest
@ValueSource(strings = {"a", "b"})
void test1(String param) {
// test code
}
@ParameterizedTest
@ValueSource(strings = {"a", "b"})
void test2(String param) {
// test code
}
{code}
and a surefire configuration that includes the following:
{code:xml}
<statelessTestsetReporter
implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
<version>3.0</version>
<usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
<usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
<usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
</statelessTestsetReporter>
{code}
the XML report will look like this:
{code:xml}
<testcase name="[1] a" classname="mypackage.MyClass" time="0.001" />
<testcase name="[2] b" classname="mypackage.MyClass" time="0.001" />
<testcase name="[1] a" classname="mypackage.MyClass" time="0.001" />
<testcase name="[2] b" classname="mypackage.MyClass" time="0.001" />
{code}
Note: The test method name/signature is not included in the name attribute.
I would expect something more like this:
{code:xml}
<testcase name="test1(String) [1] a" classname="mypackage.MyClass"
time="0.001" />
<testcase name="test1(String) [2] b" classname="mypackage.MyClass"
time="0.001" />
<testcase name="test2(String) [1] a" classname="mypackage.MyClass"
time="0.001" />
<testcase name="test2(String) [2] b" classname="mypackage.MyClass"
time="0.001" />
{code}
h3. Some context on why this is bad
Omitting the test method name makes it impossible to differentiate individual
test cases in the XML report if there are multiple
{{@ParameterizedTest}}-annotated test cases in the same class that use the same
parameters.
Any software that parses the surefire XML reports will be misled into thinking
these are multiple executions of the same test.
h3. Variant with {{@DisplayName}} usage
There is a variant of this issue: If the {{@ParameterizedTest}} has a
{{@DisplayName}} annotation, whatever was chosen as a display name is not
included in the XML report either - unless it is explicitly referenced in the
name attribute of the {{@ParameterizedTest}} like so: {{@ParameterizedTest(name
= "\{displayName} ...")}}.
h3. Solution ideas
This issue has already been brought up during the discussion of SUREFIRE-1546.
In that discussion [~srdo] and [~marcphilipp] have described an approach how
CONTAINER-type TestIdentifiers could be handled properly, but that particular
problem seems to have been deferred and has seemingly not been picked up again
since.
h3. Workaround
A workaround is to explicitly reference the displayName or include some other
unique string in the name attribute of each {{@ParameterizedTest}} as mentioned
above. In projects with a large legacy testbase that might not be feasible
though, so this issue might prevent the adoption of the feature introduced in
SUREFIRE-1546.
I'm submitting this as a bug because to me the current behaviour of surefire is
at least unexpected, but it could also be seen as a request for improvement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
