[
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815095#comment-17815095
]
Geoff Soutter commented on SUREFIRE-2147:
-----------------------------------------
[~tibordigana] [~marcphilipp] Are you able to provide any design guidance on
this?
> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---------------------------------------------------------------------------
>
> Key: SUREFIRE-2147
> URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 3.0.0-M7, 3.2.5
> Reporter: Geoff Soutter
> Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
> @TestFactory
> Collection<DynamicNode> createDynamicTests() {
> List<DynamicNode> folder2 = new ArrayList<>();
> folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
> folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
> List<DynamicNode> folder1 = new ArrayList<>();
> folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
> List<DynamicNode> rootFolder = new ArrayList<>();
> rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
> return rootFolder;
> }
> private static class MyExecutable implements Executable {
> @Override
> public void execute() throws Throwable {
> }
> }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI
> treeview
> * JUnit5DynamicTest (pkg)
> ** createDynamicTests()
> *** folder1
> **** folder2
> ***** test1
> ***** test2
> Here we can see that dynamic tests are represented by IDEA as nested children
> of the test method. We can open and close the "virtual packages" created as
> usual to explore in the dynamically created test results. It is not a "flat
> list".
> However, things are not as good when using surefire as it still uses the
> years old ant XSD for XML reporting and that XSD doesn't support anything
> apart from basic Java usage, certainly nothing like dynamic tests which don't
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased"
> reported configuration disabled, it creates XML report content in
> surefire-reports like so
> {code}
> <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
> time="0.012"/>
> <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest"
> time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get
> duplicate reports for the @TestFactory method.
> With the example "phrased" reporter configuration "use" properties enabled,
> we get
> {code:java}
> <testcase name="folder2 folder1 createDynamicTests() test1"
> classname="pkg.JUnit5DynamicTest" time="0.011"/>
> <testcase name="folder2 folder1 createDynamicTests() test2"
> classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical
> class in the HTML report, with the DynamicContainer and DynamicTest names
> injected into the phrased "method" name.
> From the perspective of someone with a lot of dynamic tests in a huge folder
> structure, this is less than ideal as the nested structure is not exposed in
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it (eg
> Jenkins JUnit plugin).
> * should surefire uses be forced to enable phrased naming so that reporting
> tools can access the dynamic folder and test names?
> * how can XML reporting tools show the nested structure of the
> DynamicFolders?
> ** can XML reporting tools easily tell whether the XML was generated with
> phrased or non phrased settings? Should they need to?
> ** if using phrased, should they parse the phrased name attribute to find the
> dynamic folder and test names?
> ** if using non phrased, is there a way of surefire safely injecting the
> dynamic folder and test names into an existing attribute which will work in a
> backwards compatible way with the XML reporting tools?
> Also the internal ordering of the phrased name attribute seems inconsistent,
> or at least I don't understand the ordering. Surely folder1 should come
> before folder2 If we expect ordering by heirarchy position then we would
> expect the folder/method names like "createDynamicTest() folder1 folder2
> test1".
> h3. Solution Ideas
> Perhaps one day https://github.com/ota4j-team/open-test-reporting will be
> supported as XML report format and at that point, I guess support for dynamic
> tests would be easy to add. However that could be years off, if at all.
> Anyway, back to the real world, Surefire using the old ant XSD has two
> variants of the XML report, using non phrased method names, or phrased method
> names.
> Dynamic tests don't have a non phrased name. So its not really clear to me
> what the distinction between phrased and non phrased names means for Dynamic
> tests. Nevertheless, phrased and non phrased is what Surefire is designed
> for, so we have to come up with something.
> h3. Non Phrased
> Dynamic tests with non phrased method names are currently not usable. So
> anything we do here would be an improvement.
> So for non phrased method names, we could add the dynamic names as synthetic
> child packages in the classname attribute.
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>
> I think we could do that by moving the method name and folder name from name=
> into classname= and adding a static placeholder for the classname.
> {code:java}
> <testcase name="test1"
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> <testcase name="test2"
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> {code}
> h3. Phrased
> For phrased usage, its less obvious what to do.
> The nested structure is injected into a single XML attribute and I haven't
> seen any documentation of the internal structure of that attribute.
> The ordering looks inconsistent as it is and possibly needs fixing. However,
> changing this value without any schema as to what the contents mean seems
> dangerous.
> Casual testing with an available Jenkins instance seems to indicate that
> Jenkins JUnit already has some capability to parse the phrased method name as
> it shows folder2 as a "package" (but folder1 is lost).
> To get an overall useful result here I guess we'd need to
> * create and publish some kind of docs / schema on the internal structure we
> want to support for downstream reporting tools
> * adjust whatever output we have now to meet the docs/schema, if required
> * wait for downstream reporting tools to support it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)