[ 
https://issues.apache.org/jira/browse/SUREFIRE-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815095#comment-17815095
 ] 

Geoff Soutter commented on SUREFIRE-2147:
-----------------------------------------

[~tibordigana] [~marcphilipp] Are you able to provide any design guidance on 
this?

> JUnit5 DynamicContainer/DynamicTest report loses the hierarchical structure
> ---------------------------------------------------------------------------
>
>                 Key: SUREFIRE-2147
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-2147
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 3.0.0-M7, 3.2.5
>            Reporter: Geoff Soutter
>            Priority: Major
>
> h3. Description of the issue
> I created a simple example of JUnit5 DynamicContainer/DynamicTest
> {code:java}
> package pkg;
> import ...
> public class JUnit5DynamicTest {
>     @TestFactory
>     Collection<DynamicNode> createDynamicTests() {
>         List<DynamicNode> folder2 = new ArrayList<>();
>         folder2.add(DynamicTest.dynamicTest("test1", new MyExecutable()));
>         folder2.add(DynamicTest.dynamicTest("test2", new MyExecutable()));
>         List<DynamicNode> folder1 = new ArrayList<>();
>         folder1.add(DynamicContainer.dynamicContainer("folder2", folder2));
>         List<DynamicNode> rootFolder = new ArrayList<>();
>         rootFolder.add(DynamicContainer.dynamicContainer("folder1", folder1));
>         return rootFolder;
>     }
>     private static class MyExecutable implements Executable {
>         @Override
>         public void execute() throws Throwable {
>         }
>     }
> }
> {code}
> When I run this in IDEA. I get the following hierarchical structure in the UI 
> treeview
>  * JUnit5DynamicTest (pkg)
>  ** createDynamicTests()
>  *** folder1
>  **** folder2
>  ***** test1
>  ***** test2
> Here we can see that dynamic tests are represented by IDEA as nested children 
> of the test method. We can open and close the "virtual packages" created as 
> usual to explore in the dynamically created test results. It is not a "flat 
> list".
> However, things are not as good when using surefire as it still uses the 
> years old ant XSD for XML reporting and that XSD doesn't support anything 
> apart from basic Java usage, certainly nothing like dynamic tests which don't 
> correspond directly to a java class and method name,
> Using surefire 3.2.5 and JUnit5Xml30StatelessReporter, with the "phrased" 
> reported configuration disabled, it creates XML report content in 
> surefire-reports like so
> {code}
>   <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
> time="0.012"/>
>   <testcase name="createDynamicTests" classname="pkg.JUnit5DynamicTest" 
> time="0.001"/>
> {code}
> In this case the Dynamic Tests are completely obscured, instead we get 
> duplicate reports for the @TestFactory method. 
> With the example "phrased" reporter configuration "use" properties enabled, 
> we get
> {code:java}
>   <testcase name="folder2 folder1 createDynamicTests() test1" 
> classname="pkg.JUnit5DynamicTest" time="0.011"/>
>   <testcase name="folder2 folder1 createDynamicTests() test2" 
> classname="pkg.JUnit5DynamicTest" time="0.001"/>
> {code}
> Here all the Dynamic tests are created as synthetic "methods" of the physical 
> class in the HTML report, with the DynamicContainer and DynamicTest names 
> injected into the phrased "method" name. 
> From the perspective of someone with a lot of dynamic tests in a huge folder 
> structure, this is less than ideal as the nested structure is not exposed in 
> the test report in a clean structure way, as it was with package names.
> So how do the tools that process this XML into HTML reports handle it (eg 
> Jenkins JUnit plugin).
> * should surefire uses be forced to enable phrased naming so that reporting 
> tools can access the dynamic folder and test names?
> * how can XML reporting tools show the nested structure of the 
> DynamicFolders? 
> ** can XML reporting tools easily tell whether the XML was generated with 
> phrased or non phrased settings? Should they need to?
> ** if using phrased, should they parse the phrased name attribute to find the 
> dynamic folder and test names?
> ** if using non phrased, is there a way of surefire safely injecting the 
> dynamic folder and test names into an existing attribute which will work in a 
> backwards compatible way with the XML reporting tools?
> Also the internal ordering of the phrased name attribute seems inconsistent, 
> or at least I don't understand the ordering. Surely folder1 should come 
> before folder2 If we expect ordering by heirarchy position then we would 
> expect the folder/method names like "createDynamicTest() folder1 folder2 
> test1".
> h3. Solution Ideas
> Perhaps one day https://github.com/ota4j-team/open-test-reporting will be 
> supported as XML report format and at that point, I guess support for dynamic 
> tests would be easy to add. However that could be years off, if at all.
> Anyway, back to the real world, Surefire using the old ant XSD has two 
> variants of the XML report, using non phrased method names, or phrased method 
> names.
> Dynamic tests don't have a non phrased name. So its not really clear to me 
> what the distinction between phrased and non phrased names means for Dynamic 
> tests. Nevertheless, phrased and non phrased is what Surefire is designed 
> for, so we have to come up with something.
> h3. Non Phrased
> Dynamic tests with non phrased method names are currently not usable. So 
> anything we do here would be an improvement.
> So for non phrased method names, we could add the dynamic names as synthetic 
> child packages in the classname attribute. 
> That is, when mapping from DynamicTests to Java packages, we have:
> || Java Structure || Dynamic Test Structure ||
> | package identifier | DynamicContainer name |
> | class name | static placeholder (e.g. Tests) |
> | method name | DynamicTest name |
>   
> I think we could do that by moving the method name and folder name from name= 
> into classname= and adding a static placeholder for the classname.
> {code:java}
>   <testcase name="test1" 
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
>   <testcase name="test2" 
> classname="pkg.JUnit5DynamicTest.createDynamicTests().folder1.folder2.Tests"/>
> {code}
> h3. Phrased
> For phrased usage, its less obvious what to do. 
> The nested structure is injected into a single XML attribute and I haven't 
> seen any documentation of the internal structure of that attribute. 
> The ordering looks inconsistent as it is and possibly needs fixing. However, 
> changing this value without any schema as to what the contents mean seems 
> dangerous. 
> Casual testing with an available Jenkins instance seems to indicate that 
> Jenkins JUnit already has some capability to parse the phrased method name as 
> it shows folder2 as a "package" (but folder1 is lost). 
> To get an overall useful result here I guess we'd need to
> * create and publish some kind of docs / schema on the internal structure we 
> want to support for downstream reporting tools
> * adjust whatever output we have now to meet the docs/schema, if required
> * wait for downstream reporting tools to support it



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to