[xwiki-devs] [Proposal] New Functional test best practice for docker-based tests

[Vincent Massol]

Hi,

Now that JUnit 5.4.0 is out we have test ordering. I propose the following best 
practices for TC-based docker tests:

@UITest
@TestMethodOrder(OrderAnnotation.class)
public class MenuIT
{
    @Test
    @Order(1)
    public void verifyMenuInApplicationsIndex(TestUtils setup)
    {
...
    }

    @Test
    @Order(2)
    public void 
verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
    {
...
    }

    @Test
    @Order(3)
    public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws 
Exception
    {
…
    }

Instead of the current:

    @Test
    public void verifyMenu(TestUtils setup) throws Exception
    {
        verifyMenuInApplicationsIndex(setup);
        verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
        verifyMenuIsAvailableInAdministration(setup);
    }


Pros:
* Easier to run each test
* Easier to debug and view recorded video for a specific failing test
* More in sync with JUnit’s practices
* It’s still a scenario and thus doesn’t incur penalty of extra test setup

Cons:
* Starts and stop the VNC docker container for each test. It takes between 1 
and 2s to start the VNC container and the stop (i.e. video recording save) 
should be the same as before.

So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too 
much (between 2% and 3% more).

WDYT?

Thanks
-Vincent