Re: [xwiki-devs] [Proposal] New Functional test best practice for docker-based tests
Improvement thanks to Simon, I’ve now moved the 2 new annotations to the UITest
annotation:
@Documented
@Retention(RUNTIME)
@Target({ TYPE, METHOD, ANNOTATION_TYPE })
@ExtendWith(XWikiDockerExtension.class)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public @interface UITest
Example usage:
@UITest
public class MenuIT
{
@Test
@Order(1)
…
Simpler, thanks Simon!
-Vincent
> On 12 Feb 2019, at 15:04, Vincent Massol wrote:
>
> Actually to be even more rigorous, we should use:
>
> @UITest
> @TestMethodOrder(OrderAnnotation.class)
> @TestInstance(Lifecycle.PER_CLASS)
> public class MenuIT
>
> See
> https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instance-lifecycle
> for more details.
>
> Thanks
> -Vincent
>
>> On 12 Feb 2019, at 14:44, Vincent Massol wrote:
>>
>> Hi,
>>
>> Now that JUnit 5.4.0 is out we have test ordering. I propose the following
>> best practices for TC-based docker tests:
>>
>> @UITest
>> @TestMethodOrder(OrderAnnotation.class)
>> public class MenuIT
>> {
>> @Test
>> @Order(1)
>> public void verifyMenuInApplicationsIndex(TestUtils setup)
>> {
>> ...
>> }
>>
>> @Test
>> @Order(2)
>> public void
>> verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
>> {
>> ...
>> }
>>
>> @Test
>> @Order(3)
>> public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws
>> Exception
>> {
>> …
>> }
>>
>> Instead of the current:
>>
>> @Test
>> public void verifyMenu(TestUtils setup) throws Exception
>> {
>> verifyMenuInApplicationsIndex(setup);
>> verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
>> verifyMenuIsAvailableInAdministration(setup);
>> }
>>
>>
>> Pros:
>> * Easier to run each test
>> * Easier to debug and view recorded video for a specific failing test
>> * More in sync with JUnit’s practices
>> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>>
>> Cons:
>> * Starts and stop the VNC docker container for each test. It takes between 1
>> and 2s to start the VNC container and the stop (i.e. video recording save)
>> should be the same as before.
>>
>> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not
>> too much (between 2% and 3% more).
>>
>> WDYT?
>>
>> Thanks
>> -Vincent
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: [xwiki-devs] [Proposal] New Functional test best practice for docker-based tests
Actually to be even more rigorous, we should use:
@UITest
@TestMethodOrder(OrderAnnotation.class)
@TestInstance(Lifecycle.PER_CLASS)
public class MenuIT
See
https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-instance-lifecycle
for more details.
Thanks
-Vincent
> On 12 Feb 2019, at 14:44, Vincent Massol wrote:
>
> Hi,
>
> Now that JUnit 5.4.0 is out we have test ordering. I propose the following
> best practices for TC-based docker tests:
>
> @UITest
> @TestMethodOrder(OrderAnnotation.class)
> public class MenuIT
> {
>@Test
>@Order(1)
>public void verifyMenuInApplicationsIndex(TestUtils setup)
>{
> ...
>}
>
>@Test
>@Order(2)
>public void
> verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
>{
> ...
>}
>
>@Test
>@Order(3)
>public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws
> Exception
>{
> …
>}
>
> Instead of the current:
>
>@Test
>public void verifyMenu(TestUtils setup) throws Exception
>{
>verifyMenuInApplicationsIndex(setup);
>verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
>verifyMenuIsAvailableInAdministration(setup);
>}
>
>
> Pros:
> * Easier to run each test
> * Easier to debug and view recorded video for a specific failing test
> * More in sync with JUnit’s practices
> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>
> Cons:
> * Starts and stop the VNC docker container for each test. It takes between 1
> and 2s to start the VNC container and the stop (i.e. video recording save)
> should be the same as before.
>
> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too
> much (between 2% and 3% more).
>
> WDYT?
>
> Thanks
> -Vincent
>
>
>
>
>
>
>
>
Re: [xwiki-devs] [Proposal] New Functional test best practice for docker-based tests
> On 12 Feb 2019, at 14:49, Thomas Mortagne wrote:
>
> On Tue, Feb 12, 2019 at 2:45 PM Vincent Massol wrote:
>>
>> Hi,
>>
>> Now that JUnit 5.4.0 is out we have test ordering. I propose the following
>> best practices for TC-based docker tests:
>>
>> @UITest
>> @TestMethodOrder(OrderAnnotation.class)
>> public class MenuIT
>> {
>>@Test
>>@Order(1)
>>public void verifyMenuInApplicationsIndex(TestUtils setup)
>>{
>> ...
>>}
>>
>>@Test
>>@Order(2)
>>public void
>> verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
>>{
>> ...
>>}
>>
>>@Test
>>@Order(3)
>>public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws
>> Exception
>>{
>> …
>>}
>>
>> Instead of the current:
>>
>>@Test
>>public void verifyMenu(TestUtils setup) throws Exception
>>{
>>verifyMenuInApplicationsIndex(setup);
>>verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
>>verifyMenuIsAvailableInAdministration(setup);
>>}
>>
>>
>> Pros:
>> * Easier to run each test
>> * Easier to debug and view recorded video for a specific failing test
>> * More in sync with JUnit’s practices
>> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>>
>> Cons:
>> * Starts and stop the VNC docker container for each test. It takes between 1
>> and 2s to start the VNC container and the stop (i.e. video recording save)
>> should be the same as before.
>>
>> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not
>> too much (between 2% and 3% more).
>>
>> WDYT?
>
> Why only docker based tests and not all UI tests ?
Yes, all UI tests, not just the docker-based ones.
I mentioned the docker-based ones because they are the most controversial ones
(with the VNC start/stop), which is the only reason I see for not wanting to do
this.
Thanks
-Vincent
>
>>
>> Thanks
>> -Vincent
>>
>>
>>
>>
>>
>>
>>
>>
>
>
> --
> Thomas Mortagne
Re: [xwiki-devs] [Proposal] New Functional test best practice for docker-based tests
On Tue, Feb 12, 2019 at 2:45 PM Vincent Massol wrote:
>
> Hi,
>
> Now that JUnit 5.4.0 is out we have test ordering. I propose the following
> best practices for TC-based docker tests:
>
> @UITest
> @TestMethodOrder(OrderAnnotation.class)
> public class MenuIT
> {
> @Test
> @Order(1)
> public void verifyMenuInApplicationsIndex(TestUtils setup)
> {
> ...
> }
>
> @Test
> @Order(2)
> public void
> verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
> {
> ...
> }
>
> @Test
> @Order(3)
> public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws
> Exception
> {
> …
> }
>
> Instead of the current:
>
> @Test
> public void verifyMenu(TestUtils setup) throws Exception
> {
> verifyMenuInApplicationsIndex(setup);
> verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
> verifyMenuIsAvailableInAdministration(setup);
> }
>
>
> Pros:
> * Easier to run each test
> * Easier to debug and view recorded video for a specific failing test
> * More in sync with JUnit’s practices
> * It’s still a scenario and thus doesn’t incur penalty of extra test setup
>
> Cons:
> * Starts and stop the VNC docker container for each test. It takes between 1
> and 2s to start the VNC container and the stop (i.e. video recording save)
> should be the same as before.
>
> So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too
> much (between 2% and 3% more).
>
> WDYT?
Why only docker based tests and not all UI tests ?
>
> Thanks
> -Vincent
>
>
>
>
>
>
>
>
--
Thomas Mortagne
[xwiki-devs] [Proposal] New Functional test best practice for docker-based tests
Hi,
Now that JUnit 5.4.0 is out we have test ordering. I propose the following best
practices for TC-based docker tests:
@UITest
@TestMethodOrder(OrderAnnotation.class)
public class MenuIT
{
@Test
@Order(1)
public void verifyMenuInApplicationsIndex(TestUtils setup)
{
...
}
@Test
@Order(2)
public void
verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(TestUtils setup)
{
...
}
@Test
@Order(3)
public void verifyMenuIsAvailableInAdministration(TestUtils setup) throws
Exception
{
…
}
Instead of the current:
@Test
public void verifyMenu(TestUtils setup) throws Exception
{
verifyMenuInApplicationsIndex(setup);
verifyMenuCreationInLeftPanelWithCurrentWikiVisibility(setup);
verifyMenuIsAvailableInAdministration(setup);
}
Pros:
* Easier to run each test
* Easier to debug and view recorded video for a specific failing test
* More in sync with JUnit’s practices
* It’s still a scenario and thus doesn’t incur penalty of extra test setup
Cons:
* Starts and stop the VNC docker container for each test. It takes between 1
and 2s to start the VNC container and the stop (i.e. video recording save)
should be the same as before.
So I think it’s worth it. Out of 3mn, 3-6 more seconds for 3 tests is not too
much (between 2% and 3% more).
WDYT?
Thanks
-Vincent
