[
https://jira.codehaus.org/browse/SUREFIRE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Halama updated SUREFIRE-1043:
------------------------------------
Description:
What we need is as following:
For sequential run:
We would like to have some means to specify run order of whole groups of tests,
during one invocation of plugin. Content of the group should be determined by
ant style expressions. Inside each such group, tests should be run in order,
specified by 'runOrder' parameter.
Example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<groupsOrder>
<group>**/*ST</group>
<group>com/something/SomeTest,com/something/SomeTest2</group>
<group>**/*IT,**/IT*</group>
<group>com/something/SomeTest3</group>
</groupsOrder>
<runOrder>alphabetical</runOrder>
</configuration>
</plugin>
In above example run order of tests should be as following:
1. all tests, which have "ST" suffix, should be run in alphabetical order
2. com/something/SomeTest and com/something/SomeTest2 should be run in
alphabetical order
3. all tests, which have "IT" suffix or prefix, should be run in alphabetical
order
4. com/something/SomeTest3 should be run
Instead of adding some new parameter (like above 'groupsOrder' section), I
think 'includes' can be used for this purpose (each entry would determine a
group), with some additional boolean value, i.e. 'groupsByIncludes'
(indicating, that each 'include' entry should be considered as a separate
group):
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<includes>
<include>**/*ST.java</include>
<include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
<include>**/*IT.java,**/IT*.java</include>
<include>com/something/SomeTest3.java</include>
</includes>
<runOrder>alphabetical</runOrder>
<groupsByIncludes>true<groupsByIncludes>
</configuration>
</plugin>
In our case we cannot simply divide one invocation of plugin into several ones
(which would solve this problem), because setuping tests context takes too much
time.
This feature will also give possibility to run tests by some specific order
(when each 'group'/'include' would consist of only one entry). Possibility to
set specific order could be very handy, when i.e. you detected, that some of
your tests are not independent and you want to reproduce the problem.
For parallel run:
We would like to have some means to specify tests groups for parallel run. By
this I mean: tests in each group will be run in parallel, but the groups itself
will be run sequentially (so only tests from the same group can be executed at
the same time)
There should be also some annotation, which can be used to marked tests, which
cannot be run in parallel at all.
If 'includes' contains such tests - they should be run sequentially at the end.
To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be
used here, in the same way as for sequential run, except 'parallel' attribute
should be set to new value: i.e. parallelGroup.
I think, that logging information, when each test starts and stops, would be
also a good idea (we would know exact tests, which are run at the same time).
was:
What we need is as following:
* For sequential run:
We would like to have some means to specify run order of whole groups of tests,
during one invocation of plugin. Content of the group should be determined by
ant style expressions. Inside each such group, tests should be run in order,
specified by 'runOrder' parameter.
Example:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<groupsOrder>
<group>**/*ST</group>
<group>com/something/SomeTest,com/something/SomeTest2</group>
<group>**/*IT,**/IT*</group>
<group>com/something/SomeTest3</group>
</groupsOrder>
<runOrder>alphabetical</runOrder>
</configuration>
</plugin>
In above example run order of tests should be as following:
1. all tests, which have "ST" suffix, should be run in alphabetical order
2. com/something/SomeTest and com/something/SomeTest2 should be run in
alphabetical order
3. all tests, which have "IT" suffix or prefix, should be run in alphabetical
order
4. com/something/SomeTest3 should be run
Instead of adding some new parameter (like above 'groupsOrder' section), I
think 'includes' can be used for this purpose (each entry would determine a
group), with some additional boolean value, i.e. 'groupsByIncludes'
(indicating, that each 'include' entry should be considered as a separate
group):
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<configuration>
<includes>
<include>**/*ST.java</include>
<include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
<include>**/*IT.java,**/IT*.java</include>
<include>com/something/SomeTest3.java</include>
</includes>
<runOrder>alphabetical</runOrder>
<groupsByIncludes>true<groupsByIncludes>
</configuration>
</plugin>
In our case we cannot simply divide one invocation of plugin into several ones
(which would solve this problem), because setuping tests context takes too much
time.
This feature will also give possibility to run tests by some specific order
(when each 'group'/'include' would consist of only one entry). Possibility to
set specific order could be very handy, when i.e. you detected, that some of
your tests are not independent and you want to reproduce the problem.
* For parallel run:
We would like to have some means to specify tests groups for parallel run. By
this I mean: tests in each group will be run in parallel, but the groups itself
will be run sequentially (so only tests from the same group can be executed at
the same time)
There should be also some annotation, which can be used to marked tests, which
cannot be run in parallel at all.
If 'includes' contains such tests - they should be run sequentially at the end.
To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be
used here, in the same way as for sequential run, except 'parallel' attribute
should be set to new value: i.e. parallelGroup.
I think, that logging information, when each test starts and stops, would be
also a good idea (we would know exact tests, which are run at the same time).
> Please add possibility to specify tests groups order for sequential run and
> tests groups content for parallel run.
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1043
> URL: https://jira.codehaus.org/browse/SUREFIRE-1043
> Project: Maven Surefire
> Issue Type: Improvement
> Affects Versions: 2.16
> Reporter: Tomasz Halama
>
> What we need is as following:
> For sequential run:
> We would like to have some means to specify run order of whole groups of
> tests, during one invocation of plugin. Content of the group should be
> determined by ant style expressions. Inside each such group, tests should be
> run in order, specified by 'runOrder' parameter.
> Example:
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-failsafe-plugin</artifactId>
> <configuration>
> <groupsOrder>
> <group>**/*ST</group>
> <group>com/something/SomeTest,com/something/SomeTest2</group>
> <group>**/*IT,**/IT*</group>
> <group>com/something/SomeTest3</group>
> </groupsOrder>
> <runOrder>alphabetical</runOrder>
> </configuration>
> </plugin>
> In above example run order of tests should be as following:
> 1. all tests, which have "ST" suffix, should be run in alphabetical order
> 2. com/something/SomeTest and com/something/SomeTest2 should be run in
> alphabetical order
> 3. all tests, which have "IT" suffix or prefix, should be run in alphabetical
> order
> 4. com/something/SomeTest3 should be run
> Instead of adding some new parameter (like above 'groupsOrder' section), I
> think 'includes' can be used for this purpose (each entry would determine a
> group), with some additional boolean value, i.e. 'groupsByIncludes'
> (indicating, that each 'include' entry should be considered as a separate
> group):
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-failsafe-plugin</artifactId>
> <configuration>
> <includes>
> <include>**/*ST.java</include>
>
> <include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
> <include>**/*IT.java,**/IT*.java</include>
> <include>com/something/SomeTest3.java</include>
> </includes>
> <runOrder>alphabetical</runOrder>
> <groupsByIncludes>true<groupsByIncludes>
> </configuration>
> </plugin>
> In our case we cannot simply divide one invocation of plugin into several
> ones (which would solve this problem), because setuping tests context takes
> too much time.
> This feature will also give possibility to run tests by some specific order
> (when each 'group'/'include' would consist of only one entry). Possibility to
> set specific order could be very handy, when i.e. you detected, that some of
> your tests are not independent and you want to reproduce the problem.
> For parallel run:
> We would like to have some means to specify tests groups for parallel run. By
> this I mean: tests in each group will be run in parallel, but the groups
> itself will be run sequentially (so only tests from the same group can be
> executed at the same time)
> There should be also some annotation, which can be used to marked tests,
> which cannot be run in parallel at all.
> If 'includes' contains such tests - they should be run sequentially at the
> end.
> To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be
> used here, in the same way as for sequential run, except 'parallel' attribute
> should be set to new value: i.e. parallelGroup.
> I think, that logging information, when each test starts and stops, would be
> also a good idea (we would know exact tests, which are run at the same time).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
