GitHub user ilooner opened a pull request:
https://github.com/apache/drill/pull/940
DRILL-5752 Speed Up Unit Tests add Test Categories
This PR does the following.
**Increased Concurrency:** Previously tests only executed in 2 simultaneous
processes, now one process per core is used to execute tests. This PR also
fixed a few issues that popped up when concurrency was increased.
**Test Categories:** Tests can now be executed according to their
categories. There are two main categories of tests **Fast Tests** and
**Secondary Tests**. **Fast Tests** are fast and are executed by default with
the following command:
```
mvn -T 1C clean install
```
**Note:** -T 1C increased the number of threads used to compile classes,
and allows us to run tests for sub modules in parallel. This command takes
about 15 minutes to run on my laptop.
**Secondary Tests** are slower. To run both fast tests and **Secondary
Tests** you can use the following command.
```
mvn -T 1C clean install -DexcludedGroups=""
```
In addition to **Fast** and **Secondary** tests there are more categories
like:
- **JdbcTest**
- **ParquetTest**
- **HiveStorageTest**
- And many more
All of these categories are in the **drill-common** sub modules in
**org.apahce.drill.categories**. All the tests are marked with one or more of
these categories, and we can use these categories to select the tests we run.
To run all the **Fast** tests that belong to the **JdbcTest** category we can
use the following command:
```
mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest"
```
In order to run all the **Fast** AND **Secondary** JdbcTests you can use
this command
```
mvn -T 1C clean install -Dgroups="org.apache.drill.categories.JdbcTest"
-DexcludedGroups=""
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilooner/drill DRILL-5752
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/940.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #940
----
commit ba3d1ed53d7cabc5f5ac860531f6e989f567bc65
Author: Timothy Farkas <[email protected]>
Date: 2017-08-30T19:53:49Z
- Removed redundant exclusion from java-exec pom
- Marked the slow tests in contrib as secondary tests
commit 9a3b7d2e1de85ce4397311e9ff2433432982fb03
Author: Timothy Farkas <[email protected]>
Date: 2017-08-30T22:02:23Z
- Made slow unit tests SecondaryTests
- Changed the number of forked test processes from just 2 processes to 1.5
process per core
commit 4159bf8f3ad4966751b9176843e240bb4885cec5
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T00:10:38Z
- Added JdbcTest category
commit 0ab8ba8474b68bae8175f4b3c4a9b372f20eef8a
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T00:24:03Z
- Removed unused imports from tests
commit bf3748c6ddd96b303762d8fde9046215cb243c02
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T00:34:22Z
- Added JdbcStorageTest Category
commit 749fa9b1c4ae113a010de33bf359a38a1d56833e
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T18:15:18Z
- Created the HiveStorageTest category
- Made excludedGroups overrideable
commit 50ddbb63f05eb08ac06d46dcf3c078e61dd3d042
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T18:39:28Z
- Added HbaseStorageTest category
commit 10097f61cb2df8898c2e7d8ff20cfb63fddcb6c5
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T18:50:33Z
- Added the KuduStorageTest category
commit 4dbad7edd5f479abc7db698e3ee2a2dad62abf20
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T19:06:34Z
- Added the MongoStorageTest Category
commit 5077b7beeccddcb0c84c70645c879cd3168f1bc1
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T19:42:09Z
- Added a MemoryTest category
commit a0e3832511041aeb18590d65ebb7166e40be045f
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T19:53:42Z
- Add another secondary test.
commit efc9b8bb5af3ce891b890217ed1595d0db26cbbb
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:03:08Z
- Added a SecurityTest category
- Made more of the security tests secondary tests
commit 8dfacb0715252e063eca7d7a51895b8da191b273
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:05:05Z
- Removed unused import for security test
commit 72c3db7f1774fb0e37dd61a25ef1215f0391da63
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:05:57Z
- Added another memory test
commit 3fd24f8062887253a27bbdb8f31f12a06aa97270
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:22:14Z
- Added an OperatorTest category
commit 11290f9d48d223f8ca8b9c2df56e458616b51614
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:29:50Z
- Added license headers
- Added a ParquetTest category
commit 3b03bb385466a2796578dd8ad96f2596a2b0f95e
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:33:35Z
- Added a VectorTest category
commit 4d242150cb8228ba044ad96706f09524664773bb
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:55:00Z
- Added the SqlTest category
commit 3cb3e895a9eb5171b88085c6984dbcb2685bbe58
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T20:59:14Z
- Added more Vector tests
commit 5f6ef77b65966b430a3b77ec577d3f2627133b59
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T21:01:07Z
- Added more security tests
commit 900911b2845e1a2b08d1f587afc505436ce6424e
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T21:03:26Z
- Added another sql test
commit 0fc54d716a9920bfd768d28314f9cdc30bf8d0d5
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T21:04:50Z
- Added another security test
commit 8e8c74b23646a3c6f203ae31468bf1e51787face
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T21:07:13Z
- Added another vector test
commit 0b0f42a85d327b5ef723aa4ba58b30a5854c8ca6
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T21:14:33Z
- Added more operator and sql tests.
commit f62ad15d3e0063032e55e857c898e08af7dc6084
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T21:39:01Z
- Added the PlannerTest category
- Categorized more tests
commit 012e4a7b4048846e7ccca3b263c065daa1156d77
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T22:26:40Z
- Added the SqlFunctionsTest category
- Categorized more unit tests
commit 04024ee2ae01d04070638e5e67f5dcc9cbc9305b
Author: Timothy Farkas <[email protected]>
Date: 2017-08-31T22:49:57Z
- Fixed checkstyle errors
commit c2d63579aaeac88ede9e369dee67db49a9595fc4
Author: Timothy Farkas <[email protected]>
Date: 2017-09-01T04:21:21Z
- Moved test categories into their own package
- Fixed race condition with jdbc storage plugin tests
- Fixed prepared statement timeout with the drill-jdbc tests
commit 47173b8616e5a08929f5a178206fbcb3431e1628
Author: Timothy Farkas <[email protected]>
Date: 2017-09-05T16:55:29Z
- Cleaned up jdbc storage plugin pom
- Increased the timeout of some unit tests
- Increased the prepared statement creation timeout for some unit tests
commit e0c9a361c7e089bec5713defbd41228369336325
Author: Timothy Farkas <[email protected]>
Date: 2017-09-12T17:55:29Z
- Decrease to one test process per core
----
---
