Henry Robinson has uploaded a new change for review. http://gerrit.cloudera.org:8080/7467
Change subject: (PREVIEW) IMPALA-5684: Optionally run be tests in sharded mode ...................................................................... (PREVIEW) IMPALA-5684: Optionally run be tests in sharded mode Googletest supports sharded execution, where multiple independent processes can run a subset of a test suite in parallel on the same machine or different machines. This patch adds preliminary support for doing so for some of our backend tests, where the parallelism speedup is worth the extra overhead. How we do this: for every backend test 'foo', add a new test target in CMake, called 'foo_sharded'. That target executes a wrapper script, bin/run-sharded-test.sh, which runs N copies of the test in parallel, with the environment correctly set up to enable sharding. You can run a sharded test like this: ctest -R "foo_sharded" Then, in bin/run-backend-tests.sh, keep a whitelist of tests that can be successfully run in sharded mode (i.e. there are no known conflicts from running multiple copies at once, and the speedup is worth the overhead). Use that list to run ctest twice, once in serial mode with no sharded tests (using -E to exclude those tests) and once in sharded mode running only the whitelisted sharded tests. Each shard process logs to its own directory 'foo_test_shard_N' under the usual logs/be_tests/ root. On my desktop machine, this improves expr-test runtime from ~10 minutes to about 5 minutes. More importantly, this opens up more opportunity for easy latency improvements if we break the large test cases into smaller ones that are parallelisable. Change-Id: I865db25b07728f3886133316ded5122c60490967 --- M be/CMakeLists.txt M bin/run-backend-tests.sh A bin/run-sharded-test.sh 3 files changed, 97 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/7467/1 -- To view, visit http://gerrit.cloudera.org:8080/7467 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I865db25b07728f3886133316ded5122c60490967 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Henry Robinson <[email protected]>
