Trimming build-all-flag-combinations and adding minicluster profile. [For the 2.x branch cherrypick, this change removes the "profile" stuff.]
build-all-flag-combinations.sh is the long pole in pre-commit testing. It takes about 2.5 hours and runs about 40 builds per the following. $ curl --silent https://jenkins.impala.io/job/all-build-options-ub1604/1707/consoleText | grep 'Building with OPTIONS' | wc -l 40 This commit changes this to run only 7 builds, exercising every path, but not every combination of paths. The paths exercised are: Options -skiptests -noclean and profile 3 Options -skiptests -noclean -so and profile 2 Options -skiptests -noclean -release and profile 3 Options -skiptests -noclean -release -so -ninja and profile 3 Options -skiptests -noclean -asan and profile 3 Options -skiptests -noclean -ubsan -so -ninja and profile 2 Options -skiptests -noclean -tsan and profile 3 I've also added explicitly building with both minicluster profiles and a trivial dryrun option to print the above. The options are simply hard-coded rather than being clever and generating them. We decided this was easier to deal with. Change-Id: I258732a3d963958f76d99f9d7b51450ed67bec21 Reviewed-on: http://gerrit.cloudera.org:8080/10489 Reviewed-by: Joe McDonnell <[email protected]> Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/10497 Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/e7d48a3d Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/e7d48a3d Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/e7d48a3d Branch: refs/heads/2.x Commit: e7d48a3daa07ce3b96c8b1cec94849be77552ee1 Parents: c98c01c Author: Philip Zeyliger <[email protected]> Authored: Wed May 23 11:41:44 2018 -0700 Committer: Philip Zeyliger <[email protected]> Committed: Thu May 24 16:18:53 2018 +0000 ---------------------------------------------------------------------- bin/jenkins/build-all-flag-combinations.sh | 71 ++++++++++++++----------- 1 file changed, 41 insertions(+), 30 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/e7d48a3d/bin/jenkins/build-all-flag-combinations.sh ---------------------------------------------------------------------- diff --git a/bin/jenkins/build-all-flag-combinations.sh b/bin/jenkins/build-all-flag-combinations.sh index ef1a89e..c0e0f6a 100755 --- a/bin/jenkins/build-all-flag-combinations.sh +++ b/bin/jenkins/build-all-flag-combinations.sh @@ -17,45 +17,56 @@ # specific language governing permissions and limitations # under the License. -# Build Impala with the most common build configurations and check that the build -# succeeds.a Intended for use as a precommit test to make sure nothing got broken. +# Build Impala with the a variety of common build configurations and check that the build +# succeeds. Intended for use as a precommit test to make sure nothing got broken. # # Assumes that ninja and ccache are installed. +# +# Usage: build-all-flag-combinations.sh [--dryrun] set -euo pipefail trap 'echo Error in $0 at line $LINENO: $(cd "'$PWD'" && awk "NR == $LINENO" $0)' ERR . bin/impala-config.sh -OPTIONS=("-skiptests" "-noclean") -FAILED_OPTIONS="" -for BUILD_TYPE in "" -asan -release -ubsan -tsan -do - OPTIONS[2]=$BUILD_TYPE - for NINJA in "" -ninja - do - OPTIONS[3]=$NINJA - for BUILD_SHARED_LIBS in "" -so - do - OPTIONS[4]=$BUILD_SHARED_LIBS - if ! ./bin/clean.sh - then - echo "Clean failed" - exit 1 - fi - echo "Building with OPTIONS: ${OPTIONS[@]}" - if ! time -p ./buildall.sh ${OPTIONS[@]} - then - echo "Build failed with OPTIONS: ${OPTIONS[@]}" - FAILED_OPTIONS="${FAILED_OPTIONS}:${OPTIONS[@]}" - fi - ccache -s - done - done + +# These are configurations for buildall: +CONFIGS=( + # Test gcc builds with and without -so: + "-skiptests -noclean" + "-skiptests -noclean -so" + "-skiptests -noclean -release" + "-skiptests -noclean -release -so -ninja" + # clang sanitizer builds: + "-skiptests -noclean -asan" + "-skiptests -noclean -ubsan -so -ninja" + "-skiptests -noclean -tsan" +) + +FAILED="" + +for CONFIG in "${CONFIGS[@]}"; do + DESCRIPTION="Options $CONFIG" + + if [[ $# == 1 && $1 == "--dryrun" ]]; then + echo $DESCRIPTION + continue + fi + + if ! ./bin/clean.sh; then + echo "Clean failed" + exit 1 + fi + echo "Building with OPTIONS: $DESCRIPTION" + if ! time -p ./buildall.sh $CONFIG; then + echo "Build failed: $DESCRIPTION" + FAILED="${FAILED}:${DESCRIPTION}" + fi + ccache -s done -if [[ "$FAILED_OPTIONS" != "" ]] +if [[ "$FAILED" != "" ]] then - echo "Builds with the following options failed:" - echo "$FAILED_OPTIONS" + echo "The following builds failed:" + echo "$FAILED" exit 1 fi
