Repository: yetus Updated Branches: refs/heads/master 25382c947 -> b6b6b0dd2
YETUS-229. split --jenkins up Signed-off-by: Sean Busbey <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/yetus/repo Commit: http://git-wip-us.apache.org/repos/asf/yetus/commit/b6b6b0dd Tree: http://git-wip-us.apache.org/repos/asf/yetus/tree/b6b6b0dd Diff: http://git-wip-us.apache.org/repos/asf/yetus/diff/b6b6b0dd Branch: refs/heads/master Commit: b6b6b0dd2fa3181adc63cfc5a62df41334c09a9a Parents: 25382c9 Author: Allen Wittenauer <[email protected]> Authored: Mon Dec 14 11:49:07 2015 -0800 Committer: Allen Wittenauer <[email protected]> Committed: Thu Feb 4 13:15:21 2016 -0800 ---------------------------------------------------------------------- .../in-progress/precommit-advanced.md | 3 +- .../in-progress/precommit-basic.md | 45 +- precommit/core.d/01-common.sh | 10 +- precommit/core.d/builtin-bugsystem.sh | 20 +- precommit/core.d/docker.sh | 463 ++++++++++--------- precommit/personality/hadoop.sh | 13 +- .../test-patch-docker/launch-test-patch.sh | 1 + precommit/test-patch.d/github.sh | 6 +- precommit/test-patch.d/jira.sh | 8 +- precommit/test-patch.sh | 159 ++++--- 10 files changed, 428 insertions(+), 300 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/asf-site-src/source/documentation/in-progress/precommit-advanced.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/in-progress/precommit-advanced.md b/asf-site-src/source/documentation/in-progress/precommit-advanced.md index a6987b1..cd256be 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-advanced.md +++ b/asf-site-src/source/documentation/in-progress/precommit-advanced.md @@ -45,8 +45,7 @@ Be aware that if the Dockerfile is found and the docker command works, test-patc NOTE: If you are using Boot2Docker, you must use directories under /Users (OSX) or C:\Users (Windows) as the base and patchprocess directories (specified by the --basedir and --patch-dir options respectively), because automatically mountable directories are limited to them. See [the Docker documentation](https://docs.docker.com/userguide/dockervolumes/#mount-a-host-directory-as-a-data-volume). -Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in --jenkins mode. In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system. - +Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in --robot mode. In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system. The docker mode can also be run in a "safe" mode that prevents deletions via the `--dockerdelrep` option. Specifying this option will cause test-patch to only report what it would have deleted, but not actually remove anything. # Plug-ins http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/asf-site-src/source/documentation/in-progress/precommit-basic.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/in-progress/precommit-basic.md b/asf-site-src/source/documentation/in-progress/precommit-basic.md index f508047..41c035d 100644 --- a/asf-site-src/source/documentation/in-progress/precommit-basic.md +++ b/asf-site-src/source/documentation/in-progress/precommit-basic.md @@ -23,6 +23,7 @@ test-patch * [Purpose](#Purpose) * [Pre-requisites](#Pre-requisites) * [Basic Usage](#Basic_Usage) +* [Automation](#Automation) * [Build Tool](#Build_Tool) * [Providing Patch Files](#Providing_Patch_Files) * [Project-Specific Capabilities](#Project-Specific_Capabilities) @@ -49,7 +50,6 @@ For Solaris and Solaris-like operating systems, the default location for the POS test-patch requires these installed components to execute: -* A project with a supported build tool (ant, gradle, maven, ...) * git-based project (and git 1.7.3 or higher installed) * bash v3.2 or higher * GNU diff @@ -68,9 +68,25 @@ Bug Systems: * [GitHub](https://github.com/)-based issue tracking * [JIRA](https://www.atlassian.com/software/jira)-based issue tracking +* [Bugzilla](https://www.bugzilla.org/)-based issue tracking (Read Only) + +Build Tools: + +* [ant](https://ant.apache.org) +* [autoconf](https://www.gnu.org/software/autoconf/autoconf.html) +* [cmake](https://www.cmake.org) +* [gradle](https://www.gradle.org) +* make +* [maven](https://maven.apache.org) + +Automation and Isolation: + +* [Docker](https://www.docker.com) version 1.6.0+ +* [Jenkins](https://www.jenkins-ci.org) Unit Test Formats: +* [ctest](https://cmake.org/Wiki/CMake/Testing_With_CTest) * [JUnit](http://junit.org/) * [TAP](https://testanything.org/) @@ -145,15 +161,40 @@ $ test-patch.sh --basedir=<testrepo> --resetrepo /tmp/patchfile We used two new options here. --basedir sets the location of the repository to use for testing. --resetrepo tells test patch that it can go into **destructive** mode. Destructive mode will wipe out any changes made to that repository, so use it with care! +# Automation + After the tests have run, there is a directory that contains all of the test-patch related artifacts. This is generally referred to as the patchprocess directory. By default, test-patch tries to make something off of /tmp to contain this content. Using the `--patch-dir` option, one can specify exactly which directory to use. This is helpful for automated precommit testing so that Jenkins or other automated workflow system knows where to look to gather up the output. For example: ```bash +$ test-patch.sh --robot --patch-dir=${WORKSPACE}/patchprocess --basedir=${WORKSPACE}/source ${WORKSPACE}/patchfile +``` + +... will trigger test-patch to run in fully automated mode, using ${WORKSPACE}/patchprocess as its scratch space, ${WORKSPACE}/source as the source repository, and ${WORKSPACE}/patchfile as the name of the patch to test against. This will always run the unit tests, write answers back to bug systems, remove old, stopped/exited Docker images and containers after 24 hours, forcibly use --resetrepo, and more. The --build-url option is also useful when running in --robot mode so that emails and such +have a location to look at the output artifacts: + +```bash +$ test-patch.sh --robot --build-url=http://server.example.name:80/${buildnumber}/ +``` + +Some plug-ins such as Maven have special handling if there are multiple executions of test-patch happening at once. It is very common when using automation systems to have multiple runs on the same host. In order to assist these plug-ins, an instance identifier may be provided: + +```bash +$ test-patch.sh --robot --instance=1 +``` + +If --robot is specified without an instance, a random number is generated and used. + +There is some special handling if Jenkins is actually your automation tool. Instead of using --robot, use --jenkins: + +```bash $ test-patch.sh --jenkins --patch-dir=${WORKSPACE}/patchprocess --basedir=${WORKSPACE}/source ${WORKSPACE}/patchfile ``` -... will trigger test-patch to run in fully automated Jenkins mode, using ${WORKSPACE}/patchprocess as its scratch space, ${WORKSPACE}/source as the source repository, and ${WORKSPACE}/patchfile as the name of the patch to test against. +This will enable --robot, set the --build-url option from the ${BUILD_URL} environment variable, and the instance identifier is set to the ${EXECUTOR_NUMBER}. + +If stuck containers are a problem, a more aggressive robot may be enabled with the --sentinel option. This option enables killing containers that have been running for over 24 hours as well. # Build Tool http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/core.d/01-common.sh ---------------------------------------------------------------------- diff --git a/precommit/core.d/01-common.sh b/precommit/core.d/01-common.sh index a6eb257..5e49caa 100755 --- a/precommit/core.d/01-common.sh +++ b/precommit/core.d/01-common.sh @@ -24,9 +24,11 @@ function common_defaults BASEDIR=$(pwd) BUGSYSTEMS="" BUILDTOOLS="" - LOAD_SYSTEM_PLUGINS=true + #shellcheck disable=SC2034 + EXEC_MODES="" #shellcheck disable=SC2034 JENKINS=false + LOAD_SYSTEM_PLUGINS=true #shellcheck disable=SC2034 OFFLINE=false OSTYPE=$(uname -s) @@ -47,9 +49,13 @@ function common_defaults PATCH_LEVEL=0 #shellcheck disable=SC2034 PATCH_SYSTEM="" - PROJECT_NAME=yetus + PROJECT_NAME=unknown RESULT=0 #shellcheck disable=SC2034 + ROBOT=false + #shellcheck disable=SC2034 + SENTINEL=false + #shellcheck disable=SC2034 TESTTYPES="" TESTFORMATS="" USER_PLUGIN_DIR="" http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/core.d/builtin-bugsystem.sh ---------------------------------------------------------------------- diff --git a/precommit/core.d/builtin-bugsystem.sh b/precommit/core.d/builtin-bugsystem.sh index f6e25f1..f83e7c9 100755 --- a/precommit/core.d/builtin-bugsystem.sh +++ b/precommit/core.d/builtin-bugsystem.sh @@ -19,6 +19,8 @@ add_bugsystem console +CONSOLE_USE_BUILD_URL=false + ## @description Print out the finished details on the console ## @audience private ## @stability evolving @@ -44,9 +46,9 @@ function console_finalreport declare spcfx=${PATCH_DIR}/spcl.txt if [[ ${result} == 0 ]]; then - if [[ ${JENKINS} == false ]]; then + if [[ ${ROBOT} == false ]]; then if declare -f ${PROJECT_NAME}_console_success >/dev/null; then - ${PROJECT_NAME}_console_success > "${spcfx}" + "${PROJECT_NAME}_console_success" > "${spcfx}" else { printf "IF9fX18gICAgICAgICAgICAgICAgICAgICAgICAgICAgICBfIAovIF9fX3wg"; @@ -60,9 +62,9 @@ function console_finalreport fi printf "\n\n+1 overall\n\n" else - if [[ ${JENKINS} == false ]]; then + if [[ ${ROBOT} == false ]]; then if declare -f ${PROJECT_NAME}_console_failure >/dev/null; then - ${PROJECT_NAME}_console_failure > "${spcfx}" + "${PROJECT_NAME}_console_failure" > "${spcfx}" else { printf "IF9fX19fICAgICBfIF8gICAgICAgICAgICAgICAgXyAKfCAgX19ffF8gXyhf"; @@ -141,8 +143,14 @@ function console_finalreport i=0 until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do - comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | - ${SED} -e "s,@@BASE@@,${PATCH_DIR},g") + if [[ "${CONSOLE_USE_BUILD_URL}" = true && + -n "${BUILD_URL}" ]]; then + comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | + ${SED} -e "s,@@BASE@@,${BUILD_URL}${BUILD_URL_ARTIFACTS},g") + else + comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | + ${SED} -e "s,@@BASE@@,${PATCH_DIR},g") + fi printf "%s\n" "${comment}" ((i=i+1)) done http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/core.d/docker.sh ---------------------------------------------------------------------- diff --git a/precommit/core.d/docker.sh b/precommit/core.d/docker.sh index 5ac8a0f..41a1afa 100755 --- a/precommit/core.d/docker.sh +++ b/precommit/core.d/docker.sh @@ -14,10 +14,77 @@ # See the License for the specific language governing permissions and # limitations under the License. +DOCKERMODE=false +DOCKERCMD=$(command -v docker) DOCKER_ID=${RANDOM} DOCKER_DESTRUCTIVE=true DOCKERFILE_DEFAULT="${BINDIR}/test-patch-docker/Dockerfile" DOCKERFAIL="fallback,continue,fail" +DOCKERSUPPORT=false +DOCKER_ENABLE_PRIVILEGED=true + +#### +#### IMPORTANT +#### +#### If these times are updated, the documentation needs to +#### be changed too! + +# stopped, exited, running, for 24 hours +DOCKER_CONTAINER_PURGE=("86400" "86400" "86400") + +# keep images for 24 hours +DOCKER_IMAGE_PURGE=86400 + +## @description Docker-specific usage +## @stability stable +## @audience private +## @replaceable no +function docker_usage +{ + yetus_add_option "--docker" "Spawn a docker container" + yetus_add_option "--dockercmd=<file>" "Command to use as docker executable (default: '${DOCKERCMD}')" + yetus_add_option "--dockerfile=<file>" "Dockerfile fragment to use as the base (default: '${DOCKERFILE_DEFAULT}')" + yetus_add_option "--dockeronfail=<list>" "If Docker fails, determine fallback method order (default: ${DOCKERFAIL})" + yetus_add_option "--dockerprivd=<bool>" "Run docker in privileged mode (default: '${DOCKER_ENABLE_PRIVILEGED}')" + yetus_add_option "--dockerdelrep" "In Docker mode, only report image/container deletions, not act on them" +} + +## @description Docker-specific argument parsing +## @stability stable +## @audience private +## @replaceable no +## @params arguments +function docker_parse_args +{ + declare i + + for i in "$@"; do + case ${i} in + --docker) + DOCKERSUPPORT=true + ;; + --dockercmd=*) + #shellcheck disable=SC2034 + DOCKERCMD=${i#*=} + ;; + --dockerdelrep) + DOCKER_DESTRUCTIVE=false + ;; + --dockerfile=*) + DOCKERFILE=${i#*=} + ;; + --dockermode) + DOCKERMODE=true + ;; + --dockeronfail=*) + DOCKERFAIL=${i#*=} + ;; + --dockerprivd=*) + DOCKER_ENABLE_PRIVILEGED=${i#*=} + ;; + esac + done +} ## @description Docker initialization pre- and post- re-exec ## @stability stable @@ -25,11 +92,27 @@ DOCKERFAIL="fallback,continue,fail" ## @replaceable no function docker_initialize { - if [[ ${DOCKERMODE} == true ]]; then + declare dockvers + + # --docker and --dockermode are mutually + # exclusive. --docker is used by the user to + # re-exec test-patch in Docker mode. + # --dockermode is used by launch-test-patch (which is + # run as the Docker EXEC in the Dockerfile, + # see elsewhere for more info) to tell test-patch that + # it has been restarted already. launch-test-patch + # also strips --docker from the command line so that we + # don't end up in a loop if the docker image + # also has the docker command in it + + # we are already in docker mode + if [[ "${DOCKERMODE}" == true ]]; then # DOCKER_VERSION is set by our creator. add_footer_table "Docker" "${DOCKER_VERSION}" + return fi + # docker mode hasn't been requested if [[ "${DOCKERSUPPORT}" != true ]]; then return fi @@ -56,6 +139,20 @@ function docker_initialize cleanup_and_exit 1 fi fi + + dockvers=$(docker_version Client) + if [[ "${dockvers}" =~ ^0 + || "${dockvers}" =~ ^1\.[0-5] ]]; then + if [[ "${DOCKERFAIL}" =~ ^12 + || "${DOCKERFAIL}" =~ ^2 ]]; then + add_vote_table 0 docker "Docker command '${DOCKERCMD}' is too old (${dockvers} < 1.6.0). Disabling docker." + DOCKERSUPPORT=false + else + add_vote_table -1 docker "Docker command '${DOCKERCMD}' is too old (${dockvers} < 1.6.0). Disabling docker." + bugsystem_finalreport 1 + cleanup_and_exit 1 + fi + fi } ## @description Verify dockerfile exists @@ -102,19 +199,6 @@ function docker_fileverify ## @return 0 if docker is working function docker_exeverify { - declare pathdocker - - if [[ -z "${DOCKERCMD}" ]]; then - pathdocker=$(which docker 2>/dev/null) - - if [[ ! -f "${pathdocker}" ]]; then - yetus_error "Docker cannot be found." - DOCKERCMD=docker - return 1 - fi - DOCKERCMD="${pathdocker}" - fi - if ! verify_command "Docker" "${DOCKERCMD}"; then return 1 fi @@ -129,252 +213,211 @@ function docker_exeverify } ## @description Run docker with some arguments, and -## @description optionally send to debug +## @description optionally send to debug. +## @description some destructive commands require +## @description DOCKER_DESTRUCTIVE to be set to true ## @audience private ## @stability evolving ## @replaceable no ## @param args function dockercmd { - yetus_debug "${DOCKERCMD} $*" - "${DOCKERCMD}" "$@" + declare subcmd=$1 + shift + + yetus_debug "dockercmd: ${DOCKERCMD} ${subcmd} $*" + if [[ ${subcmd} == rm + || ${subcmd} == rmi + || ${subcmd} == stop + || ${subcmd} == kill ]]; then + if [[ "${DOCKER_DESTRUCTIVE}" == false ]]; then + yetus_error "Safemode: not running ${DOCKERCMD} ${subcmd} $*" + return + fi + fi + "${DOCKERCMD}" "${subcmd}" "$@" } -## @description Stop and delete all defunct containers +## @description Convet docker's time format to ctime ## @audience private ## @stability evolving ## @replaceable no -## @param args -function docker_stop_exited_containers +## @param time +function dockerdate_to_ctime { - declare line - declare id - declare value - declare size - declare exitfn="${PATCH_DIR}/dsec.$$" + declare mytime=$1 - big_console_header "Removing stopped/exited containers" + # believe it or not, date is not even close to standardized... + if [[ $(uname -s) == Linux ]]; then - echo "Docker containers in exit state:" + # GNU date + date -d "${mytime}" "+%s" + else - dockercmd ps -a | ${GREP} Exited > "${exitfn}" - if [[ ! -s "${exitfn}" ]]; then - rm -f "${exitfn}" - return + # BSD date + date -j -f "%FT%T%z" "${mytime}" "+%s" fi - - # stop *all* containers that are in exit state for - # more than > 8 hours - while read -r line; do - id=$(echo "${line}" | cut -f1 -d' ') - value=$(echo "${line}" | cut -f2 -d' ') - size=$(echo "${line}" | cut -f3 -d' ') - - if [[ ${size} =~ day - || ${size} =~ week - || ${size} =~ month - || ${size} =~ year ]]; then - echo "Removing docker ${id}" - if [[ "${DOCKER_DESTRUCTIVE}" = true ]]; then - dockercmd rm "${id}" - else - echo docker rm "${id}" - fi - fi - - if [[ ${size} =~ hours - && ${value} -gt 8 ]]; then - echo "Removing docker ${id}" - if [[ "${DOCKER_DESTRUCTIVE}" = true ]]; then - dockercmd rm "${id}" - else - echo docker rm "${id}" - fi - fi - done < <( - ${SED} -e 's,ago,,g' "${exitfn}"\ - | ${AWK} '{print $1" "$(NF - 2)" "$(NF - 1)}') - rm "${exitfn}" } -## @description Remove all containers that are not -## @description are not running + older than 1 day +## @description Stop and delete all defunct containers ## @audience private ## @stability evolving ## @replaceable no -## @param args -function docker_rm_old_containers +function docker_container_maintenance { declare line declare id - declare value - declare size - declare running + declare name + declare status + declare tmptime + declare starttime declare stoptime - declare curtime + declare remove + declare difftime + declare data - big_console_header "Removing old containers" + if [[ "${ROBOT}" = false ]]; then + return + fi - while read -r line; do - id=$(echo "${line}" | cut -f1 -d, ) - running=$(echo "${line}" | cut -f2 -d, ) - stoptime=$(echo "${line}" | cut -f3 -d, | cut -f1 -d. ) + big_console_header "Docker Container Maintenance" - if [[ ${running} = true ]]; then - yetus_debug "${id} is still running, skipping." - continue - fi + dockercmd ps -a - # believe it or not, date is not even close to standardized... - if [[ $(uname -s) == Linux ]]; then + data=$(dockercmd ps -qa) - # GNU date - stoptime=$(date -d "${stoptime}" "+%s") - else + if [[ -z "${data}" ]]; then + return + fi - # BSD date - stoptime=$(date -j -f "%Y-%m-%dT%H:%M:%S" "${stoptime}" "+%s") + while read -r line; do + id=$(echo "${line}" | cut -f1 -d, ) + name=$(echo "${line}" | cut -f2 -d, ) + status=$(echo "${line}" | cut -f3 -d, ) + tmptime=$(echo "${line}" | cut -f4 -d, | cut -f1 -d. ) + starttime=$(dockerdate_to_ctime "${tmptime}") + tmptime=$(echo "${line}" | cut -f5 -d, | cut -f1 -d. ) + stoptime=$(dockerdate_to_ctime "${tmptime}") + curtime=$(TZ=UTC date "+%s") + remove=false + + case ${status} in + stopped) + ((difftime = curtime - stoptime)) + if [[ ${difftime} -gt ${DOCKER_CONTAINER_PURGE[0]} ]]; then + remove=true + fi + ;; + exited) + ((difftime = curtime - stoptime)) + if [[ ${difftime} -gt ${DOCKER_CONTAINER_PURGE[1]} ]]; then + remove=true + fi + ;; + running) + ((difftime = curtime - starttime)) + if [[ ${difftime} -gt ${DOCKER_CONTAINER_PURGE[2]} + && "${SENTINEL}" = true ]]; then + remove=true + echo "Attempting to kill docker container ${name} [${id}]" + dockercmd kill "${id}" + fi + ;; + *) + ;; + esac + + if [[ "${remove}" == true ]]; then + echo "Attempting to remove docker container ${name} [${id}]" + dockercmd rm "${id}" fi - curtime=$(date "+%s") - ((difftime = curtime - stoptime)) - if [[ ${difftime} -gt 86400 ]]; then - echo "Removing docker ${id}" - if [[ "${DOCKER_DESTRUCTIVE}" = true ]]; then - dockercmd rm "${id}" - else - echo docker rm "${id}" - fi - fi done < <( - # see https://github.com/koalaman/shellcheck/issues/375 - # shellcheck disable=SC2046 - dockercmd inspect \ - -f '{{.Id}},{{.State.Running}},{{.State.FinishedAt}}' \ - $(dockercmd ps -qa) 2>/dev/null) + # shellcheck disable=SC2086 + dockercmd inspect \ + --format '{{.Id}},{{.Name}},{{.State.Status}},{{.State.StartedAt}},{{.State.FinishedAt}}' \ + ${data}) } -## @description Remove untagged/unused images +## @description Delete images after ${DOCKER_IMAGE_PURGE} ## @audience private ## @stability evolving ## @replaceable no -## @param args -function docker_remove_untagged_images +function docker_image_maintenance_helper { + declare id + declare tmptime + declare createtime + declare difftime + declare name + + if [[ "${ROBOT}" = false ]]; then + return + fi + + if [[ -z "$*" ]]; then + return + fi - big_console_header "Removing untagged images" + for id in "$@"; do + tmptime=$(dockercmd inspect --format '{{.Created}}' "${id}" | cut -f1 -d. ) + createtime=$(dockerdate_to_ctime "${tmptime}") + curtime=$(date "+%s") - # this way is a bit more compatible with older docker versions - # shellcheck disable=SC2016 - dockercmd images | tail -n +2 | ${AWK} '$1 == "<none>" {print $3}' | \ - xargs --no-run-if-empty docker rmi + ((difftime = curtime - createtime)) + if [[ ${difftime} -gt ${DOCKER_IMAGE_PURGE} ]]; then + echo "Attempting to remove docker image ${id}" + dockercmd rmi "${id}" + fi + done } -## @description Remove defunct tagged images +## @description Remove untagged/unused images ## @audience private ## @stability evolving ## @replaceable no ## @param args -function docker_remove_old_tagged_images +function docker_image_maintenance { - declare line declare id - declare created - big_console_header "Removing old tagged images" - - while read -r line; do - # shellcheck disable=SC2016 - id=$(echo "${line}" | ${AWK} '{print $1":"$2}') - # shellcheck disable=SC2016 - created=$(echo "${line}" | ${AWK} '{print $5}') - - if [[ ${created} =~ week - || ${created} =~ month - || ${created} =~ year ]]; then - echo "Removing docker image ${id}" - if [[ "${DOCKER_DESTRUCTIVE}" = true ]]; then - dockercmd rmi "${id}" - else - echo docker rmi "${id}" - fi - fi + if [[ "${ROBOT}" = false ]]; then + return + fi - if [[ ${id} =~ yetus/${PROJECT_NAME}:date - || ${id} =~ test-patch- ]]; then - if [[ ${created} =~ day - || ${created} =~ hours ]]; then - echo "Removing docker image ${id}" - if [[ "${DOCKER_DESTRUCTIVE}" = true ]]; then - dockercmd rmi "${id}" - else - echo docker rmi "${id}" - fi - fi - fi - done < <(dockercmd images) -} + big_console_header "Removing old images" -## @description Performance docker maintenance on Jenkins -## @audience private -## @stability evolving -## @replaceable no -## @param args -function docker_cleanup_apache_jenkins -{ - echo "==========================" - echo "Docker Images:" dockercmd images - echo "==========================" - echo "Docker Containers:" - dockercmd ps -a - echo "==========================" - docker_stop_exited_containers + echo "Untagged images:" - docker_rm_old_containers + #shellcheck disable=SC2046 + docker_image_maintenance_helper $(dockercmd images --filter "dangling=true" -q --no-trunc) - docker_remove_untagged_images + echo "Apache Yetus images:" - docker_remove_old_tagged_images -} + # removing this by image id doesn't always work without a force + # in the situations that, for whatever reason, docker decided + # to use the same image. this was a rare problem with older + # releases of yetus. at some point, we should revisit this + # in the mean time, we're going to reconstruct the + # repostory:tag and send that to get removed. -## @description Clean up our old images used for patch testing -## @audience private -## @stability evolving -## @replaceable no -## @param args -function docker_cleanup_yetus_images -{ - declare images - declare imagecount - declare rmimage - declare rmi - - # we always want to leave at least one of our images - # so that the whole thing doesn't have to be rebuilt. - # This also let's us purge any old images so that - # we can get fresh stuff sometimes - # shellcheck disable=SC2016 - images=$(dockercmd images | ${GREP} "yetus/${PROJECT_NAME}" | ${GREP} tp | ${AWK} '{print $1":"$2}') 2>&1 - - # shellcheck disable=SC2086 - imagecount=$(echo ${images} | tr ' ' '\n' | wc -l) - ((imagecount = imagecount - 1 )) - - # shellcheck disable=SC2086 - rmimage=$(echo ${images} | tr ' ' '\n' | tail -${imagecount}) - for rmi in ${rmimage} - do - echo "Removing image ${rmi}" - if [[ "${DOCKER_DESTRUCTIVE}" = true ]]; then - dockercmd rmi "${rmi}" - else - echo docker rmi "${rmi}" - fi - done + #shellcheck disable=SC2046,SC2016 + docker_image_maintenance_helper $(dockercmd images | ${GREP} -e ^yetus | grep tp- | ${AWK} '{print $1":"$2}') + #shellcheck disable=SC2046,SC2016 + docker_image_maintenance_helper $(dockercmd images | ${GREP} -e ^yetus | ${GREP} -v hours | ${AWK} '{print $1":"$2}') + + if [[ "${SENTINTAL}" = false ]]; then + return + fi + + echo "Other images:" + #shellcheck disable=SC2046,SC2016 + docker_image_maintenance_helper $(dockercmd images | tail -n +2 | ${GREP} -v hours | ${AWK} '{print $1":"$2}') } + ## @description Perform pre-run maintenance to free up ## @description resources. With --jenkins, it is a lot ## @description more destructive. @@ -384,11 +427,10 @@ function docker_cleanup_yetus_images ## @param args function docker_cleanup { - if [[ ${TESTPATCHMODE} =~ jenkins ]]; then - docker_cleanup_apache_jenkins - fi - docker_cleanup_yetus_images + docker_image_maintenance + + docker_container_maintenance } ## @description Deterine the user name and user id of the user @@ -457,6 +499,7 @@ function docker_run_image declare dockerfilerev declare baseimagename declare patchimagename="yetus/${PROJECT_NAME}:tp-${DOCKER_ID}" + declare containername="yetus_tp-${DOCKER_ID}" declare client declare server declare retval @@ -469,7 +512,9 @@ function docker_run_image # make a base image, if it isn't available big_console_header "Building base image: ${baseimagename}" start_clock - dockercmd build -t "${baseimagename}" "${PATCH_DIR}/precommit/test-patch-docker" + dockercmd build \ + -t "${baseimagename}" \ + "${PATCH_DIR}/precommit/test-patch-docker" retval=$? #shellcheck disable=SC2046 @@ -489,8 +534,13 @@ function docker_run_image big_console_header "Building patch image: ${patchimagename}" start_clock # using the base image, make one that is patch specific - dockercmd build -t "${patchimagename}" - <<PatchSpecificDocker + dockercmd build \ + -t "${patchimagename}" \ + - <<PatchSpecificDocker FROM ${baseimagename} +LABEL org.apache.yetus="" +LABEL org.apache.yetus.testpatch.patch="tp-${DOCKER_ID}" +LABEL org.apache.yetus.testpatch.project=${PROJECT_NAME} RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME} RUN useradd -g ${GROUP_ID} -u ${USER_ID} -m ${USER_NAME} RUN chown -R ${USER_NAME} /home/${USER_NAME} @@ -528,7 +578,6 @@ PatchSpecificDocker server=$(docker_version Server) dockerversion="Client=${client} Server=${server}" - if [[ ${PATCH_DIR} =~ ^/ ]]; then # shellcheck disable=SC2086 exec "${DOCKERCMD}" run --rm=true -i \ @@ -544,6 +593,7 @@ PatchSpecificDocker --env=PATCH_SYSTEM="${PATCH_SYSTEM}" \ --env=PROJECT_NAME="${PROJECT_NAME}" \ --env=TESTPATCHMODE="${TESTPATCHMODE}" \ + --name "${containername}" \ "${patchimagename}" else # shellcheck disable=SC2086 @@ -559,6 +609,7 @@ PatchSpecificDocker --env=PATCH_SYSTEM="${PATCH_SYSTEM}" \ --env=PROJECT_NAME="${PROJECT_NAME}" \ --env=TESTPATCHMODE="${TESTPATCHMODE}" \ + --name "${containername}" \ "${patchimagename}" fi } http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/personality/hadoop.sh ---------------------------------------------------------------------- diff --git a/precommit/personality/hadoop.sh b/precommit/personality/hadoop.sh index 0ccf137..60944e8 100755 --- a/precommit/personality/hadoop.sh +++ b/precommit/personality/hadoop.sh @@ -229,17 +229,6 @@ function hadoop_native_flags # e.g, HADOOP-12027 for OS X. so no -Drequire.bzip2 # - # current build servers are pretty limited in - # what they support - if [[ ${JENKINS} = true - && ${DOCKERMODE} = false ]]; then - # shellcheck disable=SC2086 - echo -Pnative \ - -Drequire.snappy -Drequire.openssl -Drequire.fuse \ - -Drequire.test.libhadoop - return - fi - case ${OSTYPE} in Linux) # shellcheck disable=SC2086 @@ -265,7 +254,7 @@ function hadoop_native_flags echo \ -Pnative \ -Drequire.snappy -Drequire.openssl \ - -Drequire.libwebhdfs -Drequire.test.libhadoop + -Drequire.test.libhadoop ;; esac } http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/test-patch-docker/launch-test-patch.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch-docker/launch-test-patch.sh b/precommit/test-patch-docker/launch-test-patch.sh index e7fb4dd..aac0d21 100755 --- a/precommit/test-patch-docker/launch-test-patch.sh +++ b/precommit/test-patch-docker/launch-test-patch.sh @@ -33,6 +33,7 @@ export MAVEN_OPTS # strip out --docker param to prevent re-exec again TESTPATCHMODE=${TESTPATCHMODE/--docker } +TESTPATCHMODE=${TESTPATCHMODE%--docker} cd "${BASEDIR}" http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/test-patch.d/github.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/github.sh b/precommit/test-patch.d/github.sh index a3e4b9a..b6c9379 100755 --- a/precommit/test-patch.d/github.sh +++ b/precommit/test-patch.d/github.sh @@ -423,15 +423,13 @@ function github_finalreport rm "${commentfile}" 2>/dev/null - if [[ ${JENKINS} != "true" + if [[ ${ROBOT} = "false" || -z ${GITHUB_ISSUE} ]] ; then return 0 fi big_console_header "Adding comment to Github" - add_footer_table "Console output" "${BUILD_URL}console" - if [[ ${result} == 0 ]]; then echo ":confetti_ball: **+1 overall**" >> "${commentfile}" else @@ -480,7 +478,7 @@ function github_finalreport i=0 until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | - ${SED} -e "s,@@BASE@@,${BUILD_URL}artifact/patchprocess,g") + ${SED} -e "s,@@BASE@@,${BUILD_URL}${BUILD_URL_ARTIFACTS},g") printf "%s\n" "${comment}" >> "${commentfile}" ((i=i+1)) done http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/test-patch.d/jira.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.d/jira.sh b/precommit/test-patch.d/jira.sh index 91167a9..bf6a598 100755 --- a/precommit/test-patch.d/jira.sh +++ b/precommit/test-patch.d/jira.sh @@ -139,7 +139,7 @@ function jira_locate_patch github_jira_bridge "${fileloc}" return $? elif [[ $(${GREP} -c "${JIRA_STATUS_RE}" "${PATCH_DIR}/jira") == 0 ]]; then - if [[ ${JENKINS} == true ]]; then + if [[ ${ROBOT} == true ]]; then yetus_error "ERROR: ${input} issue status is not matched with \"${JIRA_STATUS_RE}\"." cleanup_and_exit 1 else @@ -339,7 +339,7 @@ function jira_finalreport rm "${commentfile}" 2>/dev/null - if [[ ${JENKINS} == "false" + if [[ ${ROBOT} == "false" || ${OFFLINE} == true ]] ; then return 0 fi @@ -350,8 +350,6 @@ function jira_finalreport big_console_header "Adding comment to JIRA" - add_footer_table "Console output" "${BUILD_URL}console" - if [[ ${result} == 0 ]]; then echo "| (/) *{color:green}+1 overall{color}* |" >> "${commentfile}" else @@ -429,7 +427,7 @@ function jira_finalreport i=0 until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | - ${SED} -e "s,@@BASE@@,${BUILD_URL}artifact/patchprocess,g") + ${SED} -e "s,@@BASE@@,${BUILD_URL}${BUILD_URL_ARTIFACTS},g") printf "%s\n" "${comment}" >> "${commentfile}" ((i=i+1)) done http://git-wip-us.apache.org/repos/asf/yetus/blob/b6b6b0dd/precommit/test-patch.sh ---------------------------------------------------------------------- diff --git a/precommit/test-patch.sh b/precommit/test-patch.sh index 8fa7eee..ecc9925 100755 --- a/precommit/test-patch.sh +++ b/precommit/test-patch.sh @@ -73,11 +73,10 @@ function setup_defaults ALLOWSUMMARIES=true - DOCKERMODE=false - DOCKERSUPPORT=false - DOCKER_ENABLE_PRIVILEGED=true BUILD_NATIVE=${BUILD_NATIVE:-true} + BUILD_URL_ARTIFACTS=artifact/patchprocess + BUILD_URL_CONSOLE=console BUILDTOOLCWD=module # shellcheck disable=SC2034 @@ -471,9 +470,9 @@ function verify_patchdir_still_exists echo if [[ ${JENKINS} == true ]]; then if [[ -n ${NODE_NAME} ]]; then - extra=" (node ${NODE_NAME})" + extra=" (Jenkins node ${NODE_NAME})" fi - echo "Jenkins${extra} information at ${BUILD_URL} may provide some hints. " >> "${commentfile}" + echo "Jenkins${extra} information at ${BUILD_URL}${BUILD_URL_CONSOLE} may provide some hints. " >> "${commentfile}" write_comment ${commentfile} fi @@ -684,11 +683,6 @@ function yetus_usage yetus_add_option "--contrib-guide=<url>" "URL to point new users towards project conventions. (default: ${PATCH_NAMING_RULE} )" yetus_add_option "--debug" "If set, then output some extra stuff to stderr" yetus_add_option "--dirty-workspace" "Allow the local git workspace to have uncommitted changes" - yetus_add_option "--docker" "Spawn a docker container" - yetus_add_option "--dockercmd=<file>" "Command to use as docker executable (default: docker from path)" - yetus_add_option "--dockerfile=<file>" "Dockerfile fragment to use as the base" - yetus_add_option "--dockeronfail=<list>" "If Docker fails, determine fallback method order (default: ${DOCKERFAIL})" - yetus_add_option "--dockerprivd=<bool>" "Run docker in privileged mode (default: '${DOCKER_ENABLE_PRIVILEGED}')" yetus_add_option "--java-home=<path>" "Set JAVA_HOME (In Docker mode, this should be local to the image)" yetus_add_option "--linecomments=<bug>" "Only write line comments to this comma delimited list (defaults to bugcomments)" yetus_add_option "--list-plugins" "List all installed plug-ins and then exit" @@ -728,14 +722,27 @@ function yetus_usage yetus_reset_usage echo "" - echo "Jenkins-only options:" - yetus_add_option "--jenkins" "Jenkins mode" - yetus_add_option "--build-url" "Set the build location web page" - yetus_add_option "--mv-patch-dir" "Move the patch-dir into the basedir during cleanup." + echo "Automation options:" + yetus_add_option "--build-url=<url>" "Set the build location web page (Default: '${BUILD_URL}')" + yetus_add_option "--build-url-console=<location>" "Location relative to --build-url of the console (Default: '${BUILD_URL_CONSOLE}')" + yetus_add_option "--build-url-patchdir=<location>" "Location relative to --build-url of the --patch-dir (Default: '${BUILD_URL_ARTIFACTS}')" + yetus_add_option "--console-urls" "Use the build URL instead of path on the console report" + yetus_add_option "--instance=<string>" "Parallel execution identifier string" + yetus_add_option "--jenkins" "Enable Jenkins-specifc handling (auto: --robot)" + yetus_add_option "--mv-patch-dir" "Move the patch-dir into the basedir during cleanup" + yetus_add_option "--robot" "Assume this is an automated run" + yetus_add_option "--sentinel" "A very aggressive robot (auto: --robot)" yetus_generic_columnprinter "${YETUS_OPTION_USAGE[@]}" yetus_reset_usage + + echo "" + echo "Docker options:" + docker_usage + yetus_generic_columnprinter "${YETUS_OPTION_USAGE[@]}" + yetus_reset_usage + for plugin in ${BUILDTOOLS} ${TESTTYPES} ${BUGSYSTEMS} ${TESTFORMATS}; do if declare -f ${plugin}_usage >/dev/null 2>&1; then echo "" @@ -755,8 +762,8 @@ function yetus_usage ## @return May exit on failure function parse_args { - local i - local j + declare i + declare j common_args "$@" @@ -775,38 +782,32 @@ function parse_args --build-url=*) BUILD_URL=${i#*=} ;; + --build-url-artifacts=*) + # shellcheck disable=SC2034 + BUILD_URL_ARTIFACTS=${i#*=} + ;; + --build-url-console=*) + # shellcheck disable=SC2034 + BUILD_URL_CONSOLE=${i#*=} + ;; + --console-urls) + # shellcheck disable=SC2034 + CONSOLE_USE_BUILD_URL=true + ;; --contrib-guide=*) PATCH_NAMING_RULE=${i#*=} ;; --dirty-workspace) DIRTY_WORKSPACE=true ;; - --docker) - DOCKERSUPPORT=true - ;; - --dockercmd=*) - #shellcheck disable=SC2034 - DOCKERCMD=${i#*=} - ;; - --dockerfile=*) - DOCKERFILE=${i#*=} - ;; - --dockermode) - DOCKERMODE=true - ;; - --dockeronfail=*) - DOCKERFAIL=${i#*=} - ;; - --dockerprivd=*) - DOCKER_ENABLE_PRIVILEGED=${i#*=} + --instance=*) + INSTANCE=${i#*=} ;; --java-home=*) JAVA_HOME=${i#*=} ;; --jenkins) JENKINS=true - TEST_PARALLEL=${TEST_PARALLEL:-true} - INSTANCE=${EXECUTOR_NUMBER:-RANDOM} ;; --linecomments=*) BUGLINECOMMENTS=${i#*=} @@ -820,12 +821,13 @@ function parse_args --multijdkdirs=*) JDK_DIR_LIST=${i#*=} JDK_DIR_LIST=${JDK_DIR_LIST//,/ } - yetus_debug "Multi-JVM mode activated with ${JDK_DIR_LIST}" + yetus_debug "Multi-JDK mode activated with ${JDK_DIR_LIST}" + yetus_add_entry EXEC_MODES MultiJDK ;; --multijdktests=*) JDK_TEST_LIST=${i#*=} JDK_TEST_LIST=${JDK_TEST_LIST//,/ } - yetus_debug "Multi-JVM test list: ${JDK_TEST_LIST}" + yetus_debug "Multi-JDK test list: ${JDK_TEST_LIST}" ;; --mv-patch-dir) RELOCATE_PATCH_DIR=true; @@ -839,9 +841,17 @@ function parse_args --resetrepo) RESETREPO=true ;; + --robot) + ROBOT=true + ;; --run-tests) RUN_TESTS=true ;; + --sentinel) + # shellcheck disable=SC2034 + SENTINEL=true + yetus_add_entry EXEC_MODES Sentinel + ;; --skip-dirs=*) MODULE_SKIPDIRS=${i#*=} MODULE_SKIPDIRS=${MODULE_SKIPDIRS//,/ } @@ -851,6 +861,7 @@ function parse_args ALLOWSUMMARIES=${i#*=} ;; --test-parallel=*) + # shellcheck disable=SC2034 TEST_PARALLEL=${i#*=} ;; --test-threads=*) @@ -881,10 +892,26 @@ function parse_args esac done - if [[ "${DOCKERMODE}" = true ]]; then - add_vote_table 0 reexec "Docker mode activated." - elif [[ "${REEXECED}" = true ]]; then - add_vote_table 0 reexec "Precommit patch detected." + docker_parse_args "$@" + + if [[ -z "${PATCH_OR_ISSUE}" ]]; then + yetus_usage + exit 1 + fi + + if [[ ${JENKINS} = true ]]; then + ROBOT=true + INSTANCE=${EXECUTOR_NUMBER} + yetus_add_entry EXEC_MODES Jenkins + fi + + if [[ ${ROBOT} = true ]]; then + # shellcheck disable=SC2034 + TEST_PARALLEL=true + RESETREPO=true + RUN_TESTS=true + ISSUE=${PATCH_OR_ISSUE} + yetus_add_entry EXEC_MODES Robot fi if [[ -n ${REEXECLAUNCHTIMER} ]]; then @@ -893,9 +920,18 @@ function parse_args start_clock fi - if [[ -z "${PATCH_OR_ISSUE}" ]]; then - yetus_usage - exit 1 + if [[ "${DOCKERMODE}" = true || "${DOCKERSUPPORT}" = true ]]; then + if [[ "${DOCKER_DESTRCUTIVE}" = true ]]; then + yetus_add_entry EXEC_MODES DestructiveDocker + else + yetus_add_entry EXEC_MODES Docker + fi + add_vote_table 0 reexec "Docker mode activated." + start_clock + elif [[ "${REEXECED}" = true ]]; then + yetus_add_entry EXEC_MODES Re-exec + add_vote_table 0 reexec "Precommit patch detected." + start_clock fi # we need absolute dir for ${BASEDIR} @@ -920,17 +956,12 @@ function parse_args # we need absolute dir for PATCH_DIR PATCH_DIR=$(yetus_abs "${PATCH_DIR}") - if [[ ${JENKINS} == "true" ]]; then - echo "Running in Jenkins mode" - ISSUE=${PATCH_OR_ISSUE} - RESETREPO=true - else - if [[ ${RESETREPO} == "true" ]] ; then - echo "Running in destructive (--resetrepo) developer mode" - else - echo "Running in developer mode" - fi - JENKINS=false + if [[ ${RESETREPO} == "true" ]] ; then + yetus_add_entry EXEC_MODES ResetRepo + fi + + if [[ ${RUN_TESTS} == "true" ]] ; then + yetus_add_entry EXEC_MODES UnitTests fi if [[ -n "${USER_PLUGIN_DIR}" ]]; then @@ -1564,11 +1595,11 @@ function check_reexec apply_patch_file - if [[ ${JENKINS} == true ]]; then + if [[ ${ROBOT} == true ]]; then rm "${commentfile}" 2>/dev/null echo "(!) A patch to the testing environment has been detected. " > "${commentfile}" echo "Re-executing against the patched versions to perform further tests. " >> "${commentfile}" - echo "The console is at ${BUILD_URL}console in case of problems." >> "${commentfile}" + echo "The console is at ${BUILD_URL}${BUILD_URL_CONSOLE} in case of problems." >> "${commentfile}" write_comment "${commentfile}" rm "${commentfile}" fi @@ -2105,6 +2136,11 @@ function bugsystem_finalreport version=$(cat "${BINDIR}/VERSION") fi + if [[ "${ROBOT}" = true && + -n "${BUILD_URL}" && + -n "${BUILD_URL_CONSOLE}" ]]; then + add_footer_table "Console output" "${BUILD_URL}${BUILD_URL_CONSOLE}" + fi add_footer_table "Powered by" "Apache Yetus ${version} http://yetus.apache.org"; for bugs in ${BUGCOMMENTS}; do @@ -2123,7 +2159,7 @@ function cleanup_and_exit { local result=$1 - if [[ ${JENKINS} == "true" && ${RELOCATE_PATCH_DIR} == "true" && \ + if [[ ${ROBOT} == "true" && ${RELOCATE_PATCH_DIR} == "true" && \ -e ${PATCH_DIR} && -d ${PATCH_DIR} ]] ; then # if PATCH_DIR is already inside BASEDIR, then # there is no need to move it since we assume that @@ -2149,8 +2185,7 @@ function runtests { local plugin - ### Run tests for Jenkins or if explictly asked for by a developer - if [[ ${JENKINS} == "true" || ${RUN_TESTS} == "true" ]] ; then + if [[ ${RUN_TESTS} == "true" ]] ; then verify_patchdir_still_exists check_unittests @@ -2808,6 +2843,8 @@ function initialize cleanup_and_exit 1 fi + echo "Modes: ${EXEC_MODES}" + locate_patch # from here on out, we'll be in ${BASEDIR} for cwd
