YETUS-59. Verify any Cat X runtime dependencies are optional Signed-off-by: Allen Wittenauer <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/yetus/repo Commit: http://git-wip-us.apache.org/repos/asf/yetus/commit/dbeacad3 Tree: http://git-wip-us.apache.org/repos/asf/yetus/tree/dbeacad3 Diff: http://git-wip-us.apache.org/repos/asf/yetus/diff/dbeacad3 Branch: refs/heads/master Commit: dbeacad386f3a059d6bfc4b84ecd57b092701e0f Parents: 9575b65 Author: Allen Wittenauer <[email protected]> Authored: Tue Sep 29 06:12:04 2015 -0700 Committer: Allen Wittenauer <[email protected]> Committed: Thu Oct 15 20:19:22 2015 -0700 ---------------------------------------------------------------------- .../documentation/latest/precommit-advanced.md | 54 +- .../documentation/latest/precommit-basic.md | 37 +- dev-support/core.d/00-yetuslib.sh | 95 +++ dev-support/core.d/01-common.sh | 468 ++++++++++++++ dev-support/core.d/builtin-bugsystem.sh | 149 +++++ dev-support/core.d/builtin-personality.sh | 29 + dev-support/core.d/common.sh | 602 ------------------- dev-support/core.d/patchfiles.sh | 326 ++++++++++ dev-support/personality/bigtop.sh | 31 +- dev-support/personality/flink.sh | 25 +- dev-support/personality/hadoop.sh | 29 +- dev-support/personality/hbase.sh | 37 +- dev-support/personality/kafka.sh | 25 +- dev-support/personality/pig.sh | 25 +- dev-support/personality/samza.sh | 26 +- dev-support/personality/tajo.sh | 21 +- dev-support/personality/tez.sh | 21 +- dev-support/smart-apply-patch.sh | 28 +- .../test-patch-docker/test-patch-docker.sh | 2 +- dev-support/test-patch.d/asflicense.sh | 2 +- dev-support/test-patch.d/author.sh | 2 +- dev-support/test-patch.d/builtin-bugsystem.sh | 171 ------ dev-support/test-patch.d/builtin-personality.sh | 25 - dev-support/test-patch.d/checkstyle.sh | 2 +- dev-support/test-patch.d/findbugs.sh | 33 +- dev-support/test-patch.d/java.sh | 4 +- dev-support/test-patch.d/maven.sh | 6 +- dev-support/test-patch.d/perlcritic.sh | 2 +- dev-support/test-patch.d/pylint.sh | 2 +- dev-support/test-patch.d/rubocop.sh | 2 +- dev-support/test-patch.d/ruby-lint.sh | 2 +- dev-support/test-patch.d/scala.sh | 4 +- dev-support/test-patch.d/shellcheck.sh | 2 +- dev-support/test-patch.d/test4tests.sh | 2 +- dev-support/test-patch.d/whitespace.sh | 2 +- dev-support/test-patch.d/xml.sh | 2 +- dev-support/test-patch.sh | 124 ++-- 37 files changed, 1394 insertions(+), 1025 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/asf-site-src/source/documentation/latest/precommit-advanced.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/latest/precommit-advanced.md b/asf-site-src/source/documentation/latest/precommit-advanced.md index 13d80da..ad9ac34 100644 --- a/asf-site-src/source/documentation/latest/precommit-advanced.md +++ b/asf-site-src/source/documentation/latest/precommit-advanced.md @@ -22,7 +22,7 @@ test-patch * [Docker Support](#Docker_Support) * [Plug-ins](#Plug-ins) -* [Configuring for Other Projects](#Configuring_for_Other_Projects) +* [Personalities](#Personalities) * [Important Variables](#Important_Variables) # Docker Support @@ -38,17 +38,17 @@ Dockerfile images will be named with a test-patch prefix and suffix with either # Plug-ins -test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d off of the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary. Different plug-ins have specific functions for that particular functionality. In this document, the common functions as well as test functions are covered. See other documentat for pertinent information for the other plug-in types. +test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d off of the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary. Different plug-ins have specific functions for that particular functionality. In this document, the common functions available to all/most plug-ins are covered. Test plugins are covered below. See other documentation for pertinent information for the other plug-in types. ## Common Plug-in Functions Every plug-in must have one line in order to be recognized, usually an 'add' statement. Test plug-ins, for example, have this statement: ```bash -add_plugin <pluginname> +add_test_type <pluginname> ``` -This function call registers the `pluginname` so that test-patch knows that it exists. The `pluginname` also acts as the key to the custom functions that you can define. For example: +This function call registers the `pluginname` so that test-patch knows that it exists. Plug-in names must be unique across all the different plug-in types. Additionally, the 'all' plug-in is reserved. The `pluginname` also acts as the key to the custom functions that you can define. For example: ```bash function pluginname_filefilter @@ -84,14 +84,26 @@ Similarly, there are other functions that may be defined during the test-patch r * pluginname\_rebuild - Any non-unit tests that require the source to be rebuilt in a destructive way should be run here. -Test Plug-ins -============= -Plugins geared towards independent tests are registed via: +## Plug-in Importation + +Plug-ins are imported from several key directories: + +* core.d is an internal-to-Yetus directory that first loads the basic Yetus library, followed by the common routines used by all of the precommit shell code. This order is dictated by prefixing the plug-in files with a number. Other files in this directory are loaded in shell collated order. + +* personality contains bundled personalities for various projects. These will be imported individually based upon either a project name or if specifically identified with the `--personality` flag. + +* test-patch.d contains all of the optional, bundled plug-ins. These are imported last and in shell collated order. + +If the `--skip-system-plugins` flag is passed, then only core.d is imported. + +## Test Plug-ins + +Plug-ins geared towards independent tests are registed via: ```bash -add_plugin <pluginname> +add_test_type <pluginname> ``` + pluginname\_filefilter @@ -101,7 +113,9 @@ add_plugin <pluginname> - executed after the unit tests have completed. -# Configuring for Other Projects +# Personalities + +## Configuring for Other Projects It is impossible for any general framework to be predictive about what types of special rules any given project may have, especially when it comes to ordering and Maven profiles. In order to direct test-patch to do the correct action, a project `personality` should be added that enacts these custom rules. @@ -109,6 +123,18 @@ A personality consists of two functions. One that determines which test types to There can be only **one** of each personality function defined. +## Global Definitions + +Globals for personalities should be defined in the `personality_globals` function. This function is called *after* the other plug-ins have been imported. This allows one to configure any settings for plug-ins that have been imported safely: + +```bash +funciton personality_globals +{ + PATCH_BRANCH_DEFAULT=master + GITHUB_REPO="apache/yetus" +} +``` + ## Test Determination The `personality_file_tests` function determines which tests to turn on based upon the file name. It is relatively simple. For example, to turn on a full suite of tests for Java files: @@ -180,6 +206,16 @@ function personality_modules This function will tell test-patch that when the javadoc test is being run, do the documentation build at the base of the source repository and make sure the -DskipTests flag is passed to our build tool. +## Enabling Plug-ins + +Personalities can set the base list of plug-ins to enable and disable for their project via the `personality_plugins` function. Just call it with the same pattern as the `--plugins` command line option: + +```bash +personality_plugins "all,-checkstyle,-findbugs,-asflicense" +``` + +This list is used if the user does not provide a list of plug-ins. + # Important Variables There are a handful of extremely important system variables that make life easier for personality and plug-in writers. Other variables may be provided by individual plug-ins. Check their development documentation for more information. http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/asf-site-src/source/documentation/latest/precommit-basic.md ---------------------------------------------------------------------- diff --git a/asf-site-src/source/documentation/latest/precommit-basic.md b/asf-site-src/source/documentation/latest/precommit-basic.md index 5278b74..bcf4fbc 100644 --- a/asf-site-src/source/documentation/latest/precommit-basic.md +++ b/asf-site-src/source/documentation/latest/precommit-basic.md @@ -62,8 +62,7 @@ test-patch requires these installed components to execute: ## Optional Requirements -Features are plug-in based. These are activated based upon tool availability, the languages being -tested, etc. +Features are plug-in based and enabled either individually or collectively on the command line. From there, these are activated based upon tool availability, the languages being tested, etc. The external dependencies of plug-ins may have different licensing requirements than Apache Yetus. Bug Systems: @@ -88,11 +87,39 @@ Language Support, Licensing, and more: # Basic Usage +The first step for a successful deployment is determining which features/plug-ins to enable: + +```bash +$ test-patch.sh --list-plugins +``` + +This option will list all of the available plug-ins that are installed in the default location. From this list, the specific plug-ins can be enabled: + +```bash +$ test-patch.sh --plugins="ant,maven,shellcheck,xml" <other options> +``` + +As a short-cut, every plug-in may be enabled via the special 'all' type: + +```bash +$ test-patch.sh --plugins="all" <other options> +``` + +`--plugins` also allows some basic "arithmetic": + +```bash +$ test-patch.sh --plugins="all,-checkstyle,-findbugs" <other options> +``` + +This will enable all plug-ins for potential usage, except for checkstyle and findbugs. + +**NOTE: The examples in this section will assume that the necessary `--plugins` option has been set on the command line as approriate for your particular installation.** + This command will execute basic patch testing against a patch file stored in "filename": ```bash $ cd <your repo> -$ dev-support/test-patch.sh --dirty-workspace --project=projectname <filename> +$ test-patch.sh --dirty-workspace --project=projectname <filename> ``` The `--dirty-workspace` flag tells test-patch that the repository is not clean and it is ok to continue. By default, unit tests are not run since they may take a significant amount of time. @@ -102,7 +129,7 @@ To do turn them on, we need to provide the --run-tests option: ```bash $ cd <your repo> -$ dev-support/test-patch.sh --dirty-workspace --run-tests <filename> +$ test-patch.sh --dirty-workspace --run-tests <filename> ``` This is the same command, but now runs the unit tests. @@ -113,7 +140,7 @@ A typical configuration is to have two repositories. One with the code you are $ cd <workrepo> $ git diff master > /tmp/patchfile $ cd ../<testrepo> -$ <workrepo>/dev-support/test-patch.sh --basedir=<testrepo> --resetrepo /tmp/patchfile +$ test-patch.sh --basedir=<testrepo> --resetrepo /tmp/patchfile ``` We used two new options here. --basedir sets the location of the repository to use for testing. --resetrepo tells test patch that it can go into **destructive** mode. Destructive mode will wipe out any changes made to that repository, so use it with care! http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/core.d/00-yetuslib.sh ---------------------------------------------------------------------- diff --git a/dev-support/core.d/00-yetuslib.sh b/dev-support/core.d/00-yetuslib.sh new file mode 100644 index 0000000..6ab6682 --- /dev/null +++ b/dev-support/core.d/00-yetuslib.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +## @description Print a message to stderr +## @audience public +## @stability stable +## @replaceable no +## @param string +function yetus_error +{ + echo "$*" 1>&2 +} + +## @description Print a message to stderr if --debug is turned on +## @audience public +## @stability stable +## @replaceable no +## @param string +function yetus_debug +{ + if [[ "${YETUS_SHELL_SCRIPT_DEBUG}" = true ]]; then + echo "[$(date) DEBUG]: $*" 1>&2 + fi +} + +## @description Given variable $1 delete $2 from it +## @audience public +## @stability stable +## @replaceable no +function yetus_delete_entry +{ + if [[ ${!1} =~ \ ${2}\ ]] ; then + yetus_debug "Removing ${2} from ${1}" + eval "${1}"=\""${!1// ${2} }"\" + fi +} + +## @description Given variable $1 add $2 to it +## @audience public +## @stability stable +## @replaceable no +function yetus_add_entry +{ + if [[ ! ${!1} =~ \ ${2}\ ]] ; then + yetus_debug "Adding ${2} to ${1}" + #shellcheck disable=SC2140 + eval "${1}"=\""${!1} ${2} "\" + fi +} + +## @description Given variable $1 determine if $2 is in it +## @audience public +## @stability stable +## @replaceable no +## @returns 1 = yes, 0 = no +function yetus_verify_entry +{ + [[ ! ${!1} =~ \ ${2}\ ]] +} + +## @description run the command, sending stdout and stderr to the given filename +## @audience public +## @stability stable +## @param filename +## @param command +## @param [..] +## @replaceable no +## @returns $? +function yetus_run_and_redirect +{ + declare logfile=$1 + shift + + # to the log + { + date + echo "cd $(pwd)" + echo "${*}" + } >> "${logfile}" + # run the actual command + "${@}" >> "${logfile}" 2>&1 +} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/core.d/01-common.sh ---------------------------------------------------------------------- diff --git a/dev-support/core.d/01-common.sh b/dev-support/core.d/01-common.sh new file mode 100755 index 0000000..5e0f29b --- /dev/null +++ b/dev-support/core.d/01-common.sh @@ -0,0 +1,468 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +## @description Setup the default global variables +## @audience public +## @stability stable +## @replaceable no +function common_defaults +{ + #shellcheck disable=SC2034 + BASEDIR=$(pwd) + BUGSYSTEMS="" + BUILDTOOLS="" + LOAD_SYSTEM_PLUGINS=true + #shellcheck disable=SC2034 + JENKINS=false + #shellcheck disable=SC2034 + OFFLINE=false + OSTYPE=$(uname -s) + #shellcheck disable=SC2034 + PATCH_BRANCH="" + PATCH_BRANCH_DEFAULT="master" + #shellcheck disable=SC2034 + PATCH_DRYRUNMODE=false + PATCH_DIR=/tmp + while [[ -e ${PATCH_DIR} ]]; do + PATCH_DIR=/tmp/yetus-${RANDOM}.${RANDOM} + done + #shellcheck disable=SC2034 + PATCH_METHOD="" + #shellcheck disable=SC2034 + PATCH_METHODS=("gitapply" "patchcmd") + #shellcheck disable=SC2034 + PATCH_LEVEL=0 + PROJECT_NAME=yetus + RESULT=0 + #shellcheck disable=SC2034 + TESTTYPES="" + TESTFORMATS="" + USER_PLUGIN_DIR="" + + #shellcheck disable=SC2034 + YETUS_SHELL_SCRIPT_DEBUG=false + + # Solaris needs POSIX and GNU, not SVID + case ${OSTYPE} in + SunOS) + AWK=${AWK:-/usr/xpg4/bin/awk} + CURL=${CURL:-curl} + DIFF=${DIFF:-/usr/gnu/bin/diff} + FILE=${FILE:-file} + GIT=${GIT:-git} + GREP=${GREP:-/usr/xpg4/bin/grep} + PATCH=${PATCH:-/usr/gnu/bin/patch} + SED=${SED:-/usr/xpg4/bin/sed} + ;; + *) + AWK=${AWK:-awk} + CURL=${CURL:-curl} + DIFF=${DIFF:-diff} + FILE=${FILE:-file} + GIT=${GIT:-git} + GREP=${GREP:-grep} + PATCH=${PATCH:-patch} + SED=${SED:-sed} + ;; + esac +} + +## @description Interpret the common command line parameters used by test-patch, +## @description smart-apply-patch, and the bug system plugins +## @audience private +## @stability stable +## @replaceable no +## @params $@ +## @return May exit on failure +function common_args +{ + declare i + + for i in "$@"; do + case ${i} in + --awk-cmd=*) + AWK=${i#*=} + ;; + --basedir=*) + #shellcheck disable=SC2034 + BASEDIR=${i#*=} + ;; + --branch=*) + #shellcheck disable=SC2034 + PATCH_BRANCH=${i#*=} + ;; + --branch-default=*) + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=${i#*=} + ;; + --curl-cmd=*) + CURL=${i#*=} + ;; + --debug) + #shellcheck disable=SC2034 + YETUS_SHELL_SCRIPT_DEBUG=true + ;; + --diff-cmd=*) + DIFF=${i#*=} + ;; + --file-cmd=*) + FILE=${i#*=} + ;; + --git-cmd=*) + GIT=${i#*=} + ;; + --grep-cmd=*) + GREP=${i#*=} + ;; + --help|-help|-h|help|--h|--\?|-\?|\?) + yetus_usage + exit 0 + ;; + --list-plugins) + list_plugins + exit 0 + ;; + --offline) + #shellcheck disable=SC2034 + OFFLINE=true + ;; + --patch-cmd=*) + PATCH=${i#*=} + ;; + --patch-dir=*) + PATCH_DIR=${i#*=} + ;; + --plugins=*) + ENABLED_PLUGINS=${i#*=} + ENABLED_PLUGINS=${ENABLED_PLUGINS//,/ } + ;; + --project=*) + PROJECT_NAME=${i#*=} + ;; + --skip-system-plugins) + LOAD_SYSTEM_PLUGINS=false + ;; + --sed-cmd=*) + SED=${i#*=} + ;; + --user-plugins=*) + USER_PLUGIN_DIR=${i#*=} + ;; + *) + ;; + esac + done +} + +## @description List all installed plug-ins, regardless of whether +## @description they have been enabled +## @audience public +## @stability evolving +## @replaceable no +function list_plugins +{ + declare plugintype + declare name + + ENABLED_PLUGINS="all" + importplugins + + printf "Reminder: every plug-in may be enabled via 'all'.\n\n" + for plugintype in BUILDTOOLS TESTTYPES BUGSYSTEMS TESTFORMATS; do + printf "%s:\n\t" ${plugintype} + for name in ${!plugintype}; do + printf "%s " ${name} + done + echo "" + done +} + +## @description Let plugins also get a copy of the arguments +## @audience private +## @stability evolving +## @replaceable no +function parse_args_plugins +{ + declare plugin + + for plugin in ${PLUGINS} ${BUGSYSTEMS} ${TESTFORMATS} ${BUILDTOOLS}; do + if declare -f ${plugin}_parse_args >/dev/null 2>&1; then + yetus_debug "Running ${plugin}_parse_args" + #shellcheck disable=SC2086 + ${plugin}_parse_args "$@" + (( RESULT = RESULT + $? )) + fi + done +} + +## @description Let plugins also get a copy of the arguments +## @audience private +## @stability evolving +## @replaceable no +function plugins_initialize +{ + declare plugin + + for plugin in ${PLUGINS} ${BUGSYSTEMS} ${TESTFORMATS} ${BUILDTOOLS}; do + if declare -f ${plugin}_initialize >/dev/null 2>&1; then + yetus_debug "Running ${plugin}_initialize" + #shellcheck disable=SC2086 + ${plugin}_initialize + (( RESULT = RESULT + $? )) + fi + done +} + +## @description Determine if a plugin was enabeld by the user +## @description ENABLED_PLUGINS must be defined +## @audience public +## @stability stable +## @replaceable yes +## @param test +function verify_plugin_enabled +{ + declare toadd=$1 + declare bar + declare idx + declare strip + declare stridx + + yetus_debug "Testing if $1 has been enabled by user" + + bar="" + for idx in ${ENABLED_PLUGINS}; do + stridx=${idx// } + yetus_debug "verify_plugin_enabled: processing ${stridx}" + case ${stridx} in + all) + bar=${toadd} + ;; + -*) + strip=${stridx#-} + if [[ ${strip} = "${toadd}" ]]; then + bar="" + fi + ;; + +*|*) + strip=${stridx#+} + if [[ ${strip} = "${toadd}" ]]; then + bar=${toadd} + fi + ;; + esac + done + + if [[ -n ${bar} ]]; then + yetus_debug "Post-parsing: checking ${bar} = ${toadd}" + fi + [[ ${bar} = "${toadd}" ]] +} + +## @description Personality-defined plug-in list +## @audience public +## @stability stable +## @replaceable yes +## @param plug-in list string +function personality_plugins +{ + if [[ -z "${ENABLED_PLUGINS}" ]]; then + ENABLED_PLUGINS="$1" + yetus_debug "Using personality plug-in list: ${ENABLED_PLUGINS}" + fi +} + +## @description Add the given test type +## @audience public +## @stability stable +## @replaceable yes +## @param test +function add_test +{ + if verify_plugin_enabled "${1}"; then + yetus_add_entry NEEDED_TESTS "${1}" + fi +} + +## @description Remove the given test type +## @audience public +## @stability stable +## @replaceable yes +## @param test +function delete_test +{ + yetus_delete_entry NEEDED_TESTS "${1}" +} + +## @description Verify if a given test was requested +## @audience public +## @stability stable +## @replaceable yes +## @param test +## @return 1 = yes +## @return 0 = no +function verify_needed_test +{ + yetus_verify_entry NEEDED_TESTS "${1}" +} + +## @description Add the given test type +## @audience public +## @stability stable +## @replaceable yes +## @param plugin +function add_test_type +{ + if verify_plugin_enabled "${1}"; then + yetus_add_entry TESTTYPES "${1}" + fi +} + +## @description Remove the given test type +## @audience public +## @stability stable +## @replaceable yes +## @param plugin +function delete_test_type +{ + yetus_delete_entry TESTTYPES "${1}" +} + +## @description Add the given bugsystem type +## @audience public +## @stability stable +## @replaceable yes +## @param bugsystem +function add_bugsystem +{ + if verify_plugin_enabled "${1}"; then + yetus_add_entry BUGSYSTEMS "${1}" + fi +} + +## @description Remove the given bugsystem type +## @audience public +## @stability stable +## @replaceable yes +## @param bugsystem +function delete_bugsystem +{ + yetus_delete_entry BUGSYSTEMS "${1}" +} + +## @description Add the given test format type +## @audience public +## @stability stable +## @replaceable yes +## @param test format +function add_test_format +{ + if verify_plugin_enabled "${1}"; then + yetus_add_entry TESTFORMATS "${1}" + fi +} + +## @description Remove the given test format type +## @audience public +## @stability stable +## @replaceable yes +## @param test format +function delete_test_format +{ + yetus_delete_entry TESTFORMATS "${1}" +} + +## @description Add the given build tool type +## @audience public +## @stability stable +## @replaceable yes +## @param build tool +function add_build_tool +{ + if verify_plugin_enabled "${1}"; then + yetus_add_entry BUILDTOOLS "${1}" + fi +} + +## @description Remove the given build tool type +## @audience public +## @stability stable +## @replaceable yes +## @param build tool +function delete_build_tool +{ + yetus_delete_entry BUILDTOOLS "${1}" +} + +## @description Import content from test-patch.d and optionally +## @description from user provided plugin directory +## @audience private +## @stability evolving +## @replaceable no +## @description Import content from test-patch.d and optionally +## @description from user provided plugin directory +## @audience private +## @stability evolving +## @replaceable no +function importplugins +{ + local i + local plugin + local files=() + + if [[ ${LOAD_SYSTEM_PLUGINS} == "true" ]]; then + if [[ -d "${BINDIR}/test-patch.d" ]]; then + files=(${BINDIR}/test-patch.d/*.sh) + fi + fi + + if [[ -n "${USER_PLUGIN_DIR}" && -d "${USER_PLUGIN_DIR}" ]]; then + yetus_debug "Loading user provided plugins from ${USER_PLUGIN_DIR}" + files=("${files[@]}" ${USER_PLUGIN_DIR}/*.sh) + fi + + if [[ -z ${PERSONALITY} + && -f "${BINDIR}/personality/${PROJECT_NAME}.sh" + && ${LOAD_SYSTEM_PLUGINS} = "true" ]]; then + PERSONALITY="${BINDIR}/personality/${PROJECT_NAME}.sh" + fi + + if [[ -n ${PERSONALITY} ]]; then + if [[ ! -f ${PERSONALITY} ]]; then + if [[ -f "${BINDIR}/personality/${PROJECT_NAME}.sh" + && ${LOAD_SYSTEM_PLUGINS} = "true" ]]; then + PERSONALITY="${BINDIR}/personality/${PROJECT_NAME}.sh" + else + yetus_debug "Can't find ${PERSONALITY} to import." + return + fi + fi + yetus_debug "Importing ${PERSONALITY}" + # shellcheck disable=SC1090 + . "${PERSONALITY}" + fi + + for i in "${files[@]}"; do + if [[ -f ${i} ]]; then + yetus_debug "Importing ${i}" + #shellcheck disable=SC1090 + . "${i}" + fi + done + + if declare -f personality_globals > /dev/null; then + personality_globals + fi +} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/core.d/builtin-bugsystem.sh ---------------------------------------------------------------------- diff --git a/dev-support/core.d/builtin-bugsystem.sh b/dev-support/core.d/builtin-bugsystem.sh new file mode 100755 index 0000000..f6e25f1 --- /dev/null +++ b/dev-support/core.d/builtin-bugsystem.sh @@ -0,0 +1,149 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# This bug system handles the output on the screen. + +add_bugsystem console + +## @description Print out the finished details on the console +## @audience private +## @stability evolving +## @replaceable no +## @param runresult +## @return 0 on success +## @return 1 on failure +function console_finalreport +{ + declare result=$1 + shift + declare i=0 + declare ourstring + declare vote + declare subs + declare ela + declare comment + declare commentfile1="${PATCH_DIR}/comment.1" + declare commentfile2="${PATCH_DIR}/comment.2" + declare normaltop + declare line + declare seccoladj=0 + declare spcfx=${PATCH_DIR}/spcl.txt + + if [[ ${result} == 0 ]]; then + if [[ ${JENKINS} == false ]]; then + if declare -f ${PROJECT_NAME}_console_success >/dev/null; then + ${PROJECT_NAME}_console_success > "${spcfx}" + else + { + printf "IF9fX18gICAgICAgICAgICAgICAgICAgICAgICAgICAgICBfIAovIF9fX3wg"; + printf "XyAgIF8gIF9fXyBfX18gX19fICBfX18gX19ffCB8ClxfX18gXHwgfCB8IHwv"; + printf "IF9fLyBfXy8gXyBcLyBfXy8gX198IHwKIF9fXykgfCB8X3wgfCAoX3wgKF98"; + printf "ICBfXy9cX18gXF9fIFxffAp8X19fXy8gXF9fLF98XF9fX1xfX19cX19ffHxf"; + printf "X18vX19fKF8pCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg"; + printf "ICAK"; + } > "${spcfx}" + fi + fi + printf "\n\n+1 overall\n\n" + else + if [[ ${JENKINS} == false ]]; then + if declare -f ${PROJECT_NAME}_console_failure >/dev/null; then + ${PROJECT_NAME}_console_failure > "${spcfx}" + else + { + printf "IF9fX19fICAgICBfIF8gICAgICAgICAgICAgICAgXyAKfCAgX19ffF8gXyhf"; + printf "KSB8XyAgIF8gXyBfXyBfX198IHwKfCB8XyAvIF9gIHwgfCB8IHwgfCB8ICdf"; + printf "Xy8gXyBcIHwKfCAgX3wgKF98IHwgfCB8IHxffCB8IHwgfCAgX18vX3wKfF98"; + printf "ICBcX18sX3xffF98XF9fLF98X3wgIFxfX18oXykKICAgICAgICAgICAgICAg"; + printf "ICAgICAgICAgICAgICAgICAK" + } > "${spcfx}" + fi + fi + printf "\n\n-1 overall\n\n" + fi + + if [[ -f ${spcfx} ]]; then + if which base64 >/dev/null 2>&1; then + base64 --decode "${spcfx}" 2>/dev/null + elif which openssl >/dev/null 2>&1; then + openssl enc -A -d -base64 -in "${spcfx}" 2>/dev/null + fi + echo + echo + rm "${spcfx}" + fi + + seccoladj=$(findlargest 2 "${TP_VOTE_TABLE[@]}") + if [[ ${seccoladj} -lt 10 ]]; then + seccoladj=10 + fi + + seccoladj=$((seccoladj + 2 )) + i=0 + until [[ $i -eq ${#TP_HEADER[@]} ]]; do + printf "%s\n" "${TP_HEADER[${i}]}" + ((i=i+1)) + done + + printf "| %s | %*s | %s | %s\n" "Vote" ${seccoladj} Subsystem Runtime "Comment" + echo "============================================================================" + i=0 + until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do + ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ') + vote=$(echo "${ourstring}" | cut -f2 -d\|) + subs=$(echo "${ourstring}" | cut -f3 -d\|) + ela=$(echo "${ourstring}" | cut -f4 -d\|) + comment=$(echo "${ourstring}" | cut -f5 -d\|) + + echo "${comment}" | fold -s -w $((78-seccoladj-22)) > "${commentfile1}" + normaltop=$(head -1 "${commentfile1}") + ${SED} -e '1d' "${commentfile1}" > "${commentfile2}" + + printf "| %4s | %*s | %-10s |%-s\n" "${vote}" ${seccoladj} \ + "${subs}" "${ela}" "${normaltop}" + while read -r line; do + printf "| | %*s | | %-s\n" ${seccoladj} " " "${line}" + done < "${commentfile2}" + + ((i=i+1)) + rm "${commentfile2}" "${commentfile1}" 2>/dev/null + done + + if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then + seccoladj=$(findlargest 1 "${TP_TEST_TABLE[@]}") + printf "\n\n%*s | Tests\n" "${seccoladj}" "Reason" + i=0 + until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do + ourstring=$(echo "${TP_TEST_TABLE[${i}]}" | tr -s ' ') + vote=$(echo "${ourstring}" | cut -f2 -d\|) + subs=$(echo "${ourstring}" | cut -f3 -d\|) + printf "%*s | %s\n" "${seccoladj}" "${vote}" "${subs}" + ((i=i+1)) + done + fi + + printf "\n\n|| Subsystem || Report/Notes ||\n" + echo "============================================================================" + i=0 + + until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do + comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | + ${SED} -e "s,@@BASE@@,${PATCH_DIR},g") + printf "%s\n" "${comment}" + ((i=i+1)) + done +} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/core.d/builtin-personality.sh ---------------------------------------------------------------------- diff --git a/dev-support/core.d/builtin-personality.sh b/dev-support/core.d/builtin-personality.sh new file mode 100755 index 0000000..1258d46 --- /dev/null +++ b/dev-support/core.d/builtin-personality.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +function personality_modules +{ + if declare -f "${BUILDTOOL}_builtin_personality_modules" >/dev/null; then + "${BUILDTOOL}_builtin_personality_modules" "$@" + fi +} + +function personality_file_tests +{ + if declare -f "${BUILDTOOL}_builtin_personality_file_tests" >/dev/null; then + "${BUILDTOOL}_builtin_personality_file_tests" "$@" + fi +} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/core.d/common.sh ---------------------------------------------------------------------- diff --git a/dev-support/core.d/common.sh b/dev-support/core.d/common.sh deleted file mode 100755 index 7fe108c..0000000 --- a/dev-support/core.d/common.sh +++ /dev/null @@ -1,602 +0,0 @@ -#!/usr/bin/env bash -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -## @description Setup the default global variables -## @audience public -## @stability stable -## @replaceable no -function common_defaults -{ - #shellcheck disable=SC2034 - BASEDIR=$(pwd) - LOAD_SYSTEM_PLUGINS=true - #shellcheck disable=SC2034 - JENKINS=false - #shellcheck disable=SC2034 - OFFLINE=false - OSTYPE=$(uname -s) - #shellcheck disable=SC2034 - PATCH_BRANCH="" - PATCH_BRANCH_DEFAULT="master" - #shellcheck disable=SC2034 - PATCH_DRYRUNMODE=false - PATCH_DIR=/tmp - while [[ -e ${PATCH_DIR} ]]; do - PATCH_DIR=/tmp/yetus-${RANDOM}.${RANDOM} - done - PATCH_METHOD="" - PATCH_METHODS=("gitapply" "patchcmd") - PATCH_LEVEL=0 - PROJECT_NAME=yetus - RESULT=0 - USER_PLUGIN_DIR="" - #shellcheck disable=SC2034 - YETUS_SHELL_SCRIPT_DEBUG=false - - # Solaris needs POSIX and GNU, not SVID - case ${OSTYPE} in - SunOS) - AWK=${AWK:-/usr/xpg4/bin/awk} - CURL=${CURL:-curl} - DIFF=${DIFF:-/usr/gnu/bin/diff} - FILE=${FILE:-file} - GIT=${GIT:-git} - GREP=${GREP:-/usr/xpg4/bin/grep} - PATCH=${PATCH:-/usr/gnu/bin/patch} - SED=${SED:-/usr/xpg4/bin/sed} - ;; - *) - AWK=${AWK:-awk} - CURL=${CURL:-curl} - DIFF=${DIFF:-diff} - FILE=${FILE:-file} - GIT=${GIT:-git} - GREP=${GREP:-grep} - PATCH=${PATCH:-patch} - SED=${SED:-sed} - ;; - esac -} - -## @description Interpret the common command line parameters used by test-patch, -## @description smart-apply-patch, and the bug system plugins -## @audience private -## @stability stable -## @replaceable no -## @params $@ -## @return May exit on failure -function common_args -{ - declare i - - for i in "$@"; do - case ${i} in - --awk-cmd=*) - AWK=${i#*=} - ;; - --basedir=*) - #shellcheck disable=SC2034 - BASEDIR=${i#*=} - ;; - --branch=*) - #shellcheck disable=SC2034 - PATCH_BRANCH=${i#*=} - ;; - --branch-default=*) - #shellcheck disable=SC2034 - PATCH_BRANCH_DEFAULT=${i#*=} - ;; - --curl-cmd=*) - CURL=${i#*=} - ;; - --debug) - #shellcheck disable=SC2034 - YETUS_SHELL_SCRIPT_DEBUG=true - ;; - --diff-cmd=*) - DIFF=${i#*=} - ;; - --file-cmd=*) - FILE=${i#*=} - ;; - --git-cmd=*) - GIT=${i#*=} - ;; - --grep-cmd=*) - GREP=${i#*=} - ;; - --help|-help|-h|help|--h|--\?|-\?|\?) - yetus_usage - exit 0 - ;; - --modulelist=*) - USER_MODULE_LIST=${i#*=} - USER_MODULE_LIST=${USER_MODULE_LIST//,/ } - yetus_debug "Manually forcing modules ${USER_MODULE_LIST}" - ;; - --offline) - #shellcheck disable=SC2034 - OFFLINE=true - ;; - --patch-cmd=*) - PATCH=${i#*=} - ;; - --patch-dir=*) - PATCH_DIR=${i#*=} - ;; - --plugins=*) - USER_PLUGIN_DIR=${i#*=} - ;; - --project=*) - PROJECT_NAME=${i#*=} - ;; - --skip-system-plugins) - LOAD_SYSTEM_PLUGINS=false - ;; - --sed-cmd=*) - SED=${i#*=} - ;; - *) - ;; - esac - done -} - -## @description Print a message to stderr -## @audience public -## @stability stable -## @replaceable no -## @param string -function yetus_error -{ - echo "$*" 1>&2 -} - -## @description Print a message to stderr if --debug is turned on -## @audience public -## @stability stable -## @replaceable no -## @param string -function yetus_debug -{ - if [[ "${YETUS_SHELL_SCRIPT_DEBUG}" = true ]]; then - echo "[$(date) DEBUG]: $*" 1>&2 - fi -} - -## @description run the command, sending stdout and stderr to the given filename -## @audience public -## @stability stable -## @param filename -## @param command -## @param [..] -## @replaceable no -## @returns $? -function yetus_run_and_redirect -{ - declare logfile=$1 - shift - - # to the log - { - date - echo "cd $(pwd)" - echo "${*}" - } >> "${logfile}" - # run the actual command - "${@}" >> "${logfile}" 2>&1 -} - -## @description Given a possible patch file, guess if it's a patch file -## @description only using the more intense verify if we really need to -## @audience private -## @stability evolving -## @param path to patch file to test -## @return 0 we think it's a patch file -## @return 1 we think it's not a patch file -function guess_patch_file -{ - declare patch=$1 - declare fileOutput - - if [[ ! -f ${patch} ]]; then - return 1 - fi - - yetus_debug "Trying to guess if ${patch} is a patch file." - fileOutput=$("${FILE}" "${patch}") - if [[ $fileOutput =~ \ diff\ ]]; then - yetus_debug "file magic says it's a diff." - return 0 - fi - - fileOutput=$(head -n 1 "${patch}" | "${GREP}" -E "^(From [a-z0-9]* Mon Sep 17 00:00:00 2001)|(diff .*)|(Index: .*)$") - if [[ $? == 0 ]]; then - yetus_debug "first line looks like a patch file." - return 0 - fi - - patchfile_dryrun_driver "${patch}" -} - -## @description Given ${PATCH_OR_ISSUE}, determine what type of patch file is in use, -## @description and do the necessary work to place it into ${PATCH_DIR}/patch. -## @audience private -## @stability evolving -## @replaceable no -## @return 0 on success -## @return 1 on failure, may exit -function locate_patch -{ - declare bugsys - declare patchfile="" - declare gotit=false - - yetus_debug "locate patch" - - # it's a declarely provided file - if [[ -f ${PATCH_OR_ISSUE} ]]; then - patchfile="${PATCH_OR_ISSUE}" - else - # run through the bug systems. maybe they know? - for bugsys in ${BUGSYSTEMS}; do - if declare -f ${bugsys}_locate_patch >/dev/null 2>&1; then - "${bugsys}_locate_patch" "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch" - if [[ $? == 0 ]]; then - gotit=true - fi - fi - done - - # ok, none of the bug systems know. let's see how smart we are - if [[ ${gotit} == false ]]; then - generic_locate_patch "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch" - if [[ $? != 0 ]]; then - yetus_error "ERROR: Unsure how to process ${PATCH_OR_ISSUE}." - cleanup_and_exit 1 - fi - fi - fi - - if [[ ! -f "${PATCH_DIR}/patch" - && -f "${patchfile}" ]]; then - cp "${patchfile}" "${PATCH_DIR}/patch" - if [[ $? == 0 ]] ; then - echo "Patch file ${patchfile} copied to ${PATCH_DIR}" - else - yetus_error "ERROR: Could not copy ${patchfile} to ${PATCH_DIR}" - cleanup_and_exit 1 - fi - fi -} - -## @description Let plugins also get a copy of the arguments -## @audience private -## @stability evolving -## @replaceable no -function parse_args_plugins -{ - for plugin in ${PLUGINS} ${BUGSYSTEMS} ${TESTFORMATS} ${BUILDTOOLS}; do - if declare -f ${plugin}_parse_args >/dev/null 2>&1; then - yetus_debug "Running ${plugin}_parse_args" - #shellcheck disable=SC2086 - ${plugin}_parse_args "$@" - (( RESULT = RESULT + $? )) - fi - done - - BUGCOMMENTS=${BUGCOMMENTS:-${BUGSYSTEMS}} - if [[ ! ${BUGCOMMENTS} =~ console ]]; then - BUGCOMMENTS="${BUGCOMMENTS} console" - fi - - BUGLINECOMMENTS=${BUGLINECOMMENTS:-${BUGCOMMENTS}} -} - -## @description Let plugins also get a copy of the arguments -## @audience private -## @stability evolving -## @replaceable no -function plugins_initialize -{ - declare plugin - - for plugin in ${PLUGINS} ${BUGSYSTEMS} ${TESTFORMATS} ${BUILDTOOLS}; do - if declare -f ${plugin}_initialize >/dev/null 2>&1; then - yetus_debug "Running ${plugin}_initialize" - #shellcheck disable=SC2086 - ${plugin}_initialize - (( RESULT = RESULT + $? )) - fi - done -} - -## @description Register test-patch.d plugins -## @audience public -## @stability stable -## @replaceable no -function add_plugin -{ - PLUGINS="${PLUGINS} $1" -} - -## @description Register test-patch.d bugsystems -## @audience public -## @stability stable -## @replaceable no -function add_bugsystem -{ - BUGSYSTEMS="${BUGSYSTEMS} $1" -} - -## @description Register test-patch.d test output formats -## @audience public -## @stability stable -## @replaceable no -function add_test_format -{ - TESTFORMATS="${TESTFORMATS} $1" -} - -## @description Register test-patch.d build tools -## @audience public -## @stability stable -## @replaceable no -function add_build_tool -{ - BUILDTOOLS="${BUILDTOOLS} $1" -} - -## @description Import content from test-patch.d and optionally -## @description from user provided plugin directory -## @audience private -## @stability evolving -## @replaceable no -function importplugins -{ - declare i - declare files=() - - if [[ ${LOAD_SYSTEM_PLUGINS} == "true" ]]; then - if [[ -d "${BINDIR}/test-patch.d" ]]; then - files=(${BINDIR}/test-patch.d/*.sh) - fi - fi - - if [[ -n "${USER_PLUGIN_DIR}" && -d "${USER_PLUGIN_DIR}" ]]; then - yetus_debug "Loading user provided plugins from ${USER_PLUGIN_DIR}" - files=("${files[@]}" ${USER_PLUGIN_DIR}/*.sh) - fi - - for i in "${files[@]}"; do - if [[ -f ${i} ]]; then - yetus_debug "Importing ${i}" - . "${i}" - fi - done - - if [[ -z ${PERSONALITY} - && -f "${BINDIR}/personality/${PROJECT_NAME}.sh" ]]; then - PERSONALITY="${BINDIR}/personality/${PROJECT_NAME}.sh" - fi - - if [[ -n ${PERSONALITY} ]]; then - if [[ ! -f ${PERSONALITY} ]]; then - if [[ -f "${BINDIR}/personality/${PROJECT_NAME}.sh" ]]; then - PERSONALITY="${BINDIR}/personality/${PROJECT_NAME}.sh" - else - yetus_debug "Can't find ${PERSONALITY} to import." - return - fi - fi - yetus_debug "Importing ${PERSONALITY}" - . "${PERSONALITY}" - fi -} - -## @description if patch-level zero, then verify we aren't -## @description just adding files -## @audience public -## @stability stable -## @param filename -## @param command -## @param [..] -## @replaceable no -## @returns $? -function patchfile_verify_zero -{ - declare logfile=$1 - shift - declare dir - declare changed_files1 - declare changed_files2 - declare filename - - # don't return /dev/null - # shellcheck disable=SC2016 - changed_files1=$(${AWK} 'function p(s){if(s!~"^/dev/null"){print s}} - /^diff --git / { p($3); p($4) } - /^(\+\+\+|---) / { p($2) }' "${PATCH_DIR}/patch" | sort -u) - - # maybe we interpreted the patch wrong? check the log file - # shellcheck disable=SC2016 - changed_files2=$(${GREP} -E '^[cC]heck' "${logfile}" \ - | ${AWK} '{print $3}' \ - | ${SED} -e 's,\.\.\.$,,g') - - for filename in ${changed_files1} ${changed_files2}; do - - # leading prefix = bad - if [[ ${filename} =~ ^(a|b)/ ]]; then - return 1 - fi - - # touching an existing file is proof enough - # that pl=0 is good - if [[ -f ${filename} ]]; then - return 0 - fi - - dir=$(dirname "${filename}" 2>/dev/null) - if [[ -n ${dir} && -d ${dir} ]]; then - return 0 - fi - done - - # ¯\_(ã)_/¯ - no way for us to know, all new files with no prefix! - yetus_error "WARNING: Patch only adds files; using patch level ${PATCH_LEVEL}" - return 0 -} - -## @description git apply dryrun -## @replaceable no -## @audience private -## @stability evolving -function gitapply_dryrun -{ - declare patchfile=$1 - declare prefixsize=${2:-0} - - while [[ ${prefixsize} -lt 4 - && -z ${PATCH_METHOD} ]]; do - yetus_run_and_redirect "${PATCH_DIR}/patch-dryrun.log" \ - "${GIT}" apply --binary -v --check "-p${prefixsize}" "${patchfile}" - if [[ $? == 0 ]]; then - PATCH_LEVEL=${prefixsize} - PATCH_METHOD=gitapply - break - fi - ((prefixsize=prefixsize+1)) - done - - if [[ ${prefixsize} -eq 0 ]]; then - patchfile_verify_zero "${PATCH_DIR}/patch-dryrun.log" - if [[ $? != 0 ]]; then - PATCH_METHOD="" - PATCH_LEVEL="" - gitapply_dryrun "${patchfile}" 1 - fi - fi -} - -## @description patch patch dryrun -## @replaceable no -## @audience private -## @stability evolving -function patchcmd_dryrun -{ - declare patchfile=$1 - declare prefixsize=${2:-0} - - while [[ ${prefixsize} -lt 4 - && -z ${PATCH_METHOD} ]]; do - yetus_run_and_redirect "${PATCH_DIR}/patch-dryrun.log" \ - "${PATCH}" "-p${prefixsize}" -E --dry-run < "${patchfile}" - if [[ $? == 0 ]]; then - PATCH_LEVEL=${prefixsize} - PATCH_METHOD=patchcmd - break - fi - ((prefixsize=prefixsize+1)) - done - - if [[ ${prefixsize} -eq 0 ]]; then - patchfile_verify_zero "${PATCH_DIR}/patch-dryrun.log" - if [[ $? != 0 ]]; then - PATCH_METHOD="" - PATCH_LEVEL="" - patchcmd_dryrun "${patchfile}" 1 - fi - fi -} - -## @description driver for dryrun methods -## @replaceable no -## @audience private -## @stability evolving -function patchfile_dryrun_driver -{ - declare patchfile=$1 - declare method - - for method in "${PATCH_METHODS[@]}"; do - if declare -f ${method}_dryrun >/dev/null; then - "${method}_dryrun" "${patchfile}" - fi - if [[ -n ${PATCH_METHOD} ]]; then - break - fi - done - - if [[ -n ${PATCH_METHOD} ]]; then - return 0 - fi - return 1 -} - -## @description git patch apply -## @replaceable no -## @audience private -## @stability evolving -function gitapply_apply -{ - declare patchfile=$1 - declare extraopts - - if [[ "${COMMITMODE}" = true ]]; then - extraopts="--whitespace=fix" - fi - - echo "Applying the patch:" - yetus_run_and_redirect "${PATCH_DIR}/apply-patch-git-apply.log" \ - "${GIT}" apply --binary -v --stat --apply "-p${PATCH_LEVEL}" "${patchfile}" - ${GREP} -v "^Checking" "${PATCH_DIR}/apply-patch-git-apply.log" -} - - -## @description patch patch apply -## @replaceable no -## @audience private -## @stability evolving -function patchcmd_apply -{ - declare patchfile=$1 - - echo "Applying the patch:" - yetus_run_and_redirect "${PATCH_DIR}/apply-patch-patch-apply.log" \ - "${PATCH}" "-p${PATCH_LEVEL}" -E < "${patchfile}" - cat "${PATCH_DIR}/apply-patch-patch-apply.log" -} - -## @description driver for patch apply methods -## @replaceable no -## @audience private -## @stability evolving -function patchfile_apply_driver -{ - declare patchfile=$1 - - if declare -f ${PATCH_METHOD}_apply >/dev/null; then - "${PATCH_METHOD}_apply" "${patchfile}" - if [[ $? -gt 0 ]]; then - return 1 - fi - else - yetus_error "ERROR: Patching method ${PATCH_METHOD} does not have a way to apply patches!" - return 1 - fi - return 0 -} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/core.d/patchfiles.sh ---------------------------------------------------------------------- diff --git a/dev-support/core.d/patchfiles.sh b/dev-support/core.d/patchfiles.sh new file mode 100755 index 0000000..567e80e --- /dev/null +++ b/dev-support/core.d/patchfiles.sh @@ -0,0 +1,326 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +## @description Use curl to download the patch as a last resort +## @audience private +## @stability evolving +## @param patchloc +## @param output +## @return 0 got something +## @return 1 error +function generic_locate_patch +{ + declare input=$1 + declare output=$2 + + if [[ "${OFFLINE}" == true ]]; then + yetus_debug "generic_locate_patch: offline, skipping" + return 1 + fi + + ${CURL} --silent \ + --output "${output}" \ + "${input}" + if [[ $? != 0 ]]; then + yetus_debug "generic_locate_patch: failed to download the patch." + return 1 + fi + return 0 +} + +## @description Given a possible patch file, guess if it's a patch file +## @description only using the more intense verify if we really need to +## @audience private +## @stability evolving +## @param path to patch file to test +## @return 0 we think it's a patch file +## @return 1 we think it's not a patch file +function guess_patch_file +{ + declare patch=$1 + declare fileOutput + + if [[ ! -f ${patch} ]]; then + return 1 + fi + + yetus_debug "Trying to guess if ${patch} is a patch file." + fileOutput=$("${FILE}" "${patch}") + if [[ $fileOutput =~ \ diff\ ]]; then + yetus_debug "file magic says it's a diff." + return 0 + fi + + fileOutput=$(head -n 1 "${patch}" | "${GREP}" -E "^(From [a-z0-9]* Mon Sep 17 00:00:00 2001)|(diff .*)|(Index: .*)$") + if [[ $? == 0 ]]; then + yetus_debug "first line looks like a patch file." + return 0 + fi + + patchfile_dryrun_driver "${patch}" +} + +## @description Given ${PATCH_OR_ISSUE}, determine what type of patch file is in use, +## @description and do the necessary work to place it into ${PATCH_DIR}/patch. +## @audience private +## @stability evolving +## @replaceable no +## @return 0 on success +## @return 1 on failure, may exit +function locate_patch +{ + declare bugsys + declare patchfile="" + declare gotit=false + + yetus_debug "locate patch" + + if [[ -z "${PATCH_OR_ISSUE}" ]]; then + yetus_error "ERROR: No patch provided." + cleanup_and_exit 1 + fi + + echo "Processing: ${PATCH_OR_ISSUE}" + # it's a declarely provided file + if [[ -f ${PATCH_OR_ISSUE} ]]; then + patchfile="${PATCH_OR_ISSUE}" + else + # run through the bug systems. maybe they know? + for bugsys in ${BUGSYSTEMS}; do + if declare -f ${bugsys}_locate_patch >/dev/null 2>&1; then + "${bugsys}_locate_patch" "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch" + if [[ $? == 0 ]]; then + gotit=true + fi + fi + done + + # ok, none of the bug systems know. let's see how smart we are + if [[ ${gotit} == false ]]; then + generic_locate_patch "${PATCH_OR_ISSUE}" "${PATCH_DIR}/patch" + if [[ $? != 0 ]]; then + yetus_error "ERROR: Unsure how to process ${PATCH_OR_ISSUE}." + cleanup_and_exit 1 + fi + fi + fi + + if [[ ! -f "${PATCH_DIR}/patch" + && -f "${patchfile}" ]]; then + cp "${patchfile}" "${PATCH_DIR}/patch" + if [[ $? == 0 ]] ; then + echo "Patch file ${patchfile} copied to ${PATCH_DIR}" + else + yetus_error "ERROR: Could not copy ${patchfile} to ${PATCH_DIR}" + cleanup_and_exit 1 + fi + fi +} + +## @description if patch-level zero, then verify we aren't +## @description just adding files +## @audience public +## @stability stable +## @param filename +## @param command +## @param [..] +## @replaceable no +## @returns $? +function patchfile_verify_zero +{ + declare logfile=$1 + shift + declare dir + declare changed_files1 + declare changed_files2 + declare filename + + # don't return /dev/null + # shellcheck disable=SC2016 + changed_files1=$(${AWK} 'function p(s){if(s!~"^/dev/null"){print s}} + /^diff --git / { p($3); p($4) } + /^(\+\+\+|---) / { p($2) }' "${PATCH_DIR}/patch" | sort -u) + + # maybe we interpreted the patch wrong? check the log file + # shellcheck disable=SC2016 + changed_files2=$(${GREP} -E '^[cC]heck' "${logfile}" \ + | ${AWK} '{print $3}' \ + | ${SED} -e 's,\.\.\.$,,g') + + for filename in ${changed_files1} ${changed_files2}; do + + # leading prefix = bad + if [[ ${filename} =~ ^(a|b)/ ]]; then + return 1 + fi + + # touching an existing file is proof enough + # that pl=0 is good + if [[ -f ${filename} ]]; then + return 0 + fi + + dir=$(dirname "${filename}" 2>/dev/null) + if [[ -n ${dir} && -d ${dir} ]]; then + return 0 + fi + done + + # ¯\_(ã)_/¯ - no way for us to know, all new files with no prefix! + yetus_error "WARNING: Patch only adds files; using patch level ${PATCH_LEVEL}" + return 0 +} + +## @description git apply dryrun +## @replaceable no +## @audience private +## @stability evolving +function gitapply_dryrun +{ + declare patchfile=$1 + declare prefixsize=${2:-0} + + while [[ ${prefixsize} -lt 4 + && -z ${PATCH_METHOD} ]]; do + yetus_run_and_redirect "${PATCH_DIR}/patch-dryrun.log" \ + "${GIT}" apply --binary -v --check "-p${prefixsize}" "${patchfile}" + if [[ $? == 0 ]]; then + PATCH_LEVEL=${prefixsize} + PATCH_METHOD=gitapply + break + fi + ((prefixsize=prefixsize+1)) + done + + if [[ ${prefixsize} -eq 0 ]]; then + patchfile_verify_zero "${PATCH_DIR}/patch-dryrun.log" + if [[ $? != 0 ]]; then + PATCH_METHOD="" + PATCH_LEVEL="" + gitapply_dryrun "${patchfile}" 1 + fi + fi +} + +## @description patch patch dryrun +## @replaceable no +## @audience private +## @stability evolving +function patchcmd_dryrun +{ + declare patchfile=$1 + declare prefixsize=${2:-0} + + while [[ ${prefixsize} -lt 4 + && -z ${PATCH_METHOD} ]]; do + # shellcheck disable=SC2153 + yetus_run_and_redirect "${PATCH_DIR}/patch-dryrun.log" \ + "${PATCH}" "-p${prefixsize}" -E --dry-run < "${patchfile}" + if [[ $? == 0 ]]; then + PATCH_LEVEL=${prefixsize} + PATCH_METHOD=patchcmd + break + fi + ((prefixsize=prefixsize+1)) + done + + if [[ ${prefixsize} -eq 0 ]]; then + patchfile_verify_zero "${PATCH_DIR}/patch-dryrun.log" + if [[ $? != 0 ]]; then + PATCH_METHOD="" + PATCH_LEVEL="" + patchcmd_dryrun "${patchfile}" 1 + fi + fi +} + +## @description driver for dryrun methods +## @replaceable no +## @audience private +## @stability evolving +function patchfile_dryrun_driver +{ + declare patchfile=$1 + declare method + + #shellcheck disable=SC2153 + for method in "${PATCH_METHODS[@]}"; do + if declare -f ${method}_dryrun >/dev/null; then + "${method}_dryrun" "${patchfile}" + fi + if [[ -n ${PATCH_METHOD} ]]; then + break + fi + done + + if [[ -n ${PATCH_METHOD} ]]; then + return 0 + fi + return 1 +} + +## @description git patch apply +## @replaceable no +## @audience private +## @stability evolving +function gitapply_apply +{ + declare patchfile=$1 + declare extraopts + + if [[ "${COMMITMODE}" = true ]]; then + extraopts="--whitespace=fix" + fi + + echo "Applying the patch:" + yetus_run_and_redirect "${PATCH_DIR}/apply-patch-git-apply.log" \ + "${GIT}" apply --binary ${extraopts} -v --stat --apply "-p${PATCH_LEVEL}" "${patchfile}" + ${GREP} -v "^Checking" "${PATCH_DIR}/apply-patch-git-apply.log" +} + +## @description patch patch apply +## @replaceable no +## @audience private +## @stability evolving +function patchcmd_apply +{ + declare patchfile=$1 + + echo "Applying the patch:" + yetus_run_and_redirect "${PATCH_DIR}/apply-patch-patch-apply.log" \ + "${PATCH}" "-p${PATCH_LEVEL}" -E < "${patchfile}" + cat "${PATCH_DIR}/apply-patch-patch-apply.log" +} + +## @description driver for patch apply methods +## @replaceable no +## @audience private +## @stability evolving +function patchfile_apply_driver +{ + declare patchfile=$1 + + if declare -f ${PATCH_METHOD}_apply >/dev/null; then + "${PATCH_METHOD}_apply" "${patchfile}" + if [[ $? -gt 0 ]]; then + return 1 + fi + else + yetus_error "ERROR: Patching method ${PATCH_METHOD} does not have a way to apply patches!" + return 1 + fi + return 0 +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/bigtop.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/bigtop.sh b/dev-support/personality/bigtop.sh index 33bb0f0..3854f6b 100755 --- a/dev-support/personality/bigtop.sh +++ b/dev-support/personality/bigtop.sh @@ -14,20 +14,25 @@ # See the License for the specific language governing permissions and # limitations under the License. -# shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=master -# shellcheck disable=SC2034 -JIRA_ISSUE_RE='^BIGTOP-[0-9]+$' -# shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="" -# shellcheck disable=SC2034 -BUILDTOOL=gradle -# shellcheck disable=SC2034 -GITHUB_REPO="apache/bigtop" -# shellcheck disable=SC2034 -BIGTOP_PUPPETSETUP=false +personality_plugins "all,-checkstyle,-findbugs" -add_plugin bigtop +function personality_globals +{ + # shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=master + # shellcheck disable=SC2034 + JIRA_ISSUE_RE='^BIGTOP-[0-9]+$' + # shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="" + # shellcheck disable=SC2034 + BUILDTOOL=gradle + # shellcheck disable=SC2034 + GITHUB_REPO="apache/bigtop" + # shellcheck disable=SC2034 + BIGTOP_PUPPETSETUP=false +} + +add_test_type bigtop function bigtop_usage { http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/flink.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/flink.sh b/dev-support/personality/flink.sh index 9ee11f3..356c76b 100755 --- a/dev-support/personality/flink.sh +++ b/dev-support/personality/flink.sh @@ -14,16 +14,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=master -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^FLINK-[0-9]+$' -#shellcheck disable=SC2034 -GITHUB_REPO="apache/flink" -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="" - -add_plugin flinklib +personality_plugins "all" + +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=master + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^FLINK-[0-9]+$' + #shellcheck disable=SC2034 + GITHUB_REPO="apache/flink" + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="" +} + +add_test_type flinklib function flinklib_filefilter { http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/hadoop.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/hadoop.sh b/dev-support/personality/hadoop.sh index 51cfd91..1b0abde 100755 --- a/dev-support/personality/hadoop.sh +++ b/dev-support/personality/hadoop.sh @@ -16,18 +16,23 @@ # Override these to match Apache Hadoop's requirements -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=trunk -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="https://wiki.apache.org/hadoop/HowToContribute"; -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$' -#shellcheck disable=SC2034 -GITHUB_REPO="apache/hadoop" -#shellcheck disable=SC2034 -PYLINT_OPTIONS="--indent-string=' '" - -HADOOP_MODULES="" +personality_plugins "all,-ant,-gradle,-scalac,-scaladoc" + +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=trunk + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="https://wiki.apache.org/hadoop/HowToContribute"; + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^(HADOOP|YARN|MAPREDUCE|HDFS)-[0-9]+$' + #shellcheck disable=SC2034 + GITHUB_REPO="apache/hadoop" + #shellcheck disable=SC2034 + PYLINT_OPTIONS="--indent-string=' '" + + HADOOP_MODULES="" +} function hadoop_module_manipulation { http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/hbase.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/hbase.sh b/dev-support/personality/hbase.sh index 120efa7..8451f17 100755 --- a/dev-support/personality/hbase.sh +++ b/dev-support/personality/hbase.sh @@ -14,20 +14,25 @@ # See the License for the specific language governing permissions and # limitations under the License. -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=master -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^HBASE-[0-9]+$' -#shellcheck disable=SC2034 -GITHUB_REPO="apache/hbase" -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="" +personality_plugins "all" -# All supported Hadoop versions that we want to test the compilation with -HBASE_HADOOP_VERSIONS="2.4.1 2.5.2 2.6.0" - -# Override the maven options -MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx3100M"}" +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=master + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^HBASE-[0-9]+$' + #shellcheck disable=SC2034 + GITHUB_REPO="apache/hbase" + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="" + + # All supported Hadoop versions that we want to test the compilation with + HBASE_HADOOP_VERSIONS="2.4.1 2.5.2 2.6.0" + + # Override the maven options + MAVEN_OPTS="${MAVEN_OPTS:-"-Xmx3100M"}" +} function personality_modules { @@ -68,7 +73,7 @@ function personality_modules ################################################### -add_plugin hadoopcheck +add_test_type hadoopcheck function hadoopcheck_filefilter { @@ -117,7 +122,7 @@ function hadoopcheck_rebuild ###################################### -add_plugin hbaseprotoc +add_test_type hbaseprotoc function hbaseprotoc_filefilter { @@ -184,7 +189,7 @@ function hbaseprotoc_rebuild ###################################### -add_plugin hbaseanti +add_test_type hbaseanti function hbaseanti_filefilter { http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/kafka.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/kafka.sh b/dev-support/personality/kafka.sh index 83e5313..cc4bf09 100755 --- a/dev-support/personality/kafka.sh +++ b/dev-support/personality/kafka.sh @@ -14,16 +14,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=trunk -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^KAFKA-[0-9]+$' -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="http://kafka.apache.org/contributing.html"; -# shellcheck disable=SC2034 -BUILDTOOL=gradle -#shellcheck disable=SC2034 -GITHUB_REPO="apache/kafka" +personality_plugins "all,-checkstyle,-asflicense" + +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=trunk + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^KAFKA-[0-9]+$' + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="http://kafka.apache.org/contributing.html"; + # shellcheck disable=SC2034 + BUILDTOOL=gradle + #shellcheck disable=SC2034 + GITHUB_REPO="apache/kafka" +} function personality_modules { http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/pig.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/pig.sh b/dev-support/personality/pig.sh index 91c29f8..50ea52f 100755 --- a/dev-support/personality/pig.sh +++ b/dev-support/personality/pig.sh @@ -14,16 +14,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=trunk -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^PIG-[0-9]+$' -#shellcheck disable=SC2034 -GITHUB_REPO="apache/pig" -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="" -#shellcheck disable=SC2034 -BUILDTOOL=ant +personality_plugins "all" + +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=trunk + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^PIG-[0-9]+$' + #shellcheck disable=SC2034 + GITHUB_REPO="apache/pig" + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="" + #shellcheck disable=SC2034 + BUILDTOOL=ant +} function personality_modules { http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/samza.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/samza.sh b/dev-support/personality/samza.sh index 9d39d92..131e73e 100755 --- a/dev-support/personality/samza.sh +++ b/dev-support/personality/samza.sh @@ -14,13 +14,19 @@ # See the License for the specific language governing permissions and # limitations under the License. -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=master -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^SAMZA-[0-9]+$' -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/SAMZA/Contributor's+Corner" -# shellcheck disable=SC2034 -BUILDTOOL=gradle -#shellcheck disable=SC2034 -GITHUB_REPO="apache/samza" + +personality_plugins "all,-checkstyle" + +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=master + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^SAMZA-[0-9]+$' + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/SAMZA/Contributor's+Corner" + # shellcheck disable=SC2034 + BUILDTOOL=gradle + #shellcheck disable=SC2034 + GITHUB_REPO="apache/samza" +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/tajo.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/tajo.sh b/dev-support/personality/tajo.sh index 209d75e..aa5fc48 100755 --- a/dev-support/personality/tajo.sh +++ b/dev-support/personality/tajo.sh @@ -14,11 +14,16 @@ # See the License for the specific language governing permissions and # limitations under the License. -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=master -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^TAJO-[0-9]+$' -#shellcheck disable=SC2034 -GITHUB_REPO="apache/tajo" -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TAJO/How+to+Contribute+to+Tajo"; \ No newline at end of file +personality_plugins "all" + +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=master + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^TAJO-[0-9]+$' + #shellcheck disable=SC2034 + GITHUB_REPO="apache/tajo" + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TAJO/How+to+Contribute+to+Tajo"; +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/personality/tez.sh ---------------------------------------------------------------------- diff --git a/dev-support/personality/tez.sh b/dev-support/personality/tez.sh index 39f0b53..584865e 100755 --- a/dev-support/personality/tez.sh +++ b/dev-support/personality/tez.sh @@ -14,11 +14,16 @@ # See the License for the specific language governing permissions and # limitations under the License. -#shellcheck disable=SC2034 -PATCH_BRANCH_DEFAULT=master -#shellcheck disable=SC2034 -JIRA_ISSUE_RE='^TEZ-[0-9]+$' -#shellcheck disable=SC2034 -GITHUB_REPO="apache/tez" -#shellcheck disable=SC2034 -HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez"; +personality_plugins "all" + +function personality_globals +{ + #shellcheck disable=SC2034 + PATCH_BRANCH_DEFAULT=master + #shellcheck disable=SC2034 + JIRA_ISSUE_RE='^TEZ-[0-9]+$' + #shellcheck disable=SC2034 + GITHUB_REPO="apache/tez" + #shellcheck disable=SC2034 + HOW_TO_CONTRIBUTE="https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez"; +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/smart-apply-patch.sh ---------------------------------------------------------------------- diff --git a/dev-support/smart-apply-patch.sh b/dev-support/smart-apply-patch.sh index bf5e5bf..8a10b33 100755 --- a/dev-support/smart-apply-patch.sh +++ b/dev-support/smart-apply-patch.sh @@ -28,8 +28,6 @@ BINDIR=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P) #shellcheck disable=SC2034 QATESTMODE=false -. "${BINDIR}/core.d/common.sh" - # dummy functions function add_vote_table { @@ -84,16 +82,20 @@ function setup_defaults ## @replaceable no function yetus_usage { - echo "Usage: apply-patch.sh [options] patch-file | issue-number | http" + echo "Usage: smart-apply-patch.sh [options] patch" echo echo "--committer Apply patches like a boss." echo "--debug If set, then output some extra stuff to stderr" echo "--dry-run Check for patch viability without applying" + echo "--list-plugins List all installed plug-ins and then exit" echo "--modulelist=<list> Specify additional modules to test (comma delimited)" echo "--offline Avoid connecting to the Internet" echo "--patch-dir=<dir> The directory for working and output files (default '/tmp/yetus-(random))" - echo "--plugins=<dir> A directory of user provided plugins. see test-patch.d for examples (default empty)" + echo "--personality=<file> The personality file to load" + echo "--plugins=<list> Specify which plug-ins to add/delete (comma delimited; use 'all' for all found)" + echo "--project=<name> The short name for project currently using test-patch (default 'yetus')" echo "--skip-system-plugins Do not load plugins from ${BINDIR}/test-patch.d" + echo "--user-plugins=<dir> A directory of user provided plugins. see test-patch.d for examples (default empty)" echo "" echo "Shell binary overrides:" echo "--awk-cmd=<cmd> The 'awk' command to use (default 'awk')" @@ -170,7 +172,7 @@ function gitam_dryrun # use apply instead. gitapply_dryrun "$@" - if [[ -n ${PATCH_METHOD}="gitapply" ]]; then + if [[ ${PATCH_METHOD} = "gitapply" ]]; then PATCH_METHOD="gitam" fi } @@ -189,8 +191,24 @@ function gitam_apply ${GREP} -v "^Checking" "${PATCH_DIR}/apply-patch-git-am.log" } +## @description import core library routines +## @audience private +## @stability evolving +function import_core +{ + declare filename + + for filename in "${BINDIR}/core.d"/*; do + # shellcheck disable=SC1091 + # shellcheck source=core.d/01-common.sh + . "${filename}" + done +} + trap "cleanup_and_exit 1" HUP INT QUIT TERM +import_core + setup_defaults parse_args "$@" http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch-docker/test-patch-docker.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch-docker/test-patch-docker.sh b/dev-support/test-patch-docker/test-patch-docker.sh index 0e5d24f..8b19502 100755 --- a/dev-support/test-patch-docker/test-patch-docker.sh +++ b/dev-support/test-patch-docker/test-patch-docker.sh @@ -23,7 +23,7 @@ DID=${RANDOM} ## @param string function yetus_debug { - if [[ -n "${YETUS_SHELL_SCRIPT_DEBUG}" ]]; then + if [[ "${YETUS_SHELL_SCRIPT_DEBUG}" = true ]]; then echo "[$(date) DEBUG]: $*" 1>&2 fi } http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/asflicense.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/asflicense.sh b/dev-support/test-patch.d/asflicense.sh index 255c257..3b55f91 100755 --- a/dev-support/test-patch.d/asflicense.sh +++ b/dev-support/test-patch.d/asflicense.sh @@ -15,7 +15,7 @@ # limitations under the License. -add_plugin asflicense +add_test_type asflicense add_test asflicense ## @description Verify all files have an Apache License http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/author.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/author.sh b/dev-support/test-patch.d/author.sh index 49824a5..d456bbb 100755 --- a/dev-support/test-patch.d/author.sh +++ b/dev-support/test-patch.d/author.sh @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -add_plugin author +add_test_type author ## @description Check the current directory for @author tags ## @audience private http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/builtin-bugsystem.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/builtin-bugsystem.sh b/dev-support/test-patch.d/builtin-bugsystem.sh deleted file mode 100755 index 6773ba0..0000000 --- a/dev-support/test-patch.d/builtin-bugsystem.sh +++ /dev/null @@ -1,171 +0,0 @@ -#!/usr/bin/env bash -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - - -# This bug system handles the output on the screen. - -add_bugsystem console - -# we always call this one last - -function generic_locate_patch -{ - declare input=$1 - declare output=$2 - - if [[ "${OFFLINE}" == true ]]; then - yetus_debug "generic_locate_patch: offline, skipping" - return 1 - fi - - ${CURL} --silent \ - --output "${output}" \ - "${input}" - if [[ $? != 0 ]]; then - yetus_debug "generic_locate_patch: failed to download the patch." - return 1 - fi - return 0 -} - -## @description Print out the finished details on the console -## @audience private -## @stability evolving -## @replaceable no -## @param runresult -## @return 0 on success -## @return 1 on failure -function console_finalreport -{ - declare result=$1 - shift - declare i=0 - declare ourstring - declare vote - declare subs - declare ela - declare comment - declare commentfile1="${PATCH_DIR}/comment.1" - declare commentfile2="${PATCH_DIR}/comment.2" - declare normaltop - declare line - declare seccoladj=0 - declare spcfx=${PATCH_DIR}/spcl.txt - - if [[ ${result} == 0 ]]; then - if [[ ${JENKINS} == false ]]; then - if declare -f ${PROJECT_NAME}_console_success >/dev/null; then - ${PROJECT_NAME}_console_success > "${spcfx}" - else - { - printf "IF9fX18gICAgICAgICAgICAgICAgICAgICAgICAgICAgICBfIAovIF9fX3wg"; - printf "XyAgIF8gIF9fXyBfX18gX19fICBfX18gX19ffCB8ClxfX18gXHwgfCB8IHwv"; - printf "IF9fLyBfXy8gXyBcLyBfXy8gX198IHwKIF9fXykgfCB8X3wgfCAoX3wgKF98"; - printf "ICBfXy9cX18gXF9fIFxffAp8X19fXy8gXF9fLF98XF9fX1xfX19cX19ffHxf"; - printf "X18vX19fKF8pCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg"; - printf "ICAK"; - } > "${spcfx}" - fi - fi - printf "\n\n+1 overall\n\n" - else - if [[ ${JENKINS} == false ]]; then - if declare -f ${PROJECT_NAME}_console_failure >/dev/null; then - ${PROJECT_NAME}_console_failure > "${spcfx}" - else - { - printf "IF9fX19fICAgICBfIF8gICAgICAgICAgICAgICAgXyAKfCAgX19ffF8gXyhf"; - printf "KSB8XyAgIF8gXyBfXyBfX198IHwKfCB8XyAvIF9gIHwgfCB8IHwgfCB8ICdf"; - printf "Xy8gXyBcIHwKfCAgX3wgKF98IHwgfCB8IHxffCB8IHwgfCAgX18vX3wKfF98"; - printf "ICBcX18sX3xffF98XF9fLF98X3wgIFxfX18oXykKICAgICAgICAgICAgICAg"; - printf "ICAgICAgICAgICAgICAgICAK" - } > "${spcfx}" - fi - fi - printf "\n\n-1 overall\n\n" - fi - - if [[ -f ${spcfx} ]]; then - if which base64 >/dev/null 2>&1; then - base64 --decode "${spcfx}" 2>/dev/null - elif which openssl >/dev/null 2>&1; then - openssl enc -A -d -base64 -in "${spcfx}" 2>/dev/null - fi - echo - echo - rm "${spcfx}" - fi - - seccoladj=$(findlargest 2 "${TP_VOTE_TABLE[@]}") - if [[ ${seccoladj} -lt 10 ]]; then - seccoladj=10 - fi - - seccoladj=$((seccoladj + 2 )) - i=0 - until [[ $i -eq ${#TP_HEADER[@]} ]]; do - printf "%s\n" "${TP_HEADER[${i}]}" - ((i=i+1)) - done - - printf "| %s | %*s | %s | %s\n" "Vote" ${seccoladj} Subsystem Runtime "Comment" - echo "============================================================================" - i=0 - until [[ $i -eq ${#TP_VOTE_TABLE[@]} ]]; do - ourstring=$(echo "${TP_VOTE_TABLE[${i}]}" | tr -s ' ') - vote=$(echo "${ourstring}" | cut -f2 -d\|) - subs=$(echo "${ourstring}" | cut -f3 -d\|) - ela=$(echo "${ourstring}" | cut -f4 -d\|) - comment=$(echo "${ourstring}" | cut -f5 -d\|) - - echo "${comment}" | fold -s -w $((78-seccoladj-22)) > "${commentfile1}" - normaltop=$(head -1 "${commentfile1}") - ${SED} -e '1d' "${commentfile1}" > "${commentfile2}" - - printf "| %4s | %*s | %-10s |%-s\n" "${vote}" ${seccoladj} \ - "${subs}" "${ela}" "${normaltop}" - while read line; do - printf "| | %*s | | %-s\n" ${seccoladj} " " "${line}" - done < "${commentfile2}" - - ((i=i+1)) - rm "${commentfile2}" "${commentfile1}" 2>/dev/null - done - - if [[ ${#TP_TEST_TABLE[@]} -gt 0 ]]; then - seccoladj=$(findlargest 1 "${TP_TEST_TABLE[@]}") - printf "\n\n%*s | Tests\n" "${seccoladj}" "Reason" - i=0 - until [[ $i -eq ${#TP_TEST_TABLE[@]} ]]; do - ourstring=$(echo "${TP_TEST_TABLE[${i}]}" | tr -s ' ') - vote=$(echo "${ourstring}" | cut -f2 -d\|) - subs=$(echo "${ourstring}" | cut -f3 -d\|) - printf "%*s | %s\n" "${seccoladj}" "${vote}" "${subs}" - ((i=i+1)) - done - fi - - printf "\n\n|| Subsystem || Report/Notes ||\n" - echo "============================================================================" - i=0 - - until [[ $i -eq ${#TP_FOOTER_TABLE[@]} ]]; do - comment=$(echo "${TP_FOOTER_TABLE[${i}]}" | - ${SED} -e "s,@@BASE@@,${PATCH_DIR},g") - printf "%s\n" "${comment}" - ((i=i+1)) - done -} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/builtin-personality.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/builtin-personality.sh b/dev-support/test-patch.d/builtin-personality.sh deleted file mode 100755 index 1f96e03..0000000 --- a/dev-support/test-patch.d/builtin-personality.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/usr/bin/env bash -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -function personality_modules -{ - "${BUILDTOOL}_builtin_personality_modules" "$@" -} - -function personality_file_tests -{ - "${BUILDTOOL}_builtin_personality_file_tests" "$@" -} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/checkstyle.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/checkstyle.sh b/dev-support/test-patch.d/checkstyle.sh index bc47b57..b73550f 100755 --- a/dev-support/test-patch.d/checkstyle.sh +++ b/dev-support/test-patch.d/checkstyle.sh @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -add_plugin checkstyle +add_test_type checkstyle CHECKSTYLE_TIMER=0 http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/findbugs.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/findbugs.sh b/dev-support/test-patch.d/findbugs.sh index cc251bb..ae18893 100755 --- a/dev-support/test-patch.d/findbugs.sh +++ b/dev-support/test-patch.d/findbugs.sh @@ -18,7 +18,7 @@ FINDBUGS_HOME=${FINDBUGS_HOME:-} FINDBUGS_WARNINGS_FAIL_PRECHECK=false -add_plugin findbugs +add_test_type findbugs function findbugs_filefilter { @@ -64,12 +64,20 @@ function findbugs_parse_args ## @return 1 findbugs is missing some component function findbugs_is_installed { - if [[ ! -x "${FINDBUGS_HOME}/bin/findbugs" ]]; then - printf "\n\n%s is not executable.\n\n" "${FINDBUGS_HOME}/bin/findbugs" - add_vote_table -1 findbugs "Findbugs is not installed." - return 1 - fi - return 0 + declare exec + declare status=0 + + for exec in findbugs \ + computeBugHistory \ + convertXmlToText \ + filterBugs \ + setBugDatabaseInfo; do + if [[ ! -x "${FINDBUGS_HOME}/bin/${exec}" ]]; then + yetus_error "ERROR: ${FINDBUGS_HOME}/bin/${exec} is not executable." + status=1 + fi + done + return ${status} } ## @description Run the maven findbugs plugin and record found issues in a bug database @@ -191,7 +199,8 @@ function findbugs_preapply findbugs_is_installed if [[ $? != 0 ]]; then - return 1 + add_vote_table 0 findbugs "findbugs executables are not available." + return 0 fi big_console_header "Pre-patch findbugs detection" @@ -269,13 +278,13 @@ function findbugs_postinstall return 0 fi - big_console_header "Patch findbugs detection" - findbugs_is_installed if [[ $? != 0 ]]; then - return 1 + return 0 fi + big_console_header "Patch findbugs detection" + findbugs_runner patch if [[ ${UNSUPPORTED_TEST} = true ]]; then @@ -399,4 +408,4 @@ function findbugs_rebuild else findbugs_postinstall fi -} \ No newline at end of file +} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/java.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/java.sh b/dev-support/test-patch.d/java.sh index b07c67b..01dfab4 100755 --- a/dev-support/test-patch.d/java.sh +++ b/dev-support/test-patch.d/java.sh @@ -14,8 +14,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -add_plugin javac -add_plugin javadoc +add_test_type javac +add_test_type javadoc function javac_initialize { http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/maven.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/maven.sh b/dev-support/test-patch.d/maven.sh index d4f056e..019b64d 100755 --- a/dev-support/test-patch.d/maven.sh +++ b/dev-support/test-patch.d/maven.sh @@ -25,8 +25,8 @@ fi MAVEN_CUSTOM_REPOS=false MAVEN_CUSTOM_REPOS_DIR="${HOME}/yetus-m2" -add_plugin mvnsite -add_plugin mvneclipse +add_test_type mvnsite +add_test_type mvneclipse add_build_tool maven function maven_usage @@ -437,4 +437,4 @@ function maven_docker_support echo "-v ${MAVEN_CUSTOM_REPOS_DIR}:${MAVEN_CUSTOM_REPOS_DIR}" \ >> "${PATCH_DIR}/buildtool-docker-params.txt" fi -} \ No newline at end of file +} http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/perlcritic.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/perlcritic.sh b/dev-support/test-patch.d/perlcritic.sh index c3ef6c9..c55f01a 100755 --- a/dev-support/test-patch.d/perlcritic.sh +++ b/dev-support/test-patch.d/perlcritic.sh @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -add_plugin perlcritic +add_test_type perlcritic PERLCRITIC_TIMER=0 http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/pylint.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/pylint.sh b/dev-support/test-patch.d/pylint.sh index 4519031..d5af000 100755 --- a/dev-support/test-patch.d/pylint.sh +++ b/dev-support/test-patch.d/pylint.sh @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -add_plugin pylint +add_test_type pylint PYLINT_TIMER=0 http://git-wip-us.apache.org/repos/asf/yetus/blob/dbeacad3/dev-support/test-patch.d/rubocop.sh ---------------------------------------------------------------------- diff --git a/dev-support/test-patch.d/rubocop.sh b/dev-support/test-patch.d/rubocop.sh index 0190289..c0fa2ac 100755 --- a/dev-support/test-patch.d/rubocop.sh +++ b/dev-support/test-patch.d/rubocop.sh @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -add_plugin rubocop +add_test_type rubocop RUBOCOP_TIMER=0
