Repository: metron Updated Branches: refs/heads/master 35d81cb9d -> 24822dddc
METRON-1413 Add Metron Commit Tool (nickwallen) closes apache/metron#902 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/24822ddd Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/24822ddd Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/24822ddd Branch: refs/heads/master Commit: 24822dddc68c264f59723f5e17d423cd497f6807 Parents: 35d81cb Author: nickwallen <[email protected]> Authored: Thu Jan 25 18:48:02 2018 -0500 Committer: nickallen <[email protected]> Committed: Thu Jan 25 18:48:02 2018 -0500 ---------------------------------------------------------------------- .github/PULL_REQUEST_TEMPLATE.md | 7 +- .travis.yml | 2 +- build_utils/README.md | 57 ---- build_utils/create_bundled_licenses.sh | 23 -- build_utils/generate_license.py | 108 -------- build_utils/list_dependencies.sh | 19 -- build_utils/release-utils/metron-rc-check | 269 ------------------- .../release-utils/validate-jira-for-release | 197 -------------- build_utils/verify_license.py | 44 --- build_utils/verify_licenses.sh | 19 -- dev-utilities/build-utils/README.md | 64 +++++ .../build-utils/create_bundled_licenses.sh | 24 ++ dev-utilities/build-utils/generate_license.py | 108 ++++++++ dev-utilities/build-utils/list_dependencies.sh | 19 ++ dev-utilities/build-utils/verify_license.py | 44 +++ dev-utilities/build-utils/verify_licenses.sh | 19 ++ dev-utilities/committer-utils/README.md | 114 ++++++++ dev-utilities/committer-utils/prepare-commit | 216 +++++++++++++++ dev-utilities/release-utils/metron-rc-check | 269 +++++++++++++++++++ .../release-utils/validate-jira-for-release | 197 ++++++++++++++ site-book/bin/generate-md.sh | 5 +- 21 files changed, 1080 insertions(+), 744 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/.github/PULL_REQUEST_TEMPLATE.md ---------------------------------------------------------------------- diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index ee0ed83..af6c1e7 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -12,7 +12,7 @@ Please refer also to our [Build Verification Guidelines](https://cwiki.apache.or In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: -- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). +- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? @@ -22,11 +22,11 @@ In order to streamline the review of the contribution we ask you follow these gu - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` - mvn -q clean integration-test install && build_utils/verify_licenses.sh + mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? -- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? +- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: @@ -40,4 +40,3 @@ In order to streamline the review of the contribution we ask you follow these gu #### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. - http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/.travis.yml ---------------------------------------------------------------------- diff --git a/.travis.yml b/.travis.yml index f5edfb2..71d7165 100644 --- a/.travis.yml +++ b/.travis.yml @@ -36,7 +36,7 @@ install: - time mvn -q -T 2C -DskipTests clean install script: - - time mvn -q -T 2C surefire:test@unit-tests && time mvn -q surefire:test@integration-tests && time mvn -q test --projects metron-interface/metron-config && time build_utils/verify_licenses.sh + - time mvn -q -T 2C surefire:test@unit-tests && time mvn -q surefire:test@integration-tests && time mvn -q test --projects metron-interface/metron-config && time dev-utilities/build-utils/verify_licenses.sh before_cache: - rm -rf $HOME/.m2/repository/org/apache/metron http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/README.md ---------------------------------------------------------------------- diff --git a/build_utils/README.md b/build_utils/README.md deleted file mode 100644 index 04087ee..0000000 --- a/build_utils/README.md +++ /dev/null @@ -1,57 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> -# Build Utilities - -The aim of the build utilities project is to provide some scripting -around the care and maintenance of the building infrastructure. At the -moment the primary mission is around utilities that assist us manage the -licenses of our dependencies and generate the appropriate notices or -licenses. - -## `dependencies_with_url.csv` -This file is the reference file for all of our dependencies. -If you add a dependency, you must add a line to the -`dependencies_with_url.csv` file. - -## `list_dependencies.sh` - -List all of the transitive dependencies for the project rooted at cwd. - -## `verify_licenses.sh` - -This script, as run by our travis build infrastructure, will look at the -dependencies and verify that we know about them. Travis will use this -script which takes the transitive dependency list and check against the -`dependencies_with_url.csv` file to ensure that it's listed. This will -make sure we track dependencies and do not have any unacceptable -dependencies. - -If you want to dump all of the dependencies that it doesn't know about, -from the top level directory: -`build_utils/list_dependencies.sh | python build_utils/verify_license.py ./dependencies_with_url.csv dump` - -## `create_bundled_licenses.sh` - -This script is intended to regenerate the licenses for each project that -bundles its dependencies. Because we bundle our dependencies in a -shaded jar, we -[must](http://www.apache.org/dev/licensing-howto.html#deps-of-deps) specify a `LICENSE` file with the permissively -licensed dependencies notated as per [here](http://www.apache.org/dev/licensing-howto.html#permissive-deps) - -Example command to regenerate licenses (run from top level directory): -`for i in $(find . -name LICENSE | grep src | grep META-INF | awk -Fsrc '{print $1}');do build_utils/create_bundled_licenses.sh $i;done` http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/create_bundled_licenses.sh ---------------------------------------------------------------------- diff --git a/build_utils/create_bundled_licenses.sh b/build_utils/create_bundled_licenses.sh deleted file mode 100755 index 67a8126..0000000 --- a/build_utils/create_bundled_licenses.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -TLD=$(pwd) -LICENSE_FILE="src/main/resources/META-INF/LICENSE" -pushd $1 -$TLD/build_utils/list_dependencies.sh | python $TLD/build_utils/generate_license.py $TLD/dependencies_with_url.csv $TLD/LICENSE 1> $LICENSE_FILE -popd http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/generate_license.py ---------------------------------------------------------------------- diff --git a/build_utils/generate_license.py b/build_utils/generate_license.py deleted file mode 100644 index 366e515..0000000 --- a/build_utils/generate_license.py +++ /dev/null @@ -1,108 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -import sets -import sys - -category_a_licenses = [ "BSD Software License", - "Creative Commons License", - "Common Development and Distribution License", - "Common Development and Distribution License v1.1", - "Common Development and Distribution License v1.0", - "Common Public License v1.0", - "Eclipse Public License v1.0", - "MIT Software License", - "Mozilla Public License v2.0" - ] - - -license_mapping = { - "ASLv2": "Apache Software License v2", - "Apache 2": "Apache Software License v2", - "Apache 2.0": "Apache Software License v2", - "Apache License": "Apache Software License v2", - "Apache License 2.0": "Apache Software License v2", - "Apache License V2.0": "Apache Software License v2", - "Apache License Version 2.0": "Apache Software License v2", - "Apache Software Licenses": "Apache Software License v2", - "Apache v2": "Apache Software License v2", - "The Apache Software License": "Apache Software License v2", - "BSD": "BSD Software License", - "BSD 2-clause": "BSD Software License", - "BSD 3-Clause \"New\" or \"Revised\" License (BSD-3-Clause)" : "BSD Software License", - "BSD 3-Clause License": "BSD Software License", - "BSD 3-clause": "BSD Software License", - "BSD-like": "BSD Software License", - "CC0 1.0 Universal": "Creative Commons License", - "CDDL": "Common Development and Distribution License", - "CDDL 1.1": "Common Development and Distribution License v1.1", - "COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0": "Common Development and Distribution License v1.0", - "Common Development and Distribution License (CDDL) v1.0": "Common Development and Distribution License v1.0", - "Common Public License Version 1.0": "Common Public License v1.0", - "Eclipse Public License 1.0": "Eclipse Public License v1.0", - "MIT" : "MIT Software License", - "MIT License" : "MIT Software License", - "Mozilla Public License Version 2.0" : "Mozilla Public License v2.0", - "New BSD License" : "BSD Software License", - "New BSD license" : "BSD Software License", - "Public" : "Public Domain", - "Public Domain" : "Public Domain", - "The BSD 3-Clause License" : "BSD Software License", - "The BSD License" : "BSD Software License", - "The MIT License" : "MIT Software License", - "ACCEPTABLE" : "Ignore" -} - -def read_component(i): - with open(i, 'r') as fp: - component_lines = fp.readlines() - ret = {} - for line in component_lines: - if len(line) > 0: - tokens = line.split(',') - key = tokens[0] - url = tokens[-1].strip() - license = license_mapping[tokens[1].strip()] - if license is None: - raise ValueError("unable to normalize license: " + tokens[1]) - l = line.split(',')[0].strip() - ret[key] = { 'url' : url, 'license' : license } - return ret - -def read_license(f) : - with open(f) as fp: - return fp.read() - -def get_blurb(component, license_info): - tokens = component.split(':') - artifact_id = tokens[1] - version = tokens[3] - return "This product bundles " + artifact_id + " " + version + ", which is available under a \"" + license_info['license'] + "\" license. " + "For details, see " + license_info['url'] - -if __name__ == '__main__': - components = read_component(sys.argv[1]) - license = read_license(sys.argv[2]) - for line in sys.stdin: - component = line.strip() - if len(component) == 0 or component == 'none' or component not in components: - continue - else: - license_info = components[component] - if license_info['license'] in category_a_licenses: - license = license + "\n" + get_blurb(component, license_info) - continue - print license http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/list_dependencies.sh ---------------------------------------------------------------------- diff --git a/build_utils/list_dependencies.sh b/build_utils/list_dependencies.sh deleted file mode 100755 index a56a5bd..0000000 --- a/build_utils/list_dependencies.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -{ mvn dependency:list ; mvn dependency:list -PHDP-2.5.0.0 ; } | grep "^\[INFO\] " | awk '{print $2}' | grep -v "org.apache" | grep -v "test" | grep -v "provided" | grep -v "runtime" | grep -v ":system" | sort | uniq http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/release-utils/metron-rc-check ---------------------------------------------------------------------- diff --git a/build_utils/release-utils/metron-rc-check b/build_utils/release-utils/metron-rc-check deleted file mode 100755 index 7084106..0000000 --- a/build_utils/release-utils/metron-rc-check +++ /dev/null @@ -1,269 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -shopt -s nocasematch - -function help { - echo " " - echo "usage: ${0}" - echo " -v/--version=<version> The version of the metron release. [Required]" - echo " -c/--candidate=<RC#> Defines the Release Candidate. [Required]" - echo " -b/--bro=<bro version> The version of the bro kafka plugin. [Required]" - echo " -h/--help Usage information." - echo " " - echo "example: " - echo " metron-rc-check --version=0.4.2 --candidate=RC2 --bro=0.1.0" - echo " " -} - -METRON_DIST="https://dist.apache.org/repos/dist/dev/metron/"; -# print help, if the user just runs this without any args -if [ "$#" -eq 0 ]; then - help - exit 1 -fi - -# handle command line options -for i in "$@"; do - case $i in - # - # VERSION: The release version of Metron to validate. - # - # - -v=*|--version=*) - VERSION="${i#*=}" - shift # past argument=value - ;; - - # - # RC: Defines the RC# to use - # - # -c=RC2 - # --candidate=RC2 - # - -c=*|--candidate=*) - CANDIDATE="${i#*=}" - shift # past argument=value - ;; - - # - # END: Defines the last commit to inspect - # - # -b=0.1.0 - # --bro=0.1.0 - # - -b=*|--bro=*) - BRO="${i#*=}" - shift # past argument=value - ;; - - # - # -h/--help - # - -h|--help) - help - exit 0 - shift # past argument with no value - ;; - - # - # Unknown option - # - *) - UNKNOWN_OPTION="${i#*=}" - echo "Error: unknown option: $UNKNOWN_OPTION" - help - ;; - esac -done - -# validation -if [ -z "$VERSION" ]; then - echo "Missing -v/--version is is required" - exit 1 -fi -if [[ "$VERSION" =~ ^[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,2} ]]; then - METRON_VERSION="$VERSION" -else - echo "[ERROR] "$VERSION" may not be a valid version number" - exit 1 -fi - -if [ -z "$CANDIDATE" ]; then - echo "Missing -c/--candidate which is required" - exit 1 -fi - -if [[ "$CANDIDATE" =~ ^RC[0-9]+ ]]; then - RC=$(echo "$CANDIDATE" | tr '[:upper:]' '[:lower:]') - UPPER_RC=$(echo "$CANDIDATE" | tr '[:lower:]' '[:upper:]') -elif [[ "$CANDIDATE" =~ ^[0-9]+ ]]; then - RC=rc"$CANDIDATE" - UPPER_RC=RC"$CANDIDATE" -else - echo "[ERROR] invalid RC, valid is RC# or just #" - exit 1 -fi - -if [ -z "$BRO" ]; then - echo "Missing -b/--bro which is required" - exit 1 -fi - -if [[ "$BRO" =~ ^[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,2} ]]; then - BRO_VERSION="$BRO" -else - echo "[ERROR] $BRO may not be a valid version number" - exit 1 -fi - -echo "Metron Version $METRON_VERSION" -echo "Release Candidate $RC" -echo "Bro Plugin Version $BRO_VERSION" - -METRON_RC_DIST="$METRON_DIST$METRON_VERSION-$UPPER_RC" -echo "Metron RC Distribution Root is $METRON_RC_DIST" - -# working directory -WORK="$HOME/tmp/metron-$METRON_VERSION-$RC" - -# handle tilde expansion -WORK="${WORK/#\~/$HOME}" - -# warn the user if the working directory exists -if [ -d "$WORK" ]; then - echo "[ERROR] Directory $WORK exists, please rename it and start over" - exit 1 -fi - -if [ ! -d "$WORK" ]; then - mkdir -p "$WORK" -fi -echo "Working directory $WORK" - -KEYS="$METRON_RC_DIST/KEYS" -METRON_ASSEMBLY="$METRON_RC_DIST/apache-metron-$METRON_VERSION-$RC.tar.gz" -METRON_ASSEMBLY_SIG="$METRON_ASSEMBLY.asc" -METRON_KAFKA_BRO_ASSEMBLY="$METRON_RC_DIST/apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" -METRON_KAFKA_BRO_ASSEMBLY_ASC="$METRON_KAFKA_BRO_ASSEMBLY.asc" - -echo "Downloading $KEYS" -if ! wget -P "$WORK" "$KEYS" ; then - echo "[ERROR] Failed to download $KEYS" - exit 1 -fi - -echo "Downloading $METRON_ASSEMBLY" -if ! wget -P "$WORK" "$METRON_ASSEMBLY" ; then - echo "[ERROR] Failed to download $METRON_ASSEMBLY" - exit 1 -fi -echo "Downloading $METRON_ASSEMBLY_SIG" -if ! wget -P "$WORK" "$METRON_ASSEMBLY_SIG" ; then - echo "[ERROR] Failed to download $METRON_ASSEMBLY_SIG" - exit 1 -fi -echo "Downloading $METRON_KAFKA_BRO_ASSEMBLY" -if ! wget -P "$WORK" "$METRON_KAFKA_BRO_ASSEMBLY" ; then - echo "[ERROR] Failed to download $METRON_KAFKA_BRO_ASSEMBLY" - exit 1 -fi -echo "Downloading $METRON_KAFKA_BRO_ASSEMBLY_ASC" -if ! wget -P "$WORK" "$METRON_KAFKA_BRO_ASSEMBLY_ASC" ; then - echo "[ERROR] Failed to download $METRON_KAFKA_BRO_ASSEMBLY_ASC" - exit 1 -fi - -cd "$WORK" || exit 1 -echo "importing metron keys" - -if ! gpg --import KEYS ; then - echo "[ERROR] failed to import KEYS" - exit 1 -fi - -echo "Verifying Metron Assembly" -if ! gpg --verify ./"apache-metron-$METRON_VERSION-$RC.tar.gz.asc" "apache-metron-$METRON_VERSION-$RC.tar.gz" ; then - echo "[ERROR] failed to verify Metron Assembly" - exit 1 -fi - -echo "Verifying Bro Kafka Plugin Assembly" -if ! gpg --verify ./"apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz.asc" "apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" ; then - echo "[ERROR] failed to verify Bro Kafka Plugin Assembly" - exit 1 -fi - -echo "Unpacking Assemblies" -if ! tar -xzf "apache-metron-$METRON_VERSION-$RC.tar.gz" ; then - echo "[ERROR] failed to unpack Metron Assembly" - exit 1 -fi - -if ! tar -xzf "apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" ; then - echo "[ERROR] failed to unpack Bro Kafka Plugin Assembly" - exit 1 -fi - -echo "" -echo "" -read -p " run test suite [install, unit tests, integration tests, ui tests, licenses, rpm build]? [yN] " -n 1 -r -echo -DID_BUILD=0 -if [[ $REPLY =~ ^[Yy]$ ]]; then - cd "apache-metron-$METRON_VERSION-$RC" || exit 1 - if ! mvn -q -T 2C -DskipTests clean install ; then - echo "[ERROR] failed to mvn install metron" - exit 1 - fi - if ! mvn -q -T 2C surefire:test@unit-tests ; then - echo "[ERROR] failed unit tests" - exit 1 - fi - if ! mvn -q surefire:test@integration-tests ; then - echo "[ERROR] failed integration tests" - exit 1 - fi - if ! mvn -q test --projects metron-interface/metron-config ; then - echo "[ERROR] failed metron-config tests" - exit 1 - fi - build_utils/verify_licenses.sh | tee ../build-lic.log - cd metron-deployment || exit 1 - if ! mvn -q package -DskipTests -P build-rpms ; then - echo "[ERROR] failed to build rpm" - exit 1 - fi - cd .. || exit 1 - DID_BUILD=1 -fi - -#ask if build test vagrant METRON - -# run tests? -echo "" -echo "" -read -p " run vagrant full_dev? [yN] " -n 1 -r -echo -if [[ $REPLY =~ ^[Yy]$ ]]; then - cd "$WORK/apache-metron-$METRON_VERSION-$RC/metron-deployment/vagrant/full-dev-platform" || exit 1 - if [[ ${DID_BUILD} -ne 1 ]]; then - vagrant up - else - vagrant --ansible-skip-tags="build,sensors,quick-dev" up - fi -fi http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/release-utils/validate-jira-for-release ---------------------------------------------------------------------- diff --git a/build_utils/release-utils/validate-jira-for-release b/build_utils/release-utils/validate-jira-for-release deleted file mode 100755 index df5776c..0000000 --- a/build_utils/release-utils/validate-jira-for-release +++ /dev/null @@ -1,197 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# Finds all commits since the last release tag, then ensures that each -# is marked 'Done' and that the fix version is set to the next release. -# -# For example, to validate JIRA for the 0.4.2 release, you would run the -# following command. -# -# validate-jira-for-release --version=0.4.2 --start=tags/apache-metron-0.4.1-release -# -# This will output a table containing each JIRA that was inspected along with -# the fix version, status, and assignee. If the fix version or status is incorrect -# a link will be printed so that the JIRA can be manually fixed. The JIRA -# only needs to be fixed if a URL is shown. -# -# JIRA STATUS FIX VERSION ASSIGNEE FIX -# METRON-1345 Done 0.4.2 Michael Miklavcic -# METRON-1349 Done Next + 1 Nick Allen https://issues.apache.org/jira/browse/METRON-1349 -# METRON-1343 Done Mohan https://issues.apache.org/jira/browse/METRON-1343 -# ... -# - -function help { - echo " " - echo "usage: ${0}" - echo " -v/--version=<version> The version of the next release. [Required]" - echo " -s/--start=<start> Defines the first commit to inspect. [Required]" - echo " -e/--end=<end> Defines the last commit to inspect. " - echo " -r/--repo=<repo> The Git repo to work from." - echo " -b/--branch=<branch> The branch to work from." - echo " -h/--help Usage information." - echo " " - echo "example: " - echo " validate-jira-for-release --version=0.4.2 --start=tags/apache-metron-0.4.1-release" - echo " " -} - -# define default values -END="HEAD" -REPO="https://git-wip-us.apache.org/repos/asf/metron.git"; -BRANCH="master" - -# print help, if the user just runs this without any args -if [ "$#" -eq 0 ]; then - help - exit 1 -fi - -# handle command line options -for i in "$@"; do - case $i in - # - # VERSION: The release version to validate; the 'next' release. - # - # - -v=*|--version=*) - VERSION="${i#*=}" - shift # past argument=value - ;; - - # - # START: Defines the first commit to inspect - # - # -s=tags/apache-metron-0.4.1-release - # --start=tags/apache-metron-0.4.1-release - # - -s=*|--start=*) - START="${i#*=}" - shift # past argument=value - ;; - - # - # END: Defines the last commit to inspect - # - # -e=HEAD - # --end=HEAD - # - -e=*|--end=*) - END="${i#*=}" - shift # past argument=value - ;; - - # - # REPO: Define the Git repo to work from - # - # -r=https://git-wip-us.apache.org/repos/asf/metron.git - # --repo=<repo-url> - # - -r=*|--repo=*) - REPO="${i#*=}" - shift # past argument=value - ;; - - # - # BRANCH: The branch to work from. - # - # -b=master - # --branch=master - # - -b=*|--branch=*) - BRANCH="${i#*=}" - shift # past argument with no value - ;; - - # - # -h/--help - # - -h|--help) - help - exit 0 - shift # past argument with no value - ;; - - # - # Unknown option - # - *) - UNKNOWN_OPTION="${i#*=}" - echo "Error: unknown option: $UNKNOWN_OPTION" - help - ;; - esac -done - -WORKDIR="~/tmp" - -# ensure all required values are set -if [ -z "$VERSION" ]; then - echo "Missing -v/--version is is required" - exit 1 -fi -if [ -z "$START" ]; then - echo "Missing -s/--start which is required" - exit 1 -fi -if [ -z "$END" ]; then - echo "Missing -e/--end which is required" - exit 1 -fi -if [ -z "$REPO" ]; then - echo "Missing -r/--repo which is required" - exit 1 -fi -if [ -z "$BRANCH" ]; then - echo "Missing -b/--branch which is required" - exit 1 -fi - -# clone the metron repo and fetch all tags -git clone $REPO "metron-$VERSION" -git checkout $BRANCH -cd "$WORKDIR/metron-$VERSION" -git fetch --all --tags - -# find all JIRAs that have been committed since the last release -GET_JIRAS="git log $START..$END --oneline | grep -E -o 'METRON[- ]*[0-9]+'" - -# print the header -FORMAT_STR="%15s %15s %15s %30s %50s\n" -printf "$FORMAT_STR" "JIRA" "STATUS" "FIX VERSION" "ASSIGNEE" "FIX" - -# for each JIRA since the last release tag... -eval $GET_JIRAS | while read JIRA ; do - - # fetch the JIRA content - URL="https://issues.apache.org/jira/si/jira.issueviews:issue-xml/$JIRA/$JIRA.xml"; - CONTENT=`curl -s $URL` - - # painfully extract some fields - STATUS=`echo "$CONTENT" | grep "<status[^>]*>" | sed 's/^.*<status[^>]*>//' | sed 's/<.status>.*$//'` - ASSIGNEE=`echo "$CONTENT" | grep "<assignee[^>]*>" | sed 's/^.*<assignee[^>]*>//' | sed 's/<.assignee>.*$//'` - FIXV=`echo "$CONTENT" | grep "<fixVersion[^>]*>" | sed 's/^.*<fixVersion[^>]*>//' | sed 's/<.fixVersion>.*$//'` - - # the link is only populated, if there is something to fix - LINK="" - if [ "$FIXV" != "$VERSION" ] || [ "$STATUS" != "Done" ]; then - LINK="https://issues.apache.org/jira/browse/$JIRA"; - fi - - # show the JIRA - printf "$FORMAT_STR" "$JIRA" "$STATUS" "$FIXV" "$ASSIGNEE" "$LINK" -done http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/verify_license.py ---------------------------------------------------------------------- diff --git a/build_utils/verify_license.py b/build_utils/verify_license.py deleted file mode 100644 index e9e9cfd..0000000 --- a/build_utils/verify_license.py +++ /dev/null @@ -1,44 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -import sets -import sys - -def read_component(i): - with open(i, 'r') as fp: - component_lines = fp.readlines() - ret = [] - for line in component_lines: - if len(line) > 0: - l = line.split(',')[0].strip() - ret.append(l) - return sets.Set(ret) - -if __name__ == '__main__': - components = read_component(sys.argv[1]) - components_not_found = [] - for line in sys.stdin: - component = line.strip() - if len(component) == 0 or component == 'none' or component in components: - continue - else: - if len(sys.argv) > 2: - print component - else: - components_not_found.append(component) - if len(components_not_found) > 0: - raise ValueError("Unable to find these components: \n " + "\n ".join(components_not_found) + "\nin the acceptable list of components: " + sys.argv[1]) http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/build_utils/verify_licenses.sh ---------------------------------------------------------------------- diff --git a/build_utils/verify_licenses.sh b/build_utils/verify_licenses.sh deleted file mode 100755 index 5b1d1ee..0000000 --- a/build_utils/verify_licenses.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -build_utils/list_dependencies.sh | python build_utils/verify_license.py ./dependencies_with_url.csv http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/build-utils/README.md ---------------------------------------------------------------------- diff --git a/dev-utilities/build-utils/README.md b/dev-utilities/build-utils/README.md new file mode 100644 index 0000000..495a256 --- /dev/null +++ b/dev-utilities/build-utils/README.md @@ -0,0 +1,64 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> +# Build Utilities + +The aim of the build utilities project is to provide some scripting +around the care and maintenance of the building infrastructure. At the +moment the primary mission is around utilities that assist us manage the +licenses of our dependencies and generate the appropriate notices or +licenses. + +## `dependencies_with_url.csv` +This file is the reference file for all of our dependencies. +If you add a dependency, you must add a line to the +`dependencies_with_url.csv` file. + +## `list_dependencies.sh` + +List all of the transitive dependencies for the project rooted at cwd. + +## `verify_licenses.sh` + +This script, as run by our travis build infrastructure, will look at the +dependencies and verify that we know about them. Travis will use this +script which takes the transitive dependency list and check against the +`dependencies_with_url.csv` file to ensure that it's listed. This will +make sure we track dependencies and do not have any unacceptable +dependencies. + +If you want to dump all of the dependencies that it doesn't know about, +from the top level directory: + +``` +dev-utilities/build-utils/list_dependencies.sh | python dev-utilities/build-utils/verify_license.py ./dependencies_with_url.csv dump +``` + + +## `create_bundled_licenses.sh` + +This script is intended to regenerate the licenses for each project that +bundles its dependencies. Because we bundle our dependencies in a +shaded jar, we +[must](http://www.apache.org/dev/licensing-howto.html#deps-of-deps) specify a `LICENSE` file with the permissively +licensed dependencies notated as per [here](http://www.apache.org/dev/licensing-howto.html#permissive-deps) + +Example command to regenerate licenses (run from top level directory): + +``` +for i in $(find . -name LICENSE | grep src | grep META-INF | awk -Fsrc '{print $1}');do dev-utilities/build-utils/create_bundled_licenses.sh $i;done +``` http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/build-utils/create_bundled_licenses.sh ---------------------------------------------------------------------- diff --git a/dev-utilities/build-utils/create_bundled_licenses.sh b/dev-utilities/build-utils/create_bundled_licenses.sh new file mode 100755 index 0000000..b853bad --- /dev/null +++ b/dev-utilities/build-utils/create_bundled_licenses.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +TLD=$(pwd) +LICENSE_FILE="src/main/resources/META-INF/LICENSE" +pushd $1 +$TLD/dev-utilities/build-utils/list_dependencies.sh | + python $TLD/dev-utilities/build-utils/generate_license.py $TLD/dependencies_with_url.csv $TLD/LICENSE 1> $LICENSE_FILE +popd http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/build-utils/generate_license.py ---------------------------------------------------------------------- diff --git a/dev-utilities/build-utils/generate_license.py b/dev-utilities/build-utils/generate_license.py new file mode 100644 index 0000000..366e515 --- /dev/null +++ b/dev-utilities/build-utils/generate_license.py @@ -0,0 +1,108 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +import sets +import sys + +category_a_licenses = [ "BSD Software License", + "Creative Commons License", + "Common Development and Distribution License", + "Common Development and Distribution License v1.1", + "Common Development and Distribution License v1.0", + "Common Public License v1.0", + "Eclipse Public License v1.0", + "MIT Software License", + "Mozilla Public License v2.0" + ] + + +license_mapping = { + "ASLv2": "Apache Software License v2", + "Apache 2": "Apache Software License v2", + "Apache 2.0": "Apache Software License v2", + "Apache License": "Apache Software License v2", + "Apache License 2.0": "Apache Software License v2", + "Apache License V2.0": "Apache Software License v2", + "Apache License Version 2.0": "Apache Software License v2", + "Apache Software Licenses": "Apache Software License v2", + "Apache v2": "Apache Software License v2", + "The Apache Software License": "Apache Software License v2", + "BSD": "BSD Software License", + "BSD 2-clause": "BSD Software License", + "BSD 3-Clause \"New\" or \"Revised\" License (BSD-3-Clause)" : "BSD Software License", + "BSD 3-Clause License": "BSD Software License", + "BSD 3-clause": "BSD Software License", + "BSD-like": "BSD Software License", + "CC0 1.0 Universal": "Creative Commons License", + "CDDL": "Common Development and Distribution License", + "CDDL 1.1": "Common Development and Distribution License v1.1", + "COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0": "Common Development and Distribution License v1.0", + "Common Development and Distribution License (CDDL) v1.0": "Common Development and Distribution License v1.0", + "Common Public License Version 1.0": "Common Public License v1.0", + "Eclipse Public License 1.0": "Eclipse Public License v1.0", + "MIT" : "MIT Software License", + "MIT License" : "MIT Software License", + "Mozilla Public License Version 2.0" : "Mozilla Public License v2.0", + "New BSD License" : "BSD Software License", + "New BSD license" : "BSD Software License", + "Public" : "Public Domain", + "Public Domain" : "Public Domain", + "The BSD 3-Clause License" : "BSD Software License", + "The BSD License" : "BSD Software License", + "The MIT License" : "MIT Software License", + "ACCEPTABLE" : "Ignore" +} + +def read_component(i): + with open(i, 'r') as fp: + component_lines = fp.readlines() + ret = {} + for line in component_lines: + if len(line) > 0: + tokens = line.split(',') + key = tokens[0] + url = tokens[-1].strip() + license = license_mapping[tokens[1].strip()] + if license is None: + raise ValueError("unable to normalize license: " + tokens[1]) + l = line.split(',')[0].strip() + ret[key] = { 'url' : url, 'license' : license } + return ret + +def read_license(f) : + with open(f) as fp: + return fp.read() + +def get_blurb(component, license_info): + tokens = component.split(':') + artifact_id = tokens[1] + version = tokens[3] + return "This product bundles " + artifact_id + " " + version + ", which is available under a \"" + license_info['license'] + "\" license. " + "For details, see " + license_info['url'] + +if __name__ == '__main__': + components = read_component(sys.argv[1]) + license = read_license(sys.argv[2]) + for line in sys.stdin: + component = line.strip() + if len(component) == 0 or component == 'none' or component not in components: + continue + else: + license_info = components[component] + if license_info['license'] in category_a_licenses: + license = license + "\n" + get_blurb(component, license_info) + continue + print license http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/build-utils/list_dependencies.sh ---------------------------------------------------------------------- diff --git a/dev-utilities/build-utils/list_dependencies.sh b/dev-utilities/build-utils/list_dependencies.sh new file mode 100755 index 0000000..a56a5bd --- /dev/null +++ b/dev-utilities/build-utils/list_dependencies.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +{ mvn dependency:list ; mvn dependency:list -PHDP-2.5.0.0 ; } | grep "^\[INFO\] " | awk '{print $2}' | grep -v "org.apache" | grep -v "test" | grep -v "provided" | grep -v "runtime" | grep -v ":system" | sort | uniq http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/build-utils/verify_license.py ---------------------------------------------------------------------- diff --git a/dev-utilities/build-utils/verify_license.py b/dev-utilities/build-utils/verify_license.py new file mode 100644 index 0000000..e9e9cfd --- /dev/null +++ b/dev-utilities/build-utils/verify_license.py @@ -0,0 +1,44 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +import sets +import sys + +def read_component(i): + with open(i, 'r') as fp: + component_lines = fp.readlines() + ret = [] + for line in component_lines: + if len(line) > 0: + l = line.split(',')[0].strip() + ret.append(l) + return sets.Set(ret) + +if __name__ == '__main__': + components = read_component(sys.argv[1]) + components_not_found = [] + for line in sys.stdin: + component = line.strip() + if len(component) == 0 or component == 'none' or component in components: + continue + else: + if len(sys.argv) > 2: + print component + else: + components_not_found.append(component) + if len(components_not_found) > 0: + raise ValueError("Unable to find these components: \n " + "\n ".join(components_not_found) + "\nin the acceptable list of components: " + sys.argv[1]) http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/build-utils/verify_licenses.sh ---------------------------------------------------------------------- diff --git a/dev-utilities/build-utils/verify_licenses.sh b/dev-utilities/build-utils/verify_licenses.sh new file mode 100755 index 0000000..ffd6e48 --- /dev/null +++ b/dev-utilities/build-utils/verify_licenses.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +dev-utilities/build-utils/list_dependencies.sh | python dev-utilities/build-utils/verify_license.py ./dependencies_with_url.csv http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/committer-utils/README.md ---------------------------------------------------------------------- diff --git a/dev-utilities/committer-utils/README.md b/dev-utilities/committer-utils/README.md new file mode 100644 index 0000000..60e1d51 --- /dev/null +++ b/dev-utilities/committer-utils/README.md @@ -0,0 +1,114 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +--> + +# Committer Tools + +This project contains tools to assist Apache Metron project committers. + +## Prepare Commit + +This script automates the process of merging a pull request into `apache/master`. The script will prompt for the pull request number. Most of the remaining information is automatically extracted from Github or the Apache JIRA. + +When prompted the `[value in brackets]` is used by default. To accept the default, simply press `enter`. If you would like to change the default, type it in and hit `enter` when done. + +In the following example, I enter the pull request number when prompted. Using the pull request number, the script can extract most of the remaining required information. + +1. Execute the script. + + The first time the script is run, you will be prompted for additional information including your Apache username, Apache email, and Github username. These values are persisted in `~/.metron-prepare-commit`. Subsequent executions of the script will retrieve these values, rather than prompting you again for them. + + ``` + $ prepare-commit + your github username []: nickwallen + your apache userid []: nickallen + your apache email [[email protected]]: + ``` + +1. Enter the Github pull request number. + + ``` + pull request: 897 + local working directory [/Users/nallen/tmp/metron-pr897]: + origin repo [https://github.com/apache/metron]: + + Cloning into '/Users/nallen/tmp/metron-pr897'... + remote: Counting objects: 36277, done. + remote: Compressing objects: 100% (108/108), done. + remote: Total 36277 (delta 38), reused 54 (delta 20), pack-reused 36138 + Receiving objects: 100% (36277/36277), 57.85 MiB | 7.36 MiB/s, done. + Resolving deltas: 100% (13653/13653), done. + From https://git-wip-us.apache.org/repos/asf/metron + * branch master -> FETCH_HEAD + * [new branch] master -> upstream/master + Already on 'master' + Your branch is up to date with 'origin/master'. + Already up to date. + remote: Counting objects: 5, done. + remote: Total 5 (delta 3), reused 3 (delta 3), pack-reused 2 + Unpacking objects: 100% (5/5), done. + From https://github.com/apache/metron + * [new ref] refs/pull/897/head -> pr-897 + ``` + +1. Enter contribution details. + + The contributor's username, email, along with information about the associated Apache JIRA is extracted from the commit history. + + ``` + github contributor's username [MohanDV]: + github contributor's email [[email protected]]: + issue identifier in jira [METRON-1395]: + issue description [Documentation missing for Produce a message to a Kafka topic Rest API endpoint]: + commit message [METRON-1395 Documentation missing for Produce a message to a Kafka topic Rest API endpoint (MohanDV via nickwallen) closes apache/metron#897]: + ``` + +1. The contribution is then merged with master as a single commit. The changes that have been made along with the commit message are displayed. + + ``` + Squash commit -- not updating HEAD + Automatic merge went well; stopped before committing as requested + [master 998f7915] METRON-1410 Some more upgrade fallout... Can't restart Metron Indexing. (ottobackwards via nickwallen) closes apache/metron#901 + Author: ottobackwards <[email protected]> + 3 files changed, 3 insertions(+), 3 deletions(-) + + + .../metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py | 2 +- + .../ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py | 2 +- + .../ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/profiler_commands.py | 2 +- + 3 files changed, 3 insertions(+), 3 deletions(-) + + 998f7915 (HEAD -> master) METRON-1410 Cannot restart Metron Indexing. (ottobackwards via nickwallen) closes apache/metron#901 + ``` + +1. Run the test suite. + + After the merge is complete, the script will prompt you to run the test suite. By default this is skipped, but by typing 'y' the test suite will be run. + + ``` + run test suite? [yN] + ``` + +1. Finalize the changes. + + To this point changes have only been made to your local repository. The script itself will not push changes to Apache. You are given instructions on how to do so. Review the summary and enter `y` at the prompt, if you are satisfied. If you are not happy, simply start over. + + ``` + Review commit carefully then run... + cd /Users/nallen/tmp/metron-pr897 + git push upstream master + ``` http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/committer-utils/prepare-commit ---------------------------------------------------------------------- diff --git a/dev-utilities/committer-utils/prepare-commit b/dev-utilities/committer-utils/prepare-commit new file mode 100755 index 0000000..90852c0 --- /dev/null +++ b/dev-utilities/committer-utils/prepare-commit @@ -0,0 +1,216 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# not likely to change +UPSTREAM=https://git-wip-us.apache.org/repos/asf/metron.git +BASE_BRANCH=master +CONFIG_FILE=~/.metron-prepare-commit +GITHUB_REMOTE="origin" + +# does a config file already exist? +if [ -f $CONFIG_FILE ]; then + . $CONFIG_FILE + echo " ...using settings from $CONFIG_FILE" +fi + +# github account of committer (you) +if [ -z "$GITHUB_NAME" ]; then + read -p " your github username [$GITHUB_NAME]: " INPUT + [ -n "$INPUT" ] && GITHUB_NAME=$INPUT + + # write setting to config file + echo "GITHUB_NAME=$GITHUB_NAME" >> $CONFIG_FILE +fi + +# apache id of committer (you) +if [ -z "$APACHE_NAME" ]; then + read -p " your apache userid [$APACHE_NAME]: " INPUT + [ -n "$INPUT" ] && APACHE_NAME=$INPUT + + # write setting to config file + echo "APACHE_NAME=$APACHE_NAME" >> $CONFIG_FILE +fi + +# apache email addr of committer (you) +if [ -z "$APACHE_EMAIL" ]; then + APACHE_EMAIL=${APACHE_NAME}@apache.org + read -p " your apache email [$APACHE_EMAIL]: " INPUT + [ -n "$INPUT" ] && APACHE_EMAIL=$INPUT + + # write setting to config file, so it is not needed next time + echo "APACHE_EMAIL=$APACHE_EMAIL" >> $CONFIG_FILE +fi + +# retrieve the pull request identifier +read -p " pull request: " PR +if [ -z "$PR" ]; then + echo "Error: missing pr" + exit 1 +fi + +# ensure that the pull request exists +PR_EXISTS=`curl -sI https://api.github.com/repos/apache/metron/pulls/$PR | grep Status: | sed 's/[^0-9]//g'` +if [ "$PR_EXISTS" != "200" ]; then + echo "Error: pull request #$PR does not exist" + exit 1 +fi + +# working directory +WORK=~/tmp/metron-pr$PR +read -p " local working directory [$WORK]: " INPUT +[ -n "$INPUT" ] && WORK=$INPUT + +# handle tilde expansion +WORK="${WORK/#\~/$HOME}" + +# warn the user if the working directory exists +if [ -d "$WORK" ]; then + read -p " directory exists [$WORK]. continue merge on existing repo? [yN] " -n 1 -r + echo + if [[ ! $REPLY =~ ^[Yy]$ ]]; then + exit 1 + fi +fi + +# if working directory does not exist, checkout the base branch +if [ ! -d "$WORK" ]; then + + # origin repository + ORIGIN="https://github.com/apache/metron"; + read -p " origin repo [$ORIGIN]: " INPUT + [ -n "$INPUT" ] && ORIGIN=$INPUT + + # clone the repository and fetch updates + mkdir -p $WORK + git clone $ORIGIN $WORK + cd $WORK + + # setup the git user and email for your apache account + git config user.name "$APACHE_NAME" + git config user.email $APACHE_EMAIL + + # fetch any changes from upstream + git remote add upstream $UPSTREAM + git fetch upstream $BASE_BRANCH + + # merge any changes from upstream + git checkout $BASE_BRANCH + git merge upstream/$BASE_BRANCH + +else + + # if the repo already exists, allow the user to provide the name of the Github remote + # this is needed to checkout the code for the PR + read -p " name of github remote [$GITHUB_REMOTE]: " INPUT + [ -n "$INPUT" ] && GITHUB_REMOTE=$INPUT + +fi + +PR_BRANCH_REF="pull/$PR/head:pr-$PR" +PR_BRANCH="pr-$PR" +cd $WORK +git fetch $GITHUB_REMOTE $PR_BRANCH_REF +echo "" + +# use github api to retrieve the contributor's login +USER=`curl -s https://api.github.com/repos/apache/metron/pulls/$PR | grep login | head -1 | awk -F":" '{print $2}' | sed 's/[^a-zA-Z.@_-]//g'` +read -p " github contributor's username [$USER]: " INPUT +[ -n "$INPUT" ] && USER=$INPUT + +# validate the github contributor +if [ -z "$USER" ]; then + echo "Error: missing username" + exit 1 +fi + +# retrieve the contributor's email from the git commit history +EMAIL=`git log $PR_BRANCH | grep Author | head -1 | awk -F"<" '{print $2}' | sed 's/[<>]//g'` +read -p " github contributor's email [$EMAIL]: " INPUT +[ -n "$INPUT" ] && EMAIL=$INPUT + +# validate email +if [ -z "$EMAIL" ] || [ "$EMAIL" = "null" ]; then + echo "Error: missing email" + exit 1 +fi + +# can we extract the JIRA from the PR title? +JIRA=`curl -s https://api.github.com/repos/apache/metron/pulls/$PR | grep title | head -1 | egrep -o -i 'METRON-[0-9]+' | awk '{print toupper($0)}'` +read -p " issue identifier in jira [$JIRA]: " INPUT +[ -n "$INPUT" ] && JIRA=$INPUT + +# validate the JIRA issue +if [ -z "$JIRA" ]; then + echo "Error: missing jira" + exit 1 +fi + +# attempt to use the jira api to get a description of the jira +DESC=`curl -s https://issues.apache.org/jira/si/jira.issueviews:issue-xml/$JIRA/$JIRA.xml | grep "<summary>" | sed 's/^.*<summary>//' | sed 's/<.summary>.*$//'` +read -p " issue description [$DESC]: " INPUT +[ -n "$INPUT" ] && DESC=$INPUT + +# validate description +if [ -z "$DESC" ]; then + echo "Error: missing description" + exit 1 +fi + +# commit message +AUTHOR="$USER <$EMAIL>" +if [ "$USER" == "$GITHUB_NAME" ]; then + MSG="$JIRA $DESC ($USER) closes apache/metron#$PR" +else + MSG="$JIRA $DESC ($USER via $GITHUB_NAME) closes apache/metron#$PR" +fi +read -p " commit message [$MSG]: " INPUT +[ -n "$INPUT" ] && MSG=$INPUT + +# merge the contributor's branch and commit +echo "" +if git merge --squash "$PR_BRANCH"; then + git commit --author="$AUTHOR" -a -m "$MSG" +else + exit $? +fi + +# review the commit +echo "" +echo "" +git diff --stat --color "upstream/$BASE_BRANCH..$BASE_BRANCH" +echo "" +echo "" +git log --oneline "$BASE_BRANCH" "^upstream/$BASE_BRANCH" + +# run tests? +echo "" +echo "" +read -p " run test suite? [yN] " -n 1 -r +echo +if [[ $REPLY =~ ^[Yy]$ ]]; then + mvn -q -T 2C -DskipTests clean install && + mvn -q -T 2C surefire:test@unit-tests && + mvn -q surefire:test@integration-tests && + mvn -q test --projects metron-interface/metron-config && + dev-utilities/build-utils/verify_licenses.sh +fi + +echo "" +echo "Review commit carefully then run..." +echo " cd $WORK" +echo " git push upstream master" +echo "" http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/release-utils/metron-rc-check ---------------------------------------------------------------------- diff --git a/dev-utilities/release-utils/metron-rc-check b/dev-utilities/release-utils/metron-rc-check new file mode 100755 index 0000000..da5412e --- /dev/null +++ b/dev-utilities/release-utils/metron-rc-check @@ -0,0 +1,269 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +shopt -s nocasematch + +function help { + echo " " + echo "usage: ${0}" + echo " -v/--version=<version> The version of the metron release. [Required]" + echo " -c/--candidate=<RC#> Defines the Release Candidate. [Required]" + echo " -b/--bro=<bro version> The version of the bro kafka plugin. [Required]" + echo " -h/--help Usage information." + echo " " + echo "example: " + echo " metron-rc-check --version=0.4.2 --candidate=RC2 --bro=0.1.0" + echo " " +} + +METRON_DIST="https://dist.apache.org/repos/dist/dev/metron/"; +# print help, if the user just runs this without any args +if [ "$#" -eq 0 ]; then + help + exit 1 +fi + +# handle command line options +for i in "$@"; do + case $i in + # + # VERSION: The release version of Metron to validate. + # + # + -v=*|--version=*) + VERSION="${i#*=}" + shift # past argument=value + ;; + + # + # RC: Defines the RC# to use + # + # -c=RC2 + # --candidate=RC2 + # + -c=*|--candidate=*) + CANDIDATE="${i#*=}" + shift # past argument=value + ;; + + # + # END: Defines the last commit to inspect + # + # -b=0.1.0 + # --bro=0.1.0 + # + -b=*|--bro=*) + BRO="${i#*=}" + shift # past argument=value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +# validation +if [ -z "$VERSION" ]; then + echo "Missing -v/--version is is required" + exit 1 +fi +if [[ "$VERSION" =~ ^[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,2} ]]; then + METRON_VERSION="$VERSION" +else + echo "[ERROR] "$VERSION" may not be a valid version number" + exit 1 +fi + +if [ -z "$CANDIDATE" ]; then + echo "Missing -c/--candidate which is required" + exit 1 +fi + +if [[ "$CANDIDATE" =~ ^RC[0-9]+ ]]; then + RC=$(echo "$CANDIDATE" | tr '[:upper:]' '[:lower:]') + UPPER_RC=$(echo "$CANDIDATE" | tr '[:lower:]' '[:upper:]') +elif [[ "$CANDIDATE" =~ ^[0-9]+ ]]; then + RC=rc"$CANDIDATE" + UPPER_RC=RC"$CANDIDATE" +else + echo "[ERROR] invalid RC, valid is RC# or just #" + exit 1 +fi + +if [ -z "$BRO" ]; then + echo "Missing -b/--bro which is required" + exit 1 +fi + +if [[ "$BRO" =~ ^[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,2} ]]; then + BRO_VERSION="$BRO" +else + echo "[ERROR] $BRO may not be a valid version number" + exit 1 +fi + +echo "Metron Version $METRON_VERSION" +echo "Release Candidate $RC" +echo "Bro Plugin Version $BRO_VERSION" + +METRON_RC_DIST="$METRON_DIST$METRON_VERSION-$UPPER_RC" +echo "Metron RC Distribution Root is $METRON_RC_DIST" + +# working directory +WORK="$HOME/tmp/metron-$METRON_VERSION-$RC" + +# handle tilde expansion +WORK="${WORK/#\~/$HOME}" + +# warn the user if the working directory exists +if [ -d "$WORK" ]; then + echo "[ERROR] Directory $WORK exists, please rename it and start over" + exit 1 +fi + +if [ ! -d "$WORK" ]; then + mkdir -p "$WORK" +fi +echo "Working directory $WORK" + +KEYS="$METRON_RC_DIST/KEYS" +METRON_ASSEMBLY="$METRON_RC_DIST/apache-metron-$METRON_VERSION-$RC.tar.gz" +METRON_ASSEMBLY_SIG="$METRON_ASSEMBLY.asc" +METRON_KAFKA_BRO_ASSEMBLY="$METRON_RC_DIST/apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" +METRON_KAFKA_BRO_ASSEMBLY_ASC="$METRON_KAFKA_BRO_ASSEMBLY.asc" + +echo "Downloading $KEYS" +if ! wget -P "$WORK" "$KEYS" ; then + echo "[ERROR] Failed to download $KEYS" + exit 1 +fi + +echo "Downloading $METRON_ASSEMBLY" +if ! wget -P "$WORK" "$METRON_ASSEMBLY" ; then + echo "[ERROR] Failed to download $METRON_ASSEMBLY" + exit 1 +fi +echo "Downloading $METRON_ASSEMBLY_SIG" +if ! wget -P "$WORK" "$METRON_ASSEMBLY_SIG" ; then + echo "[ERROR] Failed to download $METRON_ASSEMBLY_SIG" + exit 1 +fi +echo "Downloading $METRON_KAFKA_BRO_ASSEMBLY" +if ! wget -P "$WORK" "$METRON_KAFKA_BRO_ASSEMBLY" ; then + echo "[ERROR] Failed to download $METRON_KAFKA_BRO_ASSEMBLY" + exit 1 +fi +echo "Downloading $METRON_KAFKA_BRO_ASSEMBLY_ASC" +if ! wget -P "$WORK" "$METRON_KAFKA_BRO_ASSEMBLY_ASC" ; then + echo "[ERROR] Failed to download $METRON_KAFKA_BRO_ASSEMBLY_ASC" + exit 1 +fi + +cd "$WORK" || exit 1 +echo "importing metron keys" + +if ! gpg --import KEYS ; then + echo "[ERROR] failed to import KEYS" + exit 1 +fi + +echo "Verifying Metron Assembly" +if ! gpg --verify ./"apache-metron-$METRON_VERSION-$RC.tar.gz.asc" "apache-metron-$METRON_VERSION-$RC.tar.gz" ; then + echo "[ERROR] failed to verify Metron Assembly" + exit 1 +fi + +echo "Verifying Bro Kafka Plugin Assembly" +if ! gpg --verify ./"apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz.asc" "apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" ; then + echo "[ERROR] failed to verify Bro Kafka Plugin Assembly" + exit 1 +fi + +echo "Unpacking Assemblies" +if ! tar -xzf "apache-metron-$METRON_VERSION-$RC.tar.gz" ; then + echo "[ERROR] failed to unpack Metron Assembly" + exit 1 +fi + +if ! tar -xzf "apache-metron-bro-plugin-kafka_$BRO_VERSION.tar.gz" ; then + echo "[ERROR] failed to unpack Bro Kafka Plugin Assembly" + exit 1 +fi + +echo "" +echo "" +read -p " run test suite [install, unit tests, integration tests, ui tests, licenses, rpm build]? [yN] " -n 1 -r +echo +DID_BUILD=0 +if [[ $REPLY =~ ^[Yy]$ ]]; then + cd "apache-metron-$METRON_VERSION-$RC" || exit 1 + if ! mvn -q -T 2C -DskipTests clean install ; then + echo "[ERROR] failed to mvn install metron" + exit 1 + fi + if ! mvn -q -T 2C surefire:test@unit-tests ; then + echo "[ERROR] failed unit tests" + exit 1 + fi + if ! mvn -q surefire:test@integration-tests ; then + echo "[ERROR] failed integration tests" + exit 1 + fi + if ! mvn -q test --projects metron-interface/metron-config ; then + echo "[ERROR] failed metron-config tests" + exit 1 + fi + dev-utilities/build-utils/verify_licenses.sh | tee ../build-lic.log + cd metron-deployment || exit 1 + if ! mvn -q package -DskipTests -P build-rpms ; then + echo "[ERROR] failed to build rpm" + exit 1 + fi + cd .. || exit 1 + DID_BUILD=1 +fi + +#ask if build test vagrant METRON + +# run tests? +echo "" +echo "" +read -p " run vagrant full_dev? [yN] " -n 1 -r +echo +if [[ $REPLY =~ ^[Yy]$ ]]; then + cd "$WORK/apache-metron-$METRON_VERSION-$RC/metron-deployment/vagrant/full-dev-platform" || exit 1 + if [[ ${DID_BUILD} -ne 1 ]]; then + vagrant up + else + vagrant --ansible-skip-tags="build,sensors,quick-dev" up + fi +fi http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/dev-utilities/release-utils/validate-jira-for-release ---------------------------------------------------------------------- diff --git a/dev-utilities/release-utils/validate-jira-for-release b/dev-utilities/release-utils/validate-jira-for-release new file mode 100755 index 0000000..df5776c --- /dev/null +++ b/dev-utilities/release-utils/validate-jira-for-release @@ -0,0 +1,197 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Finds all commits since the last release tag, then ensures that each +# is marked 'Done' and that the fix version is set to the next release. +# +# For example, to validate JIRA for the 0.4.2 release, you would run the +# following command. +# +# validate-jira-for-release --version=0.4.2 --start=tags/apache-metron-0.4.1-release +# +# This will output a table containing each JIRA that was inspected along with +# the fix version, status, and assignee. If the fix version or status is incorrect +# a link will be printed so that the JIRA can be manually fixed. The JIRA +# only needs to be fixed if a URL is shown. +# +# JIRA STATUS FIX VERSION ASSIGNEE FIX +# METRON-1345 Done 0.4.2 Michael Miklavcic +# METRON-1349 Done Next + 1 Nick Allen https://issues.apache.org/jira/browse/METRON-1349 +# METRON-1343 Done Mohan https://issues.apache.org/jira/browse/METRON-1343 +# ... +# + +function help { + echo " " + echo "usage: ${0}" + echo " -v/--version=<version> The version of the next release. [Required]" + echo " -s/--start=<start> Defines the first commit to inspect. [Required]" + echo " -e/--end=<end> Defines the last commit to inspect. " + echo " -r/--repo=<repo> The Git repo to work from." + echo " -b/--branch=<branch> The branch to work from." + echo " -h/--help Usage information." + echo " " + echo "example: " + echo " validate-jira-for-release --version=0.4.2 --start=tags/apache-metron-0.4.1-release" + echo " " +} + +# define default values +END="HEAD" +REPO="https://git-wip-us.apache.org/repos/asf/metron.git"; +BRANCH="master" + +# print help, if the user just runs this without any args +if [ "$#" -eq 0 ]; then + help + exit 1 +fi + +# handle command line options +for i in "$@"; do + case $i in + # + # VERSION: The release version to validate; the 'next' release. + # + # + -v=*|--version=*) + VERSION="${i#*=}" + shift # past argument=value + ;; + + # + # START: Defines the first commit to inspect + # + # -s=tags/apache-metron-0.4.1-release + # --start=tags/apache-metron-0.4.1-release + # + -s=*|--start=*) + START="${i#*=}" + shift # past argument=value + ;; + + # + # END: Defines the last commit to inspect + # + # -e=HEAD + # --end=HEAD + # + -e=*|--end=*) + END="${i#*=}" + shift # past argument=value + ;; + + # + # REPO: Define the Git repo to work from + # + # -r=https://git-wip-us.apache.org/repos/asf/metron.git + # --repo=<repo-url> + # + -r=*|--repo=*) + REPO="${i#*=}" + shift # past argument=value + ;; + + # + # BRANCH: The branch to work from. + # + # -b=master + # --branch=master + # + -b=*|--branch=*) + BRANCH="${i#*=}" + shift # past argument with no value + ;; + + # + # -h/--help + # + -h|--help) + help + exit 0 + shift # past argument with no value + ;; + + # + # Unknown option + # + *) + UNKNOWN_OPTION="${i#*=}" + echo "Error: unknown option: $UNKNOWN_OPTION" + help + ;; + esac +done + +WORKDIR="~/tmp" + +# ensure all required values are set +if [ -z "$VERSION" ]; then + echo "Missing -v/--version is is required" + exit 1 +fi +if [ -z "$START" ]; then + echo "Missing -s/--start which is required" + exit 1 +fi +if [ -z "$END" ]; then + echo "Missing -e/--end which is required" + exit 1 +fi +if [ -z "$REPO" ]; then + echo "Missing -r/--repo which is required" + exit 1 +fi +if [ -z "$BRANCH" ]; then + echo "Missing -b/--branch which is required" + exit 1 +fi + +# clone the metron repo and fetch all tags +git clone $REPO "metron-$VERSION" +git checkout $BRANCH +cd "$WORKDIR/metron-$VERSION" +git fetch --all --tags + +# find all JIRAs that have been committed since the last release +GET_JIRAS="git log $START..$END --oneline | grep -E -o 'METRON[- ]*[0-9]+'" + +# print the header +FORMAT_STR="%15s %15s %15s %30s %50s\n" +printf "$FORMAT_STR" "JIRA" "STATUS" "FIX VERSION" "ASSIGNEE" "FIX" + +# for each JIRA since the last release tag... +eval $GET_JIRAS | while read JIRA ; do + + # fetch the JIRA content + URL="https://issues.apache.org/jira/si/jira.issueviews:issue-xml/$JIRA/$JIRA.xml"; + CONTENT=`curl -s $URL` + + # painfully extract some fields + STATUS=`echo "$CONTENT" | grep "<status[^>]*>" | sed 's/^.*<status[^>]*>//' | sed 's/<.status>.*$//'` + ASSIGNEE=`echo "$CONTENT" | grep "<assignee[^>]*>" | sed 's/^.*<assignee[^>]*>//' | sed 's/<.assignee>.*$//'` + FIXV=`echo "$CONTENT" | grep "<fixVersion[^>]*>" | sed 's/^.*<fixVersion[^>]*>//' | sed 's/<.fixVersion>.*$//'` + + # the link is only populated, if there is something to fix + LINK="" + if [ "$FIXV" != "$VERSION" ] || [ "$STATUS" != "Done" ]; then + LINK="https://issues.apache.org/jira/browse/$JIRA"; + fi + + # show the JIRA + printf "$FORMAT_STR" "$JIRA" "$STATUS" "$FIXV" "$ASSIGNEE" "$LINK" +done http://git-wip-us.apache.org/repos/asf/metron/blob/24822ddd/site-book/bin/generate-md.sh ---------------------------------------------------------------------- diff --git a/site-book/bin/generate-md.sh b/site-book/bin/generate-md.sh index 818ab99..464cb69 100755 --- a/site-book/bin/generate-md.sh +++ b/site-book/bin/generate-md.sh @@ -19,7 +19,7 @@ # ------------------------------------------------------------------ # # This script collects the *.md files and other resources needed to generate -# a book-like collection of end-user documentation. +# a book-like collection of end-user documentation. # # The Metron development community has chosen to do most documentation in README.md # files, because they are easy to write and maintain, and located near the code they @@ -48,7 +48,7 @@ METRON_SOURCE=`cd $(dirname $0); cd ../..; pwd` EXCLUSION_LIST=( '/site/' '/site-book/' - '/build_utils/' + '/dev-utilities/' '/node_modules/' '/\.github/' ) @@ -330,4 +330,3 @@ else echo " " exit 0 fi -
