From: Waldemar Kozaczuk <[email protected]> Committer: Waldemar Kozaczuk <[email protected]> Branch: master
travis CI: setup nightly publishing of OSv artifacts This patch sets up OSv travis CI pipeline to build and publish kernel as well as couple of capstan packages for every push to the main OSv repo. To make it possible, we use an open source project - ci-release-publisher - that provides python and shell scripts to automate building and publishing of build artifacts to github repo as part of Travis build chain. For more details please read https://github.com/nurupo/ci-release-publisher#ci-release-publisher. Please note that OSv setup follows "Publishing to a different repository" approach. In high-level everytime new commit is pushed to cloudius-systems/osv repo, travis would execute logic defined in .travis.yml that: - builds OSv docker builder image as defined in docker/Dockerfile.builder - runs OSv build in docker container to build kernel and unit tests - publishes kernel and unit tests in form or capstan mpm file into new repo - https://github.com/osvunikernel/osv-nightly-releases/releases. Eventually we might tweak the setup and publish artifacts once a day if there was a commit pushed. Note that .travis.yml delegates to the scripts defined in .travis/cirp that futher delegate to python scripts that are part of ci-release-publisher module. Lastly this patch adds useful python-based script github_releases_util.py aimed to help list daily releases and artifacts, push, downloads and delete artifacts. Completes #1044 Signed-off-by: Waldemar Kozaczuk <[email protected]> --- diff --git a/.travis.yml b/.travis.yml --- a/.travis.yml +++ b/.travis.yml @@ -2,8 +2,47 @@ sudo: required language: c services: - docker +branches: + only: + - master + - ipv6 before_install: - - cd docker && docker build -t osv/builder -f ./Dockerfile.builder-ubuntu . + - pushd docker && docker build -t osv/builder -f ./Dockerfile.builder --build-arg DIST="ubuntu-19.10" . && popd - docker run -it --privileged -d --name build osv/builder -script: - - docker exec build ./scripts/build image=tests +stages: + - build + - publish +env: + global: + - CIRP_GITHUB_REPO_SLUG="osvunikernel/osv-nightly-releases" +jobs: + include: + - stage: build + script: + - docker exec build ./scripts/build-capstan-mpm-packages kernel + - docker exec build ./scripts/build-capstan-mpm-packages unit_tests + - docker exec build ./scripts/osv-version.sh > /tmp/osv-version + - export ARTIFACTS_DIR="$(mktemp -d)" + - cp /tmp/osv-version "$ARTIFACTS_DIR" + - docker cp build:/root/.capstan/repository/osv-loader/osv-loader.qemu "$ARTIFACTS_DIR" + - gzip "$ARTIFACTS_DIR"/osv-loader.qemu + - docker cp build:/root/.capstan/repository/osv-loader/index.yaml "$ARTIFACTS_DIR" + - docker cp build:/root/.capstan/packages/osv.bootstrap.mpm "$ARTIFACTS_DIR" + - docker cp build:/root/.capstan/packages/osv.bootstrap.yaml "$ARTIFACTS_DIR" + - docker cp build:/root/.capstan/packages/osv.unit-tests.mpm "$ARTIFACTS_DIR" + - docker cp build:/root/.capstan/packages/osv.unit-tests.yaml "$ARTIFACTS_DIR" + - docker cp build:/git-repos/osv/modules/tests/usr.manifest "$ARTIFACTS_DIR"/unit_tests.manifest + - ./.travis/cirp/cleanup1.sh + - ./.travis/cirp/store.sh "$ARTIFACTS_DIR" + - ./.travis/cirp/cleanup2.sh + - stage: publish + script: + - docker exec build ./scripts/osv-version.sh > /tmp/osv-version + - export ARTIFACTS_DIR="$(mktemp -d)" + - ./.travis/cirp/collect.sh "$ARTIFACTS_DIR" + - ./.travis/cirp/cleanup4.sh + - ./.travis/cirp/publish.sh "$ARTIFACTS_DIR" $(cat /tmp/osv-version) + - ./.travis/cirp/cleanup5.sh + cache: + directories: + - /opt/cirp diff --git a/.travis/cirp/check_precondition.sh b/.travis/cirp/check_precondition.sh --- a/.travis/cirp/check_precondition.sh +++ b/.travis/cirp/check_precondition.sh @@ -0,0 +1,37 @@ +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +# Put here checks that exit with 0 for the cases when you don't want the CI +# Release Publisher to run. + +# Don't add exiting on "$TRAVIS_TEST_RESULT" != "0" in here as some of the +# scripts have to run even if the job has failed. The scripts that should not +# run on a job failing already include such a check and exit on their own. + +# We don't want to run CI Release Publisher in PRs. +# Travis-CI secure environment variables are not available in PRs, so running +# CI running CI Release Publisher in PRs would just error due to +# [CIRP_]GITHUB_ACCESS_TOKEN not being set. +if [ ! -z "$TRAVIS_PULL_REQUEST" ] && [ "$TRAVIS_PULL_REQUEST" != "false" ]; then + echo "Skipping publishing in a Pull Request" + exit 0 +fi diff --git a/.travis/cirp/cleanup1.sh b/.travis/cirp/cleanup1.sh --- a/.travis/cirp/cleanup1.sh +++ b/.travis/cirp/cleanup1.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh +. .travis/cirp/install.sh + +ci-release-publisher cleanup_publish +ci-release-publisher cleanup_store --scope current-job previous-finished-builds \ + --release complete incomplete diff --git a/.travis/cirp/cleanup2.sh b/.travis/cirp/cleanup2.sh --- a/.travis/cirp/cleanup2.sh +++ b/.travis/cirp/cleanup2.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh +. .travis/cirp/install.sh + +ci-release-publisher cleanup_store --scope current-build \ + --release complete \ + --on-nonallowed-failure diff --git a/.travis/cirp/cleanup3.sh b/.travis/cirp/cleanup3.sh --- a/.travis/cirp/cleanup3.sh +++ b/.travis/cirp/cleanup3.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh +. .travis/cirp/install.sh + +ci-release-publisher cleanup_publish +ci-release-publisher cleanup_store --scope current-build previous-finished-builds \ + --release complete incomplete \ + --on-nonallowed-failure diff --git a/.travis/cirp/cleanup4.sh b/.travis/cirp/cleanup4.sh --- a/.travis/cirp/cleanup4.sh +++ b/.travis/cirp/cleanup4.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh +. .travis/cirp/install.sh + +ci-release-publisher cleanup_publish +ci-release-publisher cleanup_store --scope current-build previous-finished-builds \ + --release complete incomplete diff --git a/.travis/cirp/cleanup5.sh b/.travis/cirp/cleanup5.sh --- a/.travis/cirp/cleanup5.sh +++ b/.travis/cirp/cleanup5.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh + +if [ -z "$TRAVIS_TEST_RESULT" ] && [ "$TRAVIS_TEST_RESULT" == "0" ]; then + echo "Build has not failed, skipping cleanup" + exit 0 +fi + +. .travis/cirp/install.sh + +ci-release-publisher cleanup_publish +ci-release-publisher cleanup_store --scope current-build previous-finished-builds \ + --release complete incomplete diff --git a/.travis/cirp/collect.sh b/.travis/cirp/collect.sh --- a/.travis/cirp/collect.sh +++ b/.travis/cirp/collect.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh + +if [ ! -z "$TRAVIS_TEST_RESULT" ] && [ "$TRAVIS_TEST_RESULT" != "0" ]; then + echo "Build has failed, skipping publishing" + exit 0 +fi + +if [ "$#" != "1" ]; then + echo "Error: No arguments provided. Please specify a directory where to download artifacts to as the first argument." + exit 1 +fi + +ARTIFACTS_DIR="$1" + +. .travis/cirp/install.sh + +ci-release-publisher collect "$ARTIFACTS_DIR" diff --git a/.travis/cirp/install.sh b/.travis/cirp/install.sh --- a/.travis/cirp/install.sh +++ b/.travis/cirp/install.sh @@ -0,0 +1,97 @@ +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. +# Install verifying the hash + +# Verifying PGP signature on CI is error-prone, keyservers often fail to return +# the key and even if they do, `gpg --verify` returns success with a revoked +# or expired key. Thus it's probably better to verify the signature yourself, +# on your local machine, and then rely on the hash on the CI. + +# Set the variables below for the version of ci_release_publisher you would like +# to use. The set values are provided as an example and are likely very out of +# date. + +#VERSION="0.1.0rc1" +#FILENAME="ci_release_publisher-$VERSION-py3-none-any.whl" +#HASH="5a7f0ad6ccfb6017974db42fb1ecfe8b3f9cc1c16ac68107a94979252baa16e3" + +# Get Python >=3.5 +if [ "$TRAVIS_OS_NAME" == "osx" ]; then + brew update + + # Upgrade Python 2 to Python 3 + brew upgrade python || true + + # Print python versions + python --version || true + python3 --version || true + pyenv versions || true + + cd . + cd "$(mktemp -d)" + virtualenv env -p python3 + set +u + source env/bin/activate + set -u + cd - + + # make sha256sum available + export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH" +elif [ "$TRAVIS_OS_NAME" == "linux" ]; then + # Print python versions + python --version || true + python3 --version || true + pyenv versions || true + + # Install Python >=3.5 that has a non-zero patch version + # (we assume the zero patch versions to be potentially buggier than desired) + pyenv global $(pyenv versions | grep -o ' 3\.[5-99]\.[1-99]' | tail -n1) +fi + +pip install --upgrade pip + +check_sha256() +{ + if ! ( echo "$1 $2" | sha256sum -c --status - ); then + echo "Error: sha256 of $2 doesn't match the known one." + echo "Expected: $1 $2" + echo -n "Got: " + sha256sum "$2" + exit 1 + else + echo "sha256 matches the expected one: $1" + fi +} + +# Don't install again if already installed. +# OSX keeps re-installing it tough, as it uses a temp per-script virtualenv. +if ! pip list --format=columns | grep '^ci-release-publisher '; then + cd . + cd "$(mktemp -d)" + VERSION="0.2.0" + FILENAME="ci_release_publisher-$VERSION-py3-none-any.whl" + HASH="da7f139e90c57fb64ed2eb83c883ad6434d7c0598c843f7be7b572377bed4bc4" + pip download ci_release_publisher==$VERSION + check_sha256 "$HASH" "$FILENAME" + pip install --no-index --find-links "$PWD" "$FILENAME" + cd - +fi diff --git a/.travis/cirp/publish.sh b/.travis/cirp/publish.sh --- a/.travis/cirp/publish.sh +++ b/.travis/cirp/publish.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh + +if [ ! -z "$TRAVIS_TEST_RESULT" ] && [ "$TRAVIS_TEST_RESULT" != "0" ]; then + echo "Build has failed, skipping publishing" + exit 0 +fi + +if [ "$#" != "2" ]; then + echo "Error: No arguments provided. Please specify a directory containing artifacts as the first argument and the release name." + exit 1 +fi + +ARTIFACTS_DIR="$1" +OSV_VERSION="$2" + +. .travis/cirp/install.sh + +ci-release-publisher publish --latest-release \ + --latest-release-prerelease \ + --latest-release-check-event-type push \ + --numbered-release \ + --numbered-release-keep-count 10 \ + --numbered-release-prerelease \ + --numbered-release-name "$OSV_VERSION" \ + "$ARTIFACTS_DIR" diff --git a/.travis/cirp/store.sh b/.travis/cirp/store.sh --- a/.travis/cirp/store.sh +++ b/.travis/cirp/store.sh @@ -0,0 +1,42 @@ +#!/usr/bin/env bash +# MIT License + +# Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +set -euo pipefail + +. .travis/cirp/check_precondition.sh + +if [ ! -z "$TRAVIS_TEST_RESULT" ] && [ "$TRAVIS_TEST_RESULT" != "0" ]; then + echo "Build has failed, skipping publishing" + exit 0 +fi + +if [ "$#" != "1" ]; then + echo "Error: No arguments provided. Please specify a directory containing artifacts as the first argument." + exit 1 +fi + +ARTIFACTS_DIR="$1" + +. .travis/cirp/install.sh + +ci-release-publisher store "$ARTIFACTS_DIR" diff --git a/docker/Dockerfile.builder b/docker/Dockerfile.builder --- a/docker/Dockerfile.builder +++ b/docker/Dockerfile.builder @@ -9,7 +9,7 @@ # OSv kernel as well as many applications ... # ARG DIST="fedora-29" -FROM osv/builder-${DIST}-base +FROM osvunikernel/osv-${DIST}-builder-base # # PREPARE ENVIRONMENT diff --git a/docker/Dockerfile.builder-fedora b/docker/Dockerfile.builder-fedora --- a/docker/Dockerfile.builder-fedora +++ b/docker/Dockerfile.builder-fedora @@ -1,55 +0,0 @@ -# -# Copyright (C) 2017 XLAB, Ltd. -# Copyright (C) 2018 Waldemar Kozaczuk -# -# This work is open source software, licensed under the terms of the -# BSD license as described in the LICENSE file in the top-level directory. -# -# This Docker file defines a container intended to build, test and publish -# OSv kernel as well as many applications ... -# -ARG DIST_VERSION=29 -FROM fedora:${DIST_VERSION} - -RUN yum install -y git python3 file which - -# -# PREPARE ENVIRONMENT -# - -# - prepare directories -RUN mkdir /git-repos /result - -# - clone OSv -WORKDIR /git-repos -ARG GIT_ORG_OR_USER=cloudius-systems -RUN git clone https://github.com/${GIT_ORG_OR_USER}/osv.git -WORKDIR /git-repos/osv -RUN git submodule update --init --recursive - -# - install all required packages -RUN scripts/setup.py - -# - install Capstan -RUN wget https://github.com/cloudius-systems/capstan/releases/latest/download/capstan -O /usr/local/bin/capstan -RUN chmod u+x /usr/local/bin/capstan - -CMD /bin/bash - -# -# NOTES -# -# Build the container example: -# docker build -t osv/builder-fedora -f Dockerfile.builder-fedora . -# -# Build the container based of specific Fedora version and git repo owner (if forked) example: -# docker build -t osv/builder-fedora -f Dockerfile.builder-fedora --build-arg DIST_VERSION=31 --build-arg GIT_ORG_OR_USER=a_user . -# -# Run the container FIRST time example: -# docker run -it --privileged osv/builder-fedora -# -# To restart: -# docker restart ID (from docker ps -a) && docker attach ID -# -# To open in another console -# docker exec -it ID /bin/bash diff --git a/docker/Dockerfile.builder-ubuntu b/docker/Dockerfile.builder-ubuntu --- a/docker/Dockerfile.builder-ubuntu +++ b/docker/Dockerfile.builder-ubuntu @@ -1,63 +0,0 @@ -# -# Copyright (C) 2017 XLAB, Ltd. -# Copyright (C) 2018 Waldemar Kozaczuk -# -# This work is open source software, licensed under the terms of the -# BSD license as described in the LICENSE file in the top-level directory. -# -# This Docker file defines a container intended to build, test and publish -# OSv kernel as well as many applications ... -# -ARG DIST_VERSION=19.04 -FROM ubuntu:${DIST_VERSION} - -ENV DEBIAN_FRONTEND noninteractive -ENV TERM=linux - -COPY ./etc/keyboard /etc/default/keyboard -COPY ./etc/console-setup /etc/default/console-setup - -RUN apt-get update -y && apt-get install -y git python3 - -# -# PREPARE ENVIRONMENT -# - -# - prepare directories -RUN mkdir /git-repos /result - -# - clone OSv -WORKDIR /git-repos -ARG GIT_ORG_OR_USER=cloudius-systems -RUN git clone https://github.com/${GIT_ORG_OR_USER}/osv.git -WORKDIR /git-repos/osv -RUN git submodule update --init --recursive - -# - install all required packages -RUN scripts/setup.py - -RUN update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java - -# - install Capstan -RUN wget https://github.com/cloudius-systems/capstan/releases/latest/download/capstan -O /usr/local/bin/capstan -RUN chmod u+x /usr/local/bin/capstan - -CMD /bin/bash - -# -# NOTES -# -# Build the container example: -# docker build -t osv/builder-ubuntu -f Dockerfile.builder-ubuntu . -# -# Build the container based of specific Ubuntu version and git repo owner (if forked) example: -# docker build -t osv/builder-ubuntu -f Dockerfile.builder-ubuntu --build-arg DIST_VERSION=19.10 --build-arg GIT_ORG_OR_USER=a_user . -# -# Run the container FIRST time example: -# docker run -it --privileged osv/builder-ubuntu -# -# To restart: -# docker restart ID (from docker ps -a) && docker attach ID -# -# To open in another console: -# docker exec -it ID /bin/bash diff --git a/licenses/cirp.txt b/licenses/cirp.txt --- a/licenses/cirp.txt +++ b/licenses/cirp.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2018-2020 by Maxim Biro <[email protected]> + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/scripts/github_releases_util.py b/scripts/github_releases_util.py --- a/scripts/github_releases_util.py +++ b/scripts/github_releases_util.py @@ -0,0 +1,219 @@ +#!/usr/bin/python3 + +# This script provides a thin layer around PyGithub library +# to allow easily list releases and list/upload/download artifacts published +# as part of the "nightly" OSv release process automated by Travis. +# Please note that most of commands are quite generic and can be used +# against any Github repo but some of them have been customized to the specifics +# of the 'osvunikernel/osv-nightly-releases' repo. + +import logging +import os, sys +import argparse +import importlib.util + +try: + from github import Github + from github.GithubException import UnknownObjectException,GithubException +except ImportError: + print('PyGithub module is not installed') + print('Run "pip install PyGithub"') + sys.exit(1) + +OSV_SOURCE_REPO_NAME = 'cloudius-systems/osv' +OSV_NIGHTLY_RELEASES_REPO_OWNER = 'osvunikernel' +OSV_NIGHTLY_RELEASES_REPO_NAME = 'osv-nightly-releases' + +# Various GitHub helpers +def client(github_token): + return Github(login_or_token=github_token, per_page=100) + +def get_repo(args): + gh = client(api_token) + repo_full_name = "%s/%s" %(args.owner, args.repo) + try: + return gh.get_repo(repo_full_name) + except UnknownObjectException: + print("Could not find the repo [%s]!" % repo_full_name) + return None + +def get_osv_repo(): + gh = client(api_token) + return gh.get_repo(OSV_SOURCE_REPO_NAME) + +def get_commit(repo, sha): + try: + commit = repo.get_commit(sha) + return commit.commit + except UnknownObjectException: + print("Could not find the commit [%s]!" % sha) + return None + +def list_releases(args): + def get_osv_commit_message(osv_repo, release_title): + message = '' + import re + commit_sha = re.search('-g([a-f0-9]*)', release.title) + if commit_sha: + commit = get_commit(osv_repo, commit_sha.group(1)) + if commit: + message = commit.message.split('\n')[0] + return message + + repository = get_repo(args) + if repository: + if args.message: + osv_repo = get_osv_repo() + print('%-18s %-30s %-20s %-30s' % ( + "NAME", "TITLE", "PUBLISHED_AT", "MESSAGE" + )) + else: + print('%-18s %-60s %-20s' % ( + "NAME", "TITLE", "PUBLISHED_AT" + )) + all_releases = repository.get_releases() + latest_releases = (r for r in all_releases if r.tag_name.endswith('latest')) + non_latest_releases = (r for r in all_releases if not r.tag_name.endswith('latest')) + from itertools import chain + for release in chain(latest_releases, non_latest_releases): + if args.message: + message = get_osv_commit_message(osv_repo, release.title) + print('%-18s %-30.30s %-20s %-30.30s' % (release.tag_name, release.title, str(release.published_at), message)) + else: + print('%-18s %-60.60s %-20s' % ( + release.tag_name, release.title, str(release.published_at))) + +def get_release(args): + repository = get_repo(args) + if repository: + try: + return repository.get_release(args.release) + except UnknownObjectException: + print("Could not find the release [%s]!" % args.release) + return None + else: + return None + +def list_artifacts(args): + release = get_release(args) + if release: + assets = release.get_assets() + if assets.totalCount > 0: + download_base = assets[0].browser_download_url.split('/download')[0] + print('%-40s %-10s %-50s' % ( + "NAME", "SIZE", "DOWNLOAD_URL (base=" + download_base + "/download)" + )) + for asset in assets: + download_url_suffix = asset.browser_download_url.split('/download')[1] + print('%-40s %-10s %-50s' % ( + asset.name, asset.size, download_url_suffix)) + else: + print('Could not find any artifacts!') + +def upload_artifacts(args): + def upload_artifact(release, artifact_path): + if os.path.isfile(artifact_path): + try: + release.upload_asset(artifact_path) + print('Uploaded "%s" (%d bytes) artifact in the release.' % (artifact_path, os.path.getsize(artifact_path))) + except GithubException: + print('Failed to upload the artifact "%s"' % artifact_path) + + release = get_release(args) + if release: + print('Uploading artifacts to "%s" release.' % release.tag_name) + if os.path.isdir(args.path): + dir_path = args.path + artifacts = sorted(os.listdir(dir_path)) + print('Found %d artifact(s) in "%s" directory.' % (len(artifacts), dir_path)) + for artifact in artifacts: + artifact_path = os.path.join(dir_path, artifact) + upload_artifact(release, artifact_path) + else: + upload_artifact(release, args.path) + + +def delete_artifacts(args): + release = get_release(args) + if release: + found_assets = 0 + for asset in release.get_assets(): + import re + if re.match(args.name, asset.name): + found_assets = found_assets + 1 + answer = input("Would you like to delete artifact [%s] from the release: %s of %s? [y|n]" % + (asset.name, args.release, "%s/%s" %(args.owner, args.repo))) + if answer.capitalize() == 'Y': + asset.delete_asset() + print("Deleted artifact [%s] from the release: %s of %s!" % + (asset.name, args.release, "%s/%s" %(args.owner, args.repo))) + if found_assets == 0: + print('Failed to find any artifacts matching [%s]' % args.name) + +def download_artifacts(args): + def download_artifact(download_url, name, directory): + import subprocess, sys + print('... Downloading the artifact [%s] from %s.' % (name, download_url)) + ret = subprocess.call(['wget', '-nv', download_url, '-O', directory + '/' + name]) + if ret != 0: + print('Failed to download %s!' % download_url) + + import os + if args.directory and not os.path.exists(args.directory): + print('Destination directory "%s" does not exist!' % args.directory) + return + + release = get_release(args) + if release: + found_assets = 0 + for asset in release.get_assets(): + if args.name: + import re + if re.match(args.name, asset.name): + found_assets = found_assets + 1 + download_artifact(asset.browser_download_url, asset.name, args.directory) + else: + download_artifact(asset.browser_download_url, asset.name, args.directory) + if args.name and found_assets == 0: + print('Failed to find any artifacts matching [%s]' % args.name) + +api_token = os.environ.get('GH_API_TOKEN') + +if __name__ == "__main__": + if not api_token: + print('Missing Github API token! Set GH_API_TOKEN') + sys.exit(1) + + parser = argparse.ArgumentParser(description="github util") + parser.add_argument('-o', '--owner', action="store", default=OSV_NIGHTLY_RELEASES_REPO_OWNER) + parser.add_argument('-r', '--repo', action="store", default=OSV_NIGHTLY_RELEASES_REPO_NAME) + + subparsers = parser.add_subparsers(dest='cmd', help='Command') + subparsers.required = True + + cmd_list_releases = subparsers.add_parser("list-releases", help="list releases") + cmd_list_releases.add_argument('-m', "--message", action="store_true") + cmd_list_releases.set_defaults(func=list_releases) + + cmd_list_artifacts = subparsers.add_parser("list-artifacts", help="list release artifacts") + cmd_list_artifacts.add_argument('-r', '--release', action="store", default='ci-master-latest') + cmd_list_artifacts.set_defaults(func=list_artifacts) + + cmd_upload_artifacts = subparsers.add_parser("upload-artifacts", help="upload release artifact/s") + cmd_upload_artifacts.add_argument('-r', '--release', action="store", default='ci-master-latest') + cmd_upload_artifacts.add_argument("path", action="store") + cmd_upload_artifacts.set_defaults(func=upload_artifacts) + + cmd_delete_artifacts = subparsers.add_parser("delete-artifacts", help="delete release artifact") + cmd_delete_artifacts.add_argument('-r', '--release', action="store", default='ci-master-latest') + cmd_delete_artifacts.add_argument("name", action="store") + cmd_delete_artifacts.set_defaults(func=delete_artifacts) + + cmd_download_artifacts = subparsers.add_parser("download-artifacts", help="download release artifact") + cmd_download_artifacts.add_argument('-r', '--release', action="store", default='ci-master-latest') + cmd_download_artifacts.add_argument('-n', '--name', action="store") + cmd_download_artifacts.add_argument('-d', '--directory', action="store", default='.', help='Destination directory') + cmd_download_artifacts.set_defaults(func=download_artifacts) + + args = parser.parse_args() + args.func(args) -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/000000000000eceda205a064a436%40google.com.
