This is an automated email from the ASF dual-hosted git repository. msciabarra pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/openserverless-cli.git
commit b28066758c47de0772ea7d58ed75b7d1fbddc65a Author: Michele Sciabarra <[email protected]> AuthorDate: Tue Jul 30 16:39:45 2024 +0200 fixing prereq tests --- .gitignore | 1 + README.md | 66 ++++++---------- Taskfile.yml | 6 +- prereq.go | 4 +- tests/prereq.bats | 35 ++++++--- tests/prereq/olaris/prereq.yml | 134 ++++++++++++++++++++++++++++----- tests/prereq_yml.bats | 65 ++++++++++++++++ tests/test_helper/fail_then_succeed.sh | 11 --- version.txt | 2 +- 9 files changed, 236 insertions(+), 88 deletions(-) diff --git a/.gitignore b/.gitignore index 56e600b..8214ce6 100644 --- a/.gitignore +++ b/.gitignore @@ -49,3 +49,4 @@ _patch/gosh _patch/wsk _patch/openwhisk-wskdeploy _patch/openwhisk-cli +tests/prereq/olaris/bin diff --git a/README.md b/README.md index fd66cac..7070596 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,6 @@ ~ --> - **WARNING: this is still work in progress** The code may not build, there can be errors, and it can even destroy your hard disk or send you in another dimension. @@ -29,7 +28,6 @@ Documentation is also work in progress.... `ops` is the OpenServerless all-mighty CLI tool. - It is basically the [task](https://taskfile.dev) tool enhanced to support: - a bunch of embedded commands (check tools) including `wsk` @@ -49,11 +47,6 @@ Note that to avoid an egg and chicken problem, `ops` itself is built with his an The following environment variables allows to ovverride certain defaults. -- `DEBUG` if set enable debugging messages -- `TRACE` when set gives mode detailes tracing informations, also enable DEBUG=1 -- `EXTRA` appends extra arguments to the task invocation - useful when you need to set extra variables with a nuvopts active. -- `OS` overrides the value of the detected operating system - useful to test prereq scripts -- `ARCH` overrides the value of the detected architecture - useful to test prereq scripts - `OPS_HOME` is the home dir, defaults to `~/.nuv` if not defined - `NUV_ROOT` is the folder where `ops` looks for its tasks. If not defined, if will first look in current directory for an `olaris` folder otherwise download it from githut fron the `NUV_REPO` with a git clone - `OPS_BIN` is the folder where `ops` looks for binaries (external command line tools). If not defined, it defaults to `~/.nuv/<os>-<arch>/bin`. All the prerequisites are downloaded in this directory @@ -72,38 +65,21 @@ The following environment variables allows to ovverride certain defaults. - `NUV_NO_NUVOPTS` can be defined to disable nuvopts parsing. Useful to test hidden tasks. When this is enabled it also shows all the tasks instead of just those with a description. - `OPS_NO_PREREQ` disable downloading of prerequisites - you have to ensure at least coreutils is in the path to make things work +## Special variables used for tests and debugging ## Where `nuv` looks for binaries -- `NUV_USE_COREUTILS` enables the use of [coreutils](https://github.com/uutils/coreutils) instead of the current unix tools implementation. They will eventually replace the current tools. - -Nuv requires some binary command line tools to work with ("bins"). - -They are expected to be in the folder pointed by the environment variable `NUV_BIN`. - -If this environment variable is not defined, it defaults to the same folder where `nuv` itself is located. The `NUV_BIN` folder is then added to the beginning of the `PATH` before executing anything else. - -Nuv is normally distributed with an installer that includes all the tools for the various operating systems (linux, windows, osx). - -**NOTE**: You can download the relevant tools when you run from source code executing `task install`. This task will download the command line tools and setup a link in `/usr/local/bin` to invoke `nuv`. - -## Internal environment variables - -`OPS_RUNTIMES_JSON` is used for the values of the runtimes json if the system is unable to read in other ways the current runtimes json. It is normally compiled in when you buid from the current version of the runtimes.json. It can be overriden - -`OPS_COREUTILS` is a string, a space separated list, which lists all the commands the coreutils binary provided. It should be kept updated with the values of the current version used. It can be overriden - -`OPS_TOOLS` is a string, a space separated list, which lists all the commands provided as internal tool by the ops binary. It shold be kept updated with the current list of tools provided. It can be overriden defining externally +Nuv requires some binary command line tools to work with. Those binaries are automatically downloaded for the supported platforms. ## Where `nuv` looks for tasks -Nuv is an enhanced task runner. Tasks are described by [task](https://taskfile.dev) taskfiles. +Ops is an enhanced task runner that execute **taskfiles**. How tasks are written is described in [task](https://taskfile.dev) documentation. Nuv is able either to run existing tasks or download them from github. -When you run `nuv [<args>...]` it will first look for its `nuv` root. +When you run `ops [<args>...]` it will first look for its `ops` root. -The `nuv` root is a folder with two files in it: `nuvfile.yml` (a yaml taskfile) and `nuvroot.json` (a json file with release information). +The `ops` root is a folder with two files in it: `nuvfile.yml` (a yaml taskfile) and `nuvroot.json` (a json file with release information). The first step is to locate the root folder. The algorithm to find the tools is the following. @@ -115,27 +91,25 @@ If there is not a `nuvfile.yml` it will look for a folder called `olaris` with b Then it will look in `~/.nuv` if there is an `olaris` folder with `nuvfile.yml` and `nuvroot.json`. -Finally it will look in the `NUV_BIN` folder if there is an `olaris` folder with `nuvfile.yml` and `nuvroot.json` - If everything fails, it will ask you to download some tasks with the command `nuv -update`. In this case it will download the latest version. -## Where `nuv` download tasks from GitHub +## Where `ops` download tasks from GitHub -Download tasks from GitHub is triggered by the `nuv -update` command. +Download tasks from GitHub is triggered by the `ops -update` command. -The repo to use is defined by the environment variable `NUV_REPO`, and defaults if it is missing to `https://github.com/nuvolaris/olaris` +The repo to use is defined by the environment variable `NUV_REPO`, and defaults if it is missing to `https://github.com/apache/openserverless-task` The branch to use is defined at build time. It can be overriden with the enviroment variable `NUV_BRANCH`. -When you run `nuv -update`, if there is not a `~/.nuv/olaris` it will clone the current branch, otherwise it will update it. +When you run `ops -update`, if there is not a `~/.nuv/<branch>/olaris` it will clone the current branch, otherwise it will update it. -## How `nuv` execute tasks +## How `ops` execute tasks -It will then look to the command line parameters `nuv <arg1> <arg2> <arg3>` and will consider them directory names. The list can be empty. +It will then look to the command line parameters `ops <arg1> <arg2> <arg3>` and will consider them directory names. The list can be empty. If there is a directory name `<arg1>` it will change to that directory. If there is then a subdirectory `<arg2>` it will change to that and so on until it finds a argument that is not a directory name. -If the last argument is a directory name, will look for a `nuvopts.txt`. If it is there, it will show this. If it's not there, it will execute a `task -t nuvfile.yml -l` showing tasks with description. +If the last argument is a directory name, will look for a `nuvopts.txt`. If it is there, it will show this. If it's not there, it will execute a `ops -t nuvfile.yml -l` showing tasks with description. If it finds an argument not corresponding to a directory, it will consider it a task to execute, @@ -145,11 +119,11 @@ If there is a `nuvopts.txt`, it will interpret it as a [`docopt`](http://docopt ### Example -A command like `nuv setup kubernetes install --context=k3s` will look in the folder `setup/kubernetes` in the `nuv root` if it is there, select `install` as task to execute and parse the `--context=k3s`. It is equivalent to invoke `cd setup/kubernetes ; task install -- context=k3s`. +A command like `ops setup kubernetes install --context=k3s` will look in the folder `setup/kubernetes` in the ops root, if it is there, then select `install` as task to execute and parse the `--context=k3s`. It is equivalent to invoke `cd setup/kubernetes ; task install -- context=k3s`. If there is a `nuvopts.txt` with a `command <name> --flag --fl=<val>` the passed parameters will be: `_name_=<name> __flag=true __fl=true _val_=<val>`. -Note that also this will also use the downloaded tools and the embedded commands of `nuv`. +Note that also this will also use the downloaded tools and the embedded commands of `ops`. ## Saving state @@ -170,7 +144,7 @@ at the end of `task` invocation there will be `_SERVER=myserver` `_USER=myuser` ## Embedded tools -Currently task embeds the following tools, and you can invoke them directly prefixing them with `-`: (`nuv -task`, `nuv -basename` etc). Use `nuv -help` to list them. +Currently task embeds the following tools, and you can invoke them directly prefixing them with `-`: (`ops -task`, `nuv -basename` etc). Use `nuv -help` to list them. - [task](https://taskfile.dev) the Task build tool - [wsk](https://github.com/apache/openwhisk-cli) the OpenWhisk cli @@ -200,8 +174,18 @@ Basic unix like tools (`nuv -<tool> -help for details`): - which - zip +## Internal environment variables +Those variables are special purpose and used for test and debug +- `DEBUG` if set enable debugging messages +- `TRACE` when set gives mode detailes tracing informations, also enable DEBUG=1 +- `EXTRA` appends extra arguments to the task invocation - useful when you need to set extra variables with a nuvopts active. +- `__OS` overrides the value of the detected operating system - useful to test prereq scripts +- `__ARCH` overrides the value of the detected architecture - useful to test prereq scripts +- `OPS_RUNTIMES_JSON` is used for the values of the runtimes json if the system is unable to read in other ways the current runtimes json. It is normally compiled in when you buid from the current version of the runtimes.json. It can be overriden +- `OPS_COREUTILS` is a string, a space separated list, which lists all the commands the coreutils binary provided. It should be kept updated with the values of the current version used. It can be overriden +- `OPS_TOOLS` is a string, a space separated list, which lists all the commands provided as internal tool by the ops binary. It shold be kept updated with the current list of tools provided. It can be overriden defining externally diff --git a/Taskfile.yml b/Taskfile.yml index 37e03c7..3ffa35f 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -100,15 +100,15 @@ tasks: - test -d bats itest: - desc: integration tests for nuv (T=<test-name> without .bats, empty all) + desc: integration tests for nuv (T=<test-name> without .bats, empty run all, pass flags with F=) dir: tests cmds: - task: bats - rm -Rf ~/.nuv/olaris - | if test -z "{{.T}}" - then bats/bin/bats . - else bats/bin/bats {{.T}}.bats + then bats/bin/bats {{.F}} . + else bats/bin/bats {{.F}} {{.T}}.bats fi clean: diff --git a/prereq.go b/prereq.go index ae9ce12..2f24ff1 100644 --- a/prereq.go +++ b/prereq.go @@ -76,11 +76,11 @@ func loadPrereq(dir string) (Prereq, error) { // otherwise setup one in ~/nuv/<os>-<arch>/bin // and sets OPS_BIN func EnsureBindir() (string, error) { - os_ := os.Getenv("OS") + os_ := os.Getenv("__OS") if os_ == "" { os_ = runtime.GOOS } - arch := os.Getenv("ARCH") + arch := os.Getenv("__ARCH") if arch == "" { arch = runtime.GOARCH } diff --git a/tests/prereq.bats b/tests/prereq.bats index 6725b3e..b12dd5b 100644 --- a/tests/prereq.bats +++ b/tests/prereq.bats @@ -20,16 +20,38 @@ setup() { load 'test_helper/bats-assert/load' export NO_COLOR=1 export NUV_BRANCH="0.1.0-testing" + rm -Rvf ~/.nuv/ cd prereq } +@test "ops prereq" { + run ops + assert_line --partial "ensuring prerequisite 7zz" + assert_line --partial "ensuring prerequisite coreutils" + assert_line --partial "ensuring prerequisite bun" + assert_line --partial "ensuring prerequisite kubectl" + assert_line --partial "ensuring prerequisite kind" + assert_line --partial "ensuring prerequisite k3sup" + assert_line --partial "info" + run ops + refute_line "ensuring prerequisite 7zz" + refute_line "ensuring prerequisite coreutils" + refute_line "ensuring prerequisite bun" + refute_line "ensuring prerequisite kubectl" + refute_line "ensuring prerequisite kind" + refute_line "ensuring prerequisite k3sup" + assert_line --partial "info" +} + + @test "download others" { - rm -Rvf ~/.nuv/ + skip "todo" for o in linux darwin do for a in arm64 amd64 do - run env OS=$o ARCH=$a ops - assert_line "ensuring prerequisite 7zz" --partial + #o=linux a=amd64 + run env OS=$o ARCH=$a ops + assert_line --partial "ensuring prerequisite 7zz" assert test -e ~/.nuv/$o-$a/bin/7zz done done @@ -37,10 +59,3 @@ setup() { assert test -e ~/.nuv/windows-*/bin/7zz.exe } -@test "ops prereq" { - rm -Rvf ~/.nuv/ - run ops - assert_line "ensuring prerequisite 7zz" --partial - run ops info - assert_line "7-˜Zip" --partial -} diff --git a/tests/prereq/olaris/prereq.yml b/tests/prereq/olaris/prereq.yml index 22263d6..f56def5 100644 --- a/tests/prereq/olaris/prereq.yml +++ b/tests/prereq/olaris/prereq.yml @@ -1,10 +1,51 @@ version: 3 vars: - OS: "{{or .OS OS}}" - ARCH: "{{or .ARCH ARCH}}" + OS: "{{or .__OS OS}}" + ARCH: "{{or .__ARCH ARCH}}" + ARC: '{{if eq .OS "windows"}}.zip{{else}}.tgz{{end}}' + ARC2: '{{if eq .OS "windows"}}.zip{{else}}.tar.gz{{end}}' + EXE: '{{if eq .OS "windows"}}.exe{{else}}{{end}}' tasks: + test: + vars: + FILE: file.tar.gz + cmds: + - echo "{{.FILE}}" + - echo '{{trimSuffix ".gz" .FILE }}' + + download-and-extract: + requires: + vars: + - FILE + - URL + - OUT + vars: + FILETAR: '{{trimSuffix ".gz" .FILE}}' + cmds: + - echo "{{.OUT}} {{.FILE}} {{.URL}} {{.FILETAR}}" + - curl -sL "{{.URL}}" -o {{.FILE}} + - | + {{if eq .OS "windows"}} + 7zz e -r -aoa -o. "{{.FILE}}" "{{.OUT}}.exe" + rm "{{.FILE}}" + {{else}} + 7zz e "{{.FILE}}" -o. + rm "{{.FILE}}" + 7zz e -r -aoa -o. "{{.FILETAR}}" "{{.OUT}}" + rm "{{.FILETAR}}" + {{end}} + + download: + requires: + vars: + - URL + - OUT + cmds: + - curl -sL -ccookie.txt -o"{{.OUT}}" "{{.URL}}" + - '{{if eq .OS "windows"}}mv {{.OUT}} {{.OUT}}.exe{{else}}chmod +x {{.OUT}}{{end}}' + - rm cookie.txt 7zz: desc: 7zr @@ -30,32 +71,85 @@ tasks: rm "{{.FILE}}" fi + coreutils: + desc: coreutils + vars: + VERSION: "0.0.27" + SUFFIX: + sh: | + case "{{.OS}}-{{.ARCH}}" in + linux-amd64) echo "x86_64-unknown-linux-gnu" ;; + linux-arm64) echo "aarch64-unknown-linux-gnu" ;; + darwin-amd64) echo "x86_64-apple-darwin" ;; + darwin-arm64) echo "aarch64-apple-darwin" ;; + windows-*) echo "x86_64-pc-windows-gnu" ;; + *) echo "unknown" ;; + esac + URL: "https://github.com/uutils/coreutils/releases/download/{{.VERSION}}/coreutils-{{.VERSION}}-{{.SUFFIX}}{{.ARC2}}"; + FILE: "{{base .URL}}" + OUT: coreutils + cmds: + - task: download-and-extract + vars: + URL: "{{.URL}}" + FILE: "{{.FILE}}" + OUT: coreutils bun: description: bun vars: - XVERSION: "v1.11.20" + VERSION: "v1.1.21" + ARCH2: '{{if eq .ARCH "amd64"}}x64{{else}}aarch64{{end}}' + URL: 'https://github.com/oven-sh/bun/releases/download/bun-{{.VERSION}}/bun-{{.OS}}-{{.ARCH2}}.zip' + FILE: '{{base .URL}}' cmds: - - task: _bun + - echo "{{.URL}}" + - curl -sL "{{.URL}}" -o "{{.FILE}}" + - 7zz e -r -aoa -o. {{.FILE}} bun{{.EXE}} + - rm "{{.FILE}}" + kubectl: + vars: + VERSION: v1.30.3 + cmds: + - task: download + vars: + URL: "https://dl.k8s.io/release/{{.VERSION}}/bin/{{.OS}}/{{.ARCH}}/kubectl{{.EXE}}"; + OUT: kubectl - coreutils: - desc: coreutils + kind: vars: - XVERSION: "0.0.27" - SUFFIX: - sh: | - case "{{ARCH}}-{{OS}}" in - amd64-linux) echo "i686-unknown-linux-gnu.tar.gz" ;; - arm64-linux) echo "aarch64-unknown-linux-gnu.tar.gz" ;; - amd64-darwin) echo "x86_64-apple-darwin.tar.gz" ;; - arm64-darwin) echo "aarch64-apple-darwin.tar.gz" ;; - amd64-windows) echo "x86_64-pc-windows-gnu.zip" ;; - *) echo "unknown" ;; - esac - URL: "https://github.com/uutils/coreutils/releases/download/{{.VERSION}}/coreutils-{{.VERSION}}-{{.SUFFIX}}"; + VERSION: v0.17.0 cmds: - - echo "{{.URL}}" - #- curl -sL "{{.URL}}" -o coreutils.tar.gz + - task: download + vars: + URL: "https://kind.sigs.k8s.io/dl/{{.VERSION}}/kind-{{.OS}}-{{.ARCH}}"; + OUT: kind + k3sup: + vars: + VERSION: 0.13.6 + BASE: "https://github.com/alexellis/k3sup/releases/download/{{.VERSION}}"; + FILE: + sh: | + case {{.OS}}-{{.ARCH}} in + (linux-amd64) echo k3sup ;; + (linux-arm64) echo k3sup-arm64 ;; + (darwin-amd64) echo k3sup-darwin ;; + (darwin-arm64) echo k3sup-darwin-arm64 ;; + (windows-amd64) echo k3sup.exe ;; + esac + cmds: + - task: download + vars: + URL: "{{.BASE}}/{{.FILE}}" + OUT: k3sup + all: + - task: 7zz + - task: coreutils + - task: bun + - task: kind + - task: k3sup + # skipping this as it is slow to download + #- task: kubectl diff --git a/tests/prereq_yml.bats b/tests/prereq_yml.bats new file mode 100644 index 0000000..bea3fdf --- /dev/null +++ b/tests/prereq_yml.bats @@ -0,0 +1,65 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +setup() { + load 'test_helper/bats-support/load' + load 'test_helper/bats-assert/load' + export NO_COLOR=1 + export NUV_BRANCH="0.1.0-testing" + rm -Rvf _bin >/dev/null + export COUNT=5 +} + +@test "OS=linux ARCH=amd64" { + OS=linux ARCH=amd64 + mkdir -p _bin/$OS-$ARCH + run task -t prereq/olaris/prereq.yml -d _bin/$OS-$ARCH __OS=$OS __ARCH=$ARCH all + find _bin/$OS-$ARCH -type f | xargs file | grep ELF | grep x86-64 | wc -l | tee _count + assert_equal "$(cat _count)" $COUNT +} + +@test "OS=linux ARCH=arm64" { + OS=linux ARCH=arm64 + mkdir -p _bin/$OS-$ARCH + run task -t prereq/olaris/prereq.yml -d _bin/$OS-$ARCH __OS=$OS __ARCH=$ARCH all + find _bin/$OS-$ARCH | xargs file | grep ELF | grep ARM | wc -l | tee _count + assert_equal "$(cat _count)" $COUNT +} + +@test "OS=darwin ARCH=amd64" { + OS=darwin ARCH=amd64 + mkdir -p _bin/$OS-$ARCH + run task -t prereq/olaris/prereq.yml -d _bin/$OS-$ARCH __OS=$OS __ARCH=$ARCH all + find _bin/$OS-$ARCH | xargs file | grep Mach-O | grep x86_64 | wc -l | tee _count + assert_equal "$(cat _count)" $COUNT +} + +@test "OS=darwin ARCH=arm64" { + OS=darwin ARCH=arm64 + mkdir -p _bin/$OS-$ARCH + run task -t prereq/olaris/prereq.yml -d _bin/$OS-$ARCH __OS=$OS __ARCH=$ARCH all + find _bin/$OS-$ARCH | xargs file | grep Mach-O | grep arm64 | wc -l | tee _count + assert_equal "$(cat _count)" $COUNT +} + +@test "OS=windows ARCH=amd64" { + OS=windows ARCH=amd64 + mkdir -p _bin/$OS-$ARCH + run task -t prereq/olaris/prereq.yml -d _bin/$OS-$ARCH __OS=$OS __ARCH=$ARCH all + find _bin/$OS-$ARCH | xargs file | grep PE32 | wc -l | tee _count + assert_equal "$(cat _count)" $COUNT +} diff --git a/tests/test_helper/fail_then_succeed.sh b/tests/test_helper/fail_then_succeed.sh deleted file mode 100755 index ac8e065..0000000 --- a/tests/test_helper/fail_then_succeed.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -file_path=".test_fail_then_succeed" - -if [ ! -f "$file_path" ]; then - touch "$file_path" - exit 1 -else - rm "$file_path" - exit 0 -fi diff --git a/version.txt b/version.txt index 25d5a22..4e9b7c0 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -0.1.0-testing.2407291424 +0.1.0-testing.2407301637
