This is an automated email from the ASF dual-hosted git repository. msciabarra pushed a commit to tag kind-469ee67 in repository https://gitbox.apache.org/repos/asf/openserverless-testing.git
commit 80f92529fcc5c6838f4c33c210c930e10c19f2c3 Author: miki3421 <[email protected]> AuthorDate: Wed Mar 18 22:08:51 2026 +0100 Align testing workflows with <test>-<hash> tags --- .github/workflows/operator-pr-test.yaml | 21 +- .github/workflows/platform-ci-tests.yaml | 9 +- .github/workflows/task-pr-test.yaml | 21 +- .github/workflows/tests.yaml | 8 +- gap.md | 216 ++++++++++++------- tests/all.sh | 8 + tests/lib/selector.sh | 89 +++++--- tests/run-gh-suite.sh | 60 ++++-- workflow.md | 348 ++++++++++++++++--------------- 9 files changed, 485 insertions(+), 295 deletions(-) diff --git a/.github/workflows/operator-pr-test.yaml b/.github/workflows/operator-pr-test.yaml index 1e82bb4..1dd6552 100644 --- a/.github/workflows/operator-pr-test.yaml +++ b/.github/workflows/operator-pr-test.yaml @@ -16,7 +16,7 @@ # under the License. # name: Operator PR Test -run-name: "Operator PR #${{ github.event.client_payload.pr_number }} on ${{ github.event.client_payload.selector || github.event.client_payload.platform }}" +run-name: "Operator PR #${{ github.event.client_payload.pr_number }} on ${{ github.event.client_payload.test_tag || github.event.client_payload.selector || github.event.client_payload.platform }}" on: repository_dispatch: @@ -27,7 +27,9 @@ env: PR_REF: ${{ github.event.client_payload.pr_ref }} PR_SHA: ${{ github.event.client_payload.pr_sha }} OPERATOR_REPO: ${{ github.event.client_payload.operator_repo }} - SELECTOR: ${{ github.event.client_payload.selector || github.event.client_payload.platform }} + TEST_TAG: ${{ github.event.client_payload.test_tag || github.event.client_payload.selector || github.event.client_payload.platform }} + TEST_NAME: ${{ github.event.client_payload.test_name || '' }} + TEST_HASH: ${{ github.event.client_payload.test_hash || '' }} OPS_BRANCH: main OPERATOR_IMAGE: ghcr.io/${{ github.repository_owner }}/openserverless-testing @@ -55,6 +57,19 @@ jobs: ls _operator/olaris/opsfile.yml _operator/olaris/opsroot.json echo "Submodule OK" + - name: Show requested test context + run: | + echo "PR number: $PR_NUMBER" + echo "PR ref: $PR_REF" + echo "PR sha: $PR_SHA" + echo "Requested test tag: $TEST_TAG" + if [[ -n "$TEST_NAME" ]]; then + echo "Requested test name: $TEST_NAME" + fi + if [[ -n "$TEST_HASH" ]]; then + echo "Requested test hash: $TEST_HASH" + fi + - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 @@ -110,4 +125,4 @@ jobs: APIHOST: op://OpenServerless/TESTING/APIHOST - name: Run GitHub Test Suite - run: tests/run-gh-suite.sh $SELECTOR + run: tests/run-gh-suite.sh "$TEST_TAG" diff --git a/.github/workflows/platform-ci-tests.yaml b/.github/workflows/platform-ci-tests.yaml index 3fba34c..64976c5 100644 --- a/.github/workflows/platform-ci-tests.yaml +++ b/.github/workflows/platform-ci-tests.yaml @@ -15,16 +15,15 @@ # specific language governing permissions and limitations # under the License. # -# Use client_payload.platform attribute to detect the platform being tested. name: On Tasks Testing Request -run-name: Tagging openserverless-testing for platform ${{ github.event.client_payload.platform }} +run-name: Tagging openserverless-testing for ${{ github.event.client_payload.test_tag || format('{0}-{1}', github.event.client_payload.test || github.event.client_payload.platform, github.event.client_payload.hash || github.event.client_payload.tag) }} on: repository_dispatch: types: [ olaris-testing-update ] env: - TAG: ${{ format('{0}-{1}', github.event.client_payload.platform, github.event.client_payload.tag) }} + TEST_TAG: ${{ github.event.client_payload.test_tag || format('{0}-{1}', github.event.client_payload.test || github.event.client_payload.platform, github.event.client_payload.hash || github.event.client_payload.tag) }} jobs: tags-push: @@ -35,6 +34,6 @@ jobs: with: token: ${{ secrets.PAT }} - name: Tag - run: git tag $TAG + run: git tag $TEST_TAG - name: Push Tag - run: git push origin $TAG \ No newline at end of file + run: git push origin $TEST_TAG diff --git a/.github/workflows/task-pr-test.yaml b/.github/workflows/task-pr-test.yaml index a2c2054..ffa247e 100644 --- a/.github/workflows/task-pr-test.yaml +++ b/.github/workflows/task-pr-test.yaml @@ -16,7 +16,7 @@ # under the License. # name: Task PR Test -run-name: "Task PR #${{ github.event.client_payload.pr_number }} on ${{ github.event.client_payload.selector || github.event.client_payload.platform }}" +run-name: "Task PR #${{ github.event.client_payload.pr_number }} on ${{ github.event.client_payload.test_tag || github.event.client_payload.selector || github.event.client_payload.platform }}" on: repository_dispatch: @@ -27,7 +27,9 @@ env: PR_REF: ${{ github.event.client_payload.pr_ref }} PR_SHA: ${{ github.event.client_payload.pr_sha }} TASK_REPO: ${{ github.event.client_payload.task_repo }} - SELECTOR: ${{ github.event.client_payload.selector || github.event.client_payload.platform }} + TEST_TAG: ${{ github.event.client_payload.test_tag || github.event.client_payload.selector || github.event.client_payload.platform }} + TEST_NAME: ${{ github.event.client_payload.test_name || '' }} + TEST_HASH: ${{ github.event.client_payload.test_hash || '' }} OPS_BRANCH: main NGROK_TOKEN: ${{ secrets.NGROK_TOKEN }} @@ -50,6 +52,19 @@ jobs: ls _task/opsfile.yml _task/opsroot.json git -C _task rev-parse HEAD + - name: Show requested test context + run: | + echo "PR number: $PR_NUMBER" + echo "PR ref: $PR_REF" + echo "PR sha: $PR_SHA" + echo "Requested test tag: $TEST_TAG" + if [[ -n "$TEST_NAME" ]]; then + echo "Requested test name: $TEST_NAME" + fi + if [[ -n "$TEST_HASH" ]]; then + echo "Requested test hash: $TEST_HASH" + fi + - name: Set OPS_ROOT run: echo "OPS_ROOT=$(pwd)/_task" >> "$GITHUB_ENV" @@ -75,4 +90,4 @@ jobs: APIHOST: op://OpenServerless/TESTING/APIHOST - name: Run GitHub Test Suite - run: tests/run-gh-suite.sh $SELECTOR + run: tests/run-gh-suite.sh "$TEST_TAG" diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml index 24fb5f4..44604b4 100644 --- a/.github/workflows/tests.yaml +++ b/.github/workflows/tests.yaml @@ -24,7 +24,7 @@ on: - '*-*' env: - PLATFORM: ${{ github.ref_name }} + TEST_TAG: ${{ github.ref_name }} OPS_REPO: ${{ vars.OPS_REPO || 'https://github.com/apache/openserverless-task' }} OPS_BRANCH: main NGROK_TOKEN: ${{ secrets.NGROK_TOKEN }} @@ -70,7 +70,7 @@ jobs: - name: Wait for Debug (if requested) run: .github/waitIfDebug.sh - name: Run GitHub Test Suite - run: tests/run-gh-suite.sh $PLATFORM + run: tests/run-gh-suite.sh "$TEST_TAG" test-windows: runs-on: windows-2022 @@ -135,7 +135,7 @@ jobs: run: .github/waitIfDebug.sh - name: Run GitHub Test Suite shell: bash - run: tests/run-gh-suite.sh $PLATFORM + run: tests/run-gh-suite.sh "$TEST_TAG" test-mac: runs-on: macOS-14 @@ -183,4 +183,4 @@ jobs: - name: Wait for Debug (if requested) run: .github/waitIfDebug.sh - name: Run GitHub Test Suite - run: tests/run-gh-suite.sh $PLATFORM + run: tests/run-gh-suite.sh "$TEST_TAG" diff --git a/gap.md b/gap.md index ae7c35a..8192c7f 100644 --- a/gap.md +++ b/gap.md @@ -1,170 +1,234 @@ -# Gap Analysis: PR-driven testing for `openserverless-operator` and `openserverless-task` +# Gap Analysis: tag-driven PR testing for `openserverless-operator` and `openserverless-task` + +Note: + +- this document captures the gaps identified before the current implementation branch +- the branch `feat/test-tag-hash-contract` addresses the trigger contract shift to `<test>-<hash>` and improves test log visibility +- any remaining gaps below should therefore be read as historical analysis unless they are still explicitly present after merge ## Assumption -Based on the clarification, the trigger contract is a selector shaped as: +Based on the latest clarification, the desired trigger contract is: + +- `<test>-<hash>` -- `<platform>-<architecture>` +Where: -Example: `k3s-amd`, `k3s-arm`. +- `test` already encodes the target platform or runtime shape +- no extra platform or architecture field is needed +- examples of test names are `k3s` and `k3sarm` +- `hash` is the commit hash that generated the test request -The main question is therefore whether a PR in `openserverless-operator` or `openserverless-task` can propagate that selector end-to-end, trigger the right GitHub workflow, and execute on the matching infrastructure. +The main question is therefore whether a PR in `openserverless-operator` or `openserverless-task` can propagate that tag end-to-end, trigger the right GitHub workflow, and execute the corresponding test on the intended infrastructure. ## Current state found ### Operator path -- `openserverless-testing/.github/workflows/operator-pr-test.yaml:21-135` already accepts `repository_dispatch` with PR number, ref, sha, repo and platform. -- `nuvolaris/openserverless-operator/.github/workflows/trigger-testing.yaml:21-90` can dispatch that event, but only from: - - an issue comment `/testing <platform>` - - a manual `workflow_dispatch` +- `openserverless-testing/.github/workflows/operator-pr-test.yaml` accepts `repository_dispatch` with PR number, ref, sha, repo and selector-like data. +- `nuvolaris/openserverless-operator/.github/workflows/trigger-testing.yaml` currently enables testing from PR labels, not from a `<test>-<hash>` tag. ### Task path -- `openserverless-testing/.github/workflows/platform-ci-tests.yaml:22-40` accepts `repository_dispatch` type `olaris-testing-update` and converts the payload into a git tag. -- `openserverless-testing/.github/workflows/tests.yaml:21-97` runs the full test suite on any pushed tag matching `*-*`. -- `nuvolaris/openserverless-task` currently has no `.github/workflows` directory, so there is no producer for that dispatch. +- `openserverless-testing/.github/workflows/platform-ci-tests.yaml` accepts `repository_dispatch` type `olaris-testing-update` and converts the payload into a git tag. +- `openserverless-testing/.github/workflows/tests.yaml` runs the full test suite on any pushed tag matching `*-*`. +- `nuvolaris/openserverless-task` originally had no workflow producer for that dispatch. + +### What was already close to the new specification + +- `platform-ci-tests.yaml` already materializes a tag from two fields: + - a logical test identifier + - a second suffix currently named `tag` +- `tests.yaml` already treats the incoming trigger generically as `*-*` +- the historical task flow therefore was structurally closer to `<test>-<hash>` than the newer label contract based on `kind-amd` or `k3s-arm` ## Gaps -### 1. No end-to-end PR trigger exists for `openserverless-task` +### 1. The documented trigger contract and the implemented one do not match Evidence: -- `platform-ci-tests.yaml` waits for `repository_dispatch` (`olaris-testing-update`), but `nuvolaris/openserverless-task` has no GitHub workflows at all. +- desired contract is now `<test>-<hash>` +- current trigger implementation added in the PR flow is based on labels like `kind-amd`, `k3s-amd`, `k3s-arm` Impact: -- A PR on `openserverless-task` cannot currently trigger anything in this repository. +- the system is not yet aligned with the latest specification +- a PR tag like `k3s-abcdef1` or `k3sarm-abcdef1` is conceptually different from the currently implemented selector labels -### 2. Operator flow is not tag-driven today +### 2. No end-to-end PR trigger existed for `openserverless-task` Evidence: -- `nuvolaris/openserverless-operator/.github/workflows/trigger-testing.yaml:21-57` only parses: - - `workflow_dispatch` inputs - - issue comments starting with `/testing ` +- `platform-ci-tests.yaml` waits for `repository_dispatch` (`olaris-testing-update`) +- `nuvolaris/openserverless-task` originally had no GitHub workflow producing it Impact: -- The current operator flow still requires a comment or manual dispatch instead of a tag-shaped selector propagated automatically from the PR flow. +- a PR on `openserverless-task` could not trigger anything in `openserverless-testing` -### 3. The selector contract is only partially implemented across repos +### 3. The tag contract is only partially implemented across repos Evidence: -- `operator` dispatch passes a single `platform` field, whose value is expected to already contain the combined selector such as `k3s-amd`. -- `platform-ci-tests.yaml:27` builds a tag as `<platform>-<tag>`, but the second field is still generically named `tag`, not `architecture`. -- `tests.yaml:21-24` reacts to any tag matching `*-*`. +- `tests.yaml` reacts generically to any tag matching `*-*` +- `platform-ci-tests.yaml` builds a tag from two payload fields, but still names the first field `platform` +- there is no canonical schema for: + - test name + - commit hash + - source repository + - PR number/ref/sha Impact: -- The system has no canonical representation for: - - which repo is under test - - which platform/infrastructure should be targeted - - which architecture is being targeted - - which PR commit should be tested +- the system still lacks one explicit contract shared by `operator`, `task`, and `testing` +- the current codebase does not consistently treat the trigger as `<test>-<hash>` +- some code paths are semantically aligned already, but their naming is still tied to the older abstraction -### 4. There is no task-specific PR workflow in `openserverless-testing` +### 4. The old task tag transport loses PR identity and is not reproducible Evidence: -- The operator has a dedicated entry point: `operator-pr-test.yaml:21-135`. -- The task path only has the generic tag flow: `platform-ci-tests.yaml` -> `tests.yaml`. +- `platform-ci-tests.yaml` turns the payload into a git tag only +- that tag does not carry `pr_number`, `pr_ref`, `pr_sha`, or source repository as first-class workflow inputs +- `tests.yaml` uses static defaults for `OPS_REPO` and `OPS_BRANCH` Impact: -- `openserverless-task` does not have a PR-aware execution path comparable to the operator one. -- The generic tag flow cannot attach PR metadata to the run in the same way. +- a task-triggered run cannot be pinned reliably to the exact PR commit that requested the test unless the commit identity is explicitly reconstructed from the tag +- reruns may test different code than the originally requested change -### 5. The tag-based transport loses PR identity and is not reproducible +### 5. Tag parsing logic was historically built around split selectors, not `<test>-<hash>` Evidence: -- `platform-ci-tests.yaml:27-40` turns the payload into a git tag only. -- That tag stores no `pr_number`, `pr_ref`, `pr_sha`, or source repository. -- `tests.yaml:27-29` uses static defaults for `OPS_REPO` and `OPS_BRANCH`. +- `tests/1-deploy.sh` historically stripped a trailing suffix from the incoming tag +- many test scripts historically collapsed values at the first `-` +- this made sense for forms like `k3s-amd` or `k3s-arm`, but not for the new contract where the separator is between test name and commit hash Impact: -- A task-triggered run cannot be pinned to the exact PR commit that requested the test. -- Two runs with the same `<platform>-<tag>` collide. -- A rerun may test different code than the one originally requested. +- parsing must be normalized around: + - `test` + - `hash` +- not around: + - platform + - architecture +- this matters both in shell parsing and in the workflow payload field names -### 6. Selector parsing is inconsistent with the `<platform>-<architecture>` contract +### 6. The current implementation still assumes architecture-encoded test names in deploy logic Evidence: -- `tests/1-deploy.sh:18-22` keeps selectors like `k3s-amd` and `k3s-arm`, but only strips a trailing numeric suffix. -- Many later scripts collapse the selector to the first `-` segment, for example: - - `tests/6-login.sh:18-19` - - `tests/7-static.sh:18-19` - - `tests/8-user-redis.sh:18-19` - - same pattern also appears in `tests/9a-user-ferretdb.sh`, `tests/9b-user-postgres.sh`, `tests/10-user-minio.sh`, `tests/14-runtime-testing.sh` +- deploy/test support in `tests/1-deploy.sh` is centered on names like `kind`, `k3s-amd`, `k3s-arm`, `k8s` +- the new examples provided are `k3s` and `k3sarm` Impact: -- The selector is not preserved consistently end-to-end. -- `k3s-amd` and `k3s-arm` are distinguished in deploy, but become just `k3s` in many subsequent test steps. +- there is a naming mismatch between the latest desired contract and the currently supported deploy/test names +- either: + - test names must be normalized to the new canonical names + - or the specification must explicitly preserve the current names + +Clarification: + +- under the latest requirement, the platform distinction is not a second dimension +- it is already encoded in the test name itself ### 7. Platform support is incomplete in the actual deploy path Evidence: -- `tests/1-deploy.sh:45-79` actively supports only `kind`, `k3s-amd`, `k3s-arm`. -- `tests/1-deploy.sh:86-212` has `mk8s`, `eks`, `aks`, `gke`, `osh` commented out. -- `tests/1-deploy.sh:214-229` supports generic `k8s`. -- `tests/all.sh:29-37` still advertises more platforms than `1-deploy.sh` actually enables. +- active deploy support in `tests/1-deploy.sh` is limited +- several other targets remain commented out Impact: -- "Selected infrastructure" is only partially true today. -- Any tag/label selecting `mk8s`, `eks`, `aks`, `gke`, or `osh` will not reach a working deploy path in the current GitHub workflow. +- even with a correct `<test>-<hash>` contract, only a subset of tests can currently execute end-to-end -### 8. Operator PR execution is not pinned to the requested SHA +### 8. Operator PR execution originally was not pinned to the requested SHA Evidence: -- `operator-pr-test.yaml:45-49` clones `--branch "$PR_REF"` but does not checkout `PR_SHA`. -- `PR_SHA` is only used for the image tag at `operator-pr-test.yaml:69-75`. +- the earlier operator PR flow cloned the PR branch without checking out the requested `PR_SHA` Impact: -- If the PR branch moves after dispatch, the workflow may build and test a newer commit than the one that triggered the run. +- if the branch moved after dispatch, the workflow could test code different from the commit that requested the run + +Note: -### 9. There is no equivalent "materialize PR code locally" step for `openserverless-task` +- this was one of the concrete problems identified during implementation and has already been addressed in the new workflow changes + +### 9. There was no equivalent "materialize PR code locally" step for `openserverless-task` Evidence: -- Operator flow clones the PR repo and patches `_operator/olaris/opsroot.json`, then exports `OPS_ROOT` (`operator-pr-test.yaml:45-87`). -- The generic tag flow in `tests.yaml` does not clone `openserverless-task`, does not checkout a PR ref/sha, and does not set `OPS_ROOT`. +- operator flow had dedicated PR checkout logic +- generic task tag flow did not clone `openserverless-task` at the requested PR revision Impact: -- Even after adding a PR trigger in `openserverless-task`, this repo still lacks the execution path required to test the actual PR contents instead of a default branch. +- task PRs could not be tested against their actual contents in a deterministic way -### 10. Validation and guardrails are missing +### 10. Validation and guardrails are still tied to the wrong abstraction Evidence: -- `platform-ci-tests.yaml:27-40` accepts raw `platform` and `tag` strings and pushes them directly as git tags. -- `operator-pr-test.yaml:26-30` accepts raw payload fields and passes `platform` directly into the test scripts. +- the recent implementation validates selector-like values +- the latest specification says the trigger should be `<test>-<hash>` Impact: -- Unsupported `<platform>-<architecture>` combinations fail late, inside the test workflow. -- There is no central allowlist for valid repositories or selectors. +- validation should move to: + - allowed test names + - valid commit-hash format +- instead of validating synthetic platform-architecture pairs + +### 11. Workflow and payload naming still use legacy terms + +Evidence: + +- `platform-ci-tests.yaml` still refers to `client_payload.platform` +- several documents and scripts still speak about selector or platform-architecture values + +Impact: + +- even where behavior is compatible with `<test>-<hash>`, the naming remains misleading +- future maintenance will be error-prone until the terminology becomes: + - `test` + - `hash` + - source repo / PR metadata + +## What in the previous analysis was still useful + +These earlier findings remain relevant even after the specification change: + +1. the system needed a real PR trigger for `openserverless-task` +2. the task tag flow was losing PR identity +3. parsing was inconsistent and needed normalization +4. deploy support was only partial +5. deterministic checkout by SHA was missing in some paths +6. workflow naming was obscuring the real trigger contract -## Minimum missing pieces before the target flow can work +## Minimum missing pieces under the new specification 1. A single trigger contract shared by `operator`, `task`, and `testing`: - repo - pr number - ref - sha - - selector `<platform>-<architecture>` -2. A PR-trigger workflow in `nuvolaris/openserverless-task`. -3. A task-specific workflow in `openserverless-testing` that checks out the requested task PR revision, not just a tag. -4. Deterministic checkout by SHA for both operator and task flows. -5. A routing layer in `openserverless-testing` that maps `<platform>-<architecture>` to supported infrastructures. -6. A normalized parser for selectors, instead of the current mixed comment/tag/string conventions. + - tag `<test>-<hash>` +2. A clear canonical list of test names: + - for example `k3s`, `k3sarm`, `kind` +3. Parsing logic that extracts: + - test name + - commit hash +4. A task-specific workflow in `openserverless-testing` that checks out the requested task PR revision, not just a tag +5. Deterministic checkout by SHA for both operator and task flows +6. A mapping layer in `openserverless-testing` from canonical test name to the actual infrastructure setup used by the tests +7. Terminology cleanup so workflows, payloads and docs consistently say: + - `test` + - `hash` + - not `platform` + - not `architecture` diff --git a/tests/all.sh b/tests/all.sh index 90d9a85..96084a3 100755 --- a/tests/all.sh +++ b/tests/all.sh @@ -19,6 +19,14 @@ resolve_test_selector "${1:?test selector}" TYPE="$TEST_SELECTOR" +echo "*** requested tag: $TEST_TAG ***" +echo "*** resolved test: $TEST_NAME -> $TEST_SELECTOR ***" +echo "*** platform: $TEST_PLATFORM | arch: $TEST_ARCH ***" +if test -n "$TEST_HASH" +then + echo "*** commit hash: $TEST_HASH ***" +fi + # Always use the main OPS branch for the local all.sh sequence. export OPS_BRANCH=main echo "*** using $OPS_BRANCH ***" diff --git a/tests/lib/selector.sh b/tests/lib/selector.sh index 9a55bf5..c6ebe45 100755 --- a/tests/lib/selector.sh +++ b/tests/lib/selector.sh @@ -4,50 +4,91 @@ resolve_test_selector() { local raw_selector="${1:?test selector}" local selector="$raw_selector" - if [[ "$selector" =~ ^(.+)-([0-9]{6,})$ ]]; then - selector="${BASH_REMATCH[1]}" + TEST_TAG="$raw_selector" + TEST_HASH="" + + if [[ "$selector" =~ ^([a-z0-9]+)-([0-9a-f]{7,40})$ ]]; then + TEST_NAME="${BASH_REMATCH[1]}" + TEST_HASH="${BASH_REMATCH[2]}" + else + TEST_NAME="$selector" fi - case "$selector" in - kind | kind-amd | kind-arm) + case "$TEST_NAME" in + kind | kind-amd) TEST_SELECTOR="kind" TEST_PLATFORM="kind" - TEST_ARCH="${selector#kind-}" + TEST_ARCH="amd" + ;; + k3s | k3s-amd) + TEST_SELECTOR="k3s-amd" + TEST_PLATFORM="k3s" + TEST_ARCH="amd" ;; - k8s | k8s-amd | k8s-arm) + k3sarm | k3s-arm) + TEST_SELECTOR="k3s-arm" + TEST_PLATFORM="k3s" + TEST_ARCH="arm" + ;; + k8s | k8s-amd) TEST_SELECTOR="k8s" TEST_PLATFORM="k8s" - TEST_ARCH="${selector#k8s-}" + TEST_ARCH="amd" ;; - k3s-amd | k3s-arm) - TEST_SELECTOR="$selector" - TEST_PLATFORM="k3s" - TEST_ARCH="${selector#k3s-}" + k8sarm | k8s-arm) + TEST_SELECTOR="k8s" + TEST_PLATFORM="k8s" + TEST_ARCH="arm" + ;; + mk8s | mk8s-amd) + TEST_SELECTOR="mk8s" + TEST_PLATFORM="mk8s" + TEST_ARCH="amd" ;; - mk8s | mk8s-amd | mk8s-arm) + mk8sarm | mk8s-arm) TEST_SELECTOR="mk8s" TEST_PLATFORM="mk8s" - TEST_ARCH="${selector#mk8s-}" + TEST_ARCH="arm" + ;; + eks | eks-amd) + TEST_SELECTOR="eks" + TEST_PLATFORM="eks" + TEST_ARCH="amd" ;; - eks | eks-amd | eks-arm) + eksarm | eks-arm) TEST_SELECTOR="eks" TEST_PLATFORM="eks" - TEST_ARCH="${selector#eks-}" + TEST_ARCH="arm" ;; - aks | aks-amd | aks-arm) + aks | aks-amd) TEST_SELECTOR="aks" TEST_PLATFORM="aks" - TEST_ARCH="${selector#aks-}" + TEST_ARCH="amd" ;; - gke | gke-amd | gke-arm) + aksarm | aks-arm) + TEST_SELECTOR="aks" + TEST_PLATFORM="aks" + TEST_ARCH="arm" + ;; + gke | gke-amd) TEST_SELECTOR="gke" TEST_PLATFORM="gke" - TEST_ARCH="${selector#gke-}" + TEST_ARCH="amd" ;; - osh | osh-amd | osh-arm) + gkearm | gke-arm) + TEST_SELECTOR="gke" + TEST_PLATFORM="gke" + TEST_ARCH="arm" + ;; + osh | osh-amd) + TEST_SELECTOR="osh" + TEST_PLATFORM="osh" + TEST_ARCH="amd" + ;; + osharm | osh-arm) TEST_SELECTOR="osh" TEST_PLATFORM="osh" - TEST_ARCH="${selector#osh-}" + TEST_ARCH="arm" ;; *) echo "ERROR: unsupported test selector '$raw_selector'" >&2 @@ -55,9 +96,5 @@ resolve_test_selector() { ;; esac - if [[ "$TEST_ARCH" == "$selector" ]]; then - TEST_ARCH="amd" - fi - - export TEST_SELECTOR TEST_PLATFORM TEST_ARCH + export TEST_TAG TEST_NAME TEST_HASH TEST_SELECTOR TEST_PLATFORM TEST_ARCH } diff --git a/tests/run-gh-suite.sh b/tests/run-gh-suite.sh index abd7e8d..bcee7e7 100755 --- a/tests/run-gh-suite.sh +++ b/tests/run-gh-suite.sh @@ -1,24 +1,58 @@ #!/bin/bash set -euo pipefail -SELECTOR="${1:?test selector}" +TEST_INPUT="${1:?test selector}" cd "$(dirname "$0")" +. "$(dirname "$0")/lib/selector.sh" +resolve_test_selector "$TEST_INPUT" + export OPS_BRANCH=main echo "*** using $OPS_BRANCH ***" +echo "*** requested tag: $TEST_TAG ***" +echo "*** resolved test: $TEST_NAME -> $TEST_SELECTOR ***" +echo "*** platform: $TEST_PLATFORM | arch: $TEST_ARCH ***" +if test -n "$TEST_HASH" +then + echo "*** commit hash: $TEST_HASH ***" +fi touch ../.secrets -./1-deploy.sh "$SELECTOR" -./3-sys-redis.sh "$SELECTOR" -./4a-sys-ferretdb.sh "$SELECTOR" -./4b-sys-postgres.sh "$SELECTOR" -./5-sys-minio.sh "$SELECTOR" -./6-login.sh "$SELECTOR" -./7-static.sh "$SELECTOR" -./8-user-redis.sh "$SELECTOR" -./9a-user-ferretdb.sh "$SELECTOR" -./9b-user-postgres.sh "$SELECTOR" -./10-user-minio.sh "$SELECTOR" -./14-runtime-testing.sh "$SELECTOR" +run_step() { + local label="${1:?step label}" + shift + local status=0 + + echo "::group::$label" + printf '\n[%s] START %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" "$label" + + set +e + "$@" + status=$? + set -e + + if test "$status" -eq 0 + then + printf '[%s] PASS %s\n\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" "$label" + else + printf '[%s] FAIL %s (exit %s)\n\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" "$label" "$status" + fi + + echo "::endgroup::" + return "$status" +} + +run_step "Deploy ($TEST_SELECTOR)" ./1-deploy.sh "$TEST_INPUT" +run_step "System Redis ($TEST_SELECTOR)" ./3-sys-redis.sh "$TEST_INPUT" +run_step "System FerretDB ($TEST_SELECTOR)" ./4a-sys-ferretdb.sh "$TEST_INPUT" +run_step "System Postgres ($TEST_SELECTOR)" ./4b-sys-postgres.sh "$TEST_INPUT" +run_step "System Minio ($TEST_SELECTOR)" ./5-sys-minio.sh "$TEST_INPUT" +run_step "Login ($TEST_SELECTOR)" ./6-login.sh "$TEST_INPUT" +run_step "Static ($TEST_SELECTOR)" ./7-static.sh "$TEST_INPUT" +run_step "User Redis ($TEST_SELECTOR)" ./8-user-redis.sh "$TEST_INPUT" +run_step "User FerretDB ($TEST_SELECTOR)" ./9a-user-ferretdb.sh "$TEST_INPUT" +run_step "User Postgres ($TEST_SELECTOR)" ./9b-user-postgres.sh "$TEST_INPUT" +run_step "User Minio ($TEST_SELECTOR)" ./10-user-minio.sh "$TEST_INPUT" +run_step "Runtime Testing ($TEST_SELECTOR)" ./14-runtime-testing.sh "$TEST_INPUT" diff --git a/workflow.md b/workflow.md index 3126c09..d3e50d5 100644 --- a/workflow.md +++ b/workflow.md @@ -1,224 +1,221 @@ # Workflow Notes +Note: + +- this document describes both the historical state and the target contract +- the branch `feat/test-tag-hash-contract` implements the `<test>-<hash>` trigger flow and clearer step-by-step test logging +- references below to the earlier selector-style contract are kept only to explain the transition + ## Repositories involved - `nuvolaris/openserverless-operator` - `nuvolaris/openserverless-task` - `nuvolaris/openserverless-testing` -## Goal +## Latest desired contract -Enable a PR in `openserverless-operator` or `openserverless-task` to trigger the corresponding test workflow in `openserverless-testing`, using a PR label/tag shaped as: +The latest specification says the trigger should be: -- `<platform>-<architecture>` +- `<test>-<hash>` -Examples: +Where: -- `k3s-amd` -- `k3s-arm` -- `kind-amd` +- `test` already includes the platform distinction +- no extra platform or architecture field is needed +- examples given are: + - `k3s` + - `k3sarm` +- `hash` is the commit hash that generated the test -## Current repositories and branches used during this analysis +This is different from the earlier interpretation based on selectors like `k3s-amd` or `kind-amd`. -Implementation work was pushed to: +The key clarification is: -- `feat/pr-tag-platform-arch-testing` in `openserverless-testing` -- `feat/pr-tag-platform-arch-testing` in `openserverless-operator` -- `feat/pr-tag-platform-arch-testing` in `openserverless-task` +- platform is already part of the test name +- there is no independent platform or architecture dimension to reconstruct later ## Important GitHub Actions behavior ### Why branch-only changes are not enough for end-to-end PR testing -For the new PR-trigger flow: +For PR-triggered flows: - source repos use `pull_request_target` - `pull_request_target` reads the workflow from the base branch of the PR -- `repository_dispatch` workflows in `openserverless-testing` must also exist on the target branch used by GitHub +- `repository_dispatch` workflows in `openserverless-testing` must also exist on the branch GitHub uses for the target repo Practical consequence: -- opening a PR on `openserverless-operator` against `main` does not automatically use the new trigger logic if that logic only exists in the PR branch -- the new flow becomes real only after the relevant workflow changes are available on the base branch, typically `main` +- opening a PR against `main` does not automatically use workflow logic that exists only in the PR branch +- the new behavior becomes real only after the relevant workflow changes are available on `main` + +## What was implemented during this work + +### Source repo triggers + +Feature branches added PR-trigger logic in: + +- `openserverless-operator/.github/workflows/trigger-testing.yaml` +- `openserverless-task/.github/workflows/trigger-testing.yaml` + +Those changes currently enable testing from PR labels and dispatch to `openserverless-testing`. +That behavior was enough to validate the PR-trigger pipeline, but its naming contract is still older than the final requirement. + +### Testing repo flows + +Feature branches added or updated: + +- `openserverless-testing/.github/workflows/operator-pr-test.yaml` +- `openserverless-testing/.github/workflows/task-pr-test.yaml` +- `openserverless-testing/.github/workflows/tests.yaml` + +Main improvements introduced: + +- deterministic checkout by `PR_SHA` +- dedicated task PR workflow +- common test sequence extraction +- more consistent selector parsing than before +- explicit `OPS_BRANCH=main` in the shared GitHub test path + +## Important note: current implementation vs latest specification + +The implemented PR-trigger flow was built around label values like: + +- `kind-amd` +- `k3s-amd` +- `k3s-arm` + +The latest specification instead says the tag should be: -Minimum merge set for real usage: +- `<test>-<hash>` -1. merge `openserverless-testing` -2. merge `openserverless-operator` -3. merge `openserverless-task` if task PRs must also trigger tests +Examples: + +- `k3s-<hash>` +- `k3sarm-<hash>` + +So the code currently merged on `main` improved the trigger pipeline, but it is not yet fully aligned with this final naming contract. + +One important nuance: -## Current testing flow before the new changes +- the older tag-driven task path in `platform-ci-tests.yaml` is structurally closer to `<test>-<hash>` +- the newer selector-based PR trigger improved orchestration and PR metadata handling, but uses the wrong external naming contract + +## Historical testing flow ### Operator -Existing trigger path: +Previous operator path: 1. `openserverless-operator/.github/workflows/trigger-testing.yaml` -2. trigger came from: +2. trigger originally came from: - issue comment `/testing <platform>` - manual `workflow_dispatch` -3. workflow collected: - - PR number - - PR ref - - PR sha - - source repo - - platform selector -4. workflow dispatched `repository_dispatch` to `openserverless-testing` -5. `openserverless-testing/.github/workflows/operator-pr-test.yaml` built a temporary operator image and executed tests +3. workflow dispatched `repository_dispatch` to `openserverless-testing` +4. `openserverless-testing/.github/workflows/operator-pr-test.yaml` built a temporary operator image and ran tests ### Task -Existing trigger path: +Previous task path: -1. `openserverless-testing/.github/workflows/platform-ci-tests.yaml` could receive `repository_dispatch` -2. it converted payload into a git tag +1. `openserverless-testing/.github/workflows/platform-ci-tests.yaml` accepted `repository_dispatch` +2. payload was converted into a git tag 3. `openserverless-testing/.github/workflows/tests.yaml` ran tests on tag push -Missing piece: - -- `openserverless-task` had no `.github/workflows` directory, so there was no workflow producing that dispatch from a PR +Original missing piece: -## New testing flow implemented in the feature branches +- `openserverless-task` did not have a workflow producer for that dispatch -### Source of truth for enabling tests +What was already aligned in spirit: -A PR label/tag matching one of these selectors enables the process: +- `platform-ci-tests.yaml` already creates a tag from two parts +- `tests.yaml` already accepts any tag matching `*-*` +- this means the historical task path was already closer to a canonical `<test>-<hash>` trigger than the later `kind-amd` style label naming -- `kind-amd` -- `kind-arm` -- `k3s-amd` -- `k3s-arm` -- `k8s-amd` -- `k8s-arm` -- `mk8s-amd` -- `mk8s-arm` -- `eks-amd` -- `eks-arm` -- `aks-amd` -- `aks-arm` -- `gke-amd` -- `gke-arm` -- `osh-amd` -- `osh-arm` - -### Operator flow after the feature changes +## Current code paths that are still useful under the new contract -Files: +Even with the new `<test>-<hash>` specification, these pieces remain useful: -- `openserverless-operator/.github/workflows/trigger-testing.yaml` - `openserverless-testing/.github/workflows/operator-pr-test.yaml` + Purpose: + checkout operator PR by SHA, build temporary image, run tests +- `openserverless-testing/.github/workflows/task-pr-test.yaml` + Purpose: + checkout task PR by SHA and run tests from the requested PR contents +- `openserverless-testing/.github/workflows/tests.yaml` + Purpose: + generic tag-based test entry point for `*-*` +- `openserverless-testing/tests/run-gh-suite.sh` + Purpose: + central GitHub test sequence -Flow: - -1. PR in `openserverless-operator` receives a valid label -2. `pull_request_target` runs on: - - `labeled` - - `synchronize` - - `reopened` -3. workflow scans PR labels and picks the first valid `<platform>-<architecture>` selector -4. workflow fetches PR details via `gh api` -5. workflow dispatches `operator-pr-test` to `openserverless-testing` -6. testing workflow clones the operator PR branch -7. testing workflow explicitly checks out the requested `PR_SHA` -8. testing workflow builds a temporary operator image -9. testing workflow patches `opsroot.json` -10. testing workflow runs the test suite on the selected infrastructure +## What needs realignment for the latest specification -### Task flow after the feature changes +### 1. Trigger naming -Files: +The system must treat the tag as: -- `openserverless-task/.github/workflows/trigger-testing.yaml` -- `openserverless-testing/.github/workflows/task-pr-test.yaml` +- `<test>-<hash>` -Flow: +Not as: -1. PR in `openserverless-task` receives a valid label -2. `pull_request_target` runs on: - - `labeled` - - `synchronize` - - `reopened` -3. workflow scans PR labels and picks the first valid selector -4. workflow fetches PR details via `gh api` -5. workflow dispatches `task-pr-test` to `openserverless-testing` -6. testing workflow clones the task PR branch -7. testing workflow explicitly checks out the requested `PR_SHA` -8. testing workflow sets `OPS_ROOT` to the checked out task tree -9. testing workflow runs the test suite on the selected infrastructure +- `<platform>-<architecture>` -## Changes made in `openserverless-testing` +### 2. Canonical test names -### New workflows +The system needs one stable list of accepted test names. -- `./.github/workflows/task-pr-test.yaml` +Examples currently mentioned by the latest clarification: -Purpose: +- `k3s` +- `k3sarm` -- dedicated repository-dispatch workflow for PR testing of `openserverless-task` +That means the code should not rely on split forms like: -### Updated workflows +- `k3s-amd` +- `k3s-arm` -- `./.github/workflows/operator-pr-test.yaml` -- `./.github/workflows/tests.yaml` +unless those remain officially supported aliases. -Main updates: +### 3. Parser behavior -- operator PR testing now checks out the exact requested SHA -- common test execution is centralized through `tests/run-gh-suite.sh` -- selector is passed consistently as one input to the suite +Parsing should extract: -### New selector normalization +- `test` +- `hash` -Files: +The earlier selector parser was built for names that embedded an extra `-amd` or `-arm` part. -- `./tests/lib/selector.sh` -- `./tests/run-gh-suite.sh` +### 4. Mapping from test name to infrastructure -Purpose: +Because platform is already encoded in the test name, the test runner should map: -- normalize `<platform>-<architecture>` once -- preserve distinctions like `k3s-amd` vs `k3s-arm` -- remove inconsistent parsing scattered across test scripts +- `k3s` -> the corresponding infrastructure path +- `k3sarm` -> the corresponding infrastructure path -### Test scripts updated to use shared selector parsing +instead of reconstructing platform and architecture as separate dimensions. -Updated: +### 5. Workflow payload naming -- `./tests/1-deploy.sh` -- `./tests/2-ssl.sh` -- `./tests/6-login.sh` -- `./tests/7-static.sh` -- `./tests/8-user-redis.sh` -- `./tests/9a-user-ferretdb.sh` -- `./tests/9b-user-postgres.sh` -- `./tests/10-user-minio.sh` -- `./tests/12-nuv-mac.sh` -- `./tests/14-runtime-testing.sh` -- `./tests/all.sh` +Even where behavior is already close, workflow fields should stop implying a separate platform dimension. -Why this mattered: +Examples of terminology that should be renamed in the next code alignment: -- before the change, many scripts reduced `k3s-amd` and `k3s-arm` to `k3s` -- deploy logic distinguished them, but later test steps did not preserve the selector consistently +- `platform` -> `test` +- selector -> tag or test selector only if it really means `<test>-<hash>` -## Current deploy/test platform support in `openserverless-testing` +## Current deploy/test support in `openserverless-testing` -Effective support in `tests/1-deploy.sh`: +Current deploy logic is still based on names such as: - `kind` - `k3s-amd` - `k3s-arm` - `k8s` -Present but commented out in deploy path: - -- `mk8s` -- `eks` -- `aks` -- `gke` -- `osh` +This is one of the main remaining mismatches with the latest desired naming. -This means the label set is broader than the currently active deploy implementation. Some selectors are accepted by workflow validation, but their deploy path still depends on commented or incomplete code in `tests/1-deploy.sh`. +The current deploy logic therefore supports the infrastructure execution, but not yet the final canonical naming expected by the new contract. ## Operator image build and publication @@ -231,15 +228,16 @@ File: What happens: 1. testing workflow clones the operator PR -2. builds an image from the PR source -3. pushes it to GHCR as a temporary test image -4. patches the task configuration to use that image during tests +2. checks out the requested `PR_SHA` +3. builds an image from the PR source +4. pushes it to GHCR as a temporary test image +5. patches the task configuration to use that image during tests Destination: - `ghcr.io/<repository_owner>/openserverless-testing:pr-<PR_NUMBER>-<SHORT_SHA>` -This is a testing image, not the official published operator image. +This is only the test image. ### Official operator image publication @@ -255,8 +253,8 @@ Trigger: Important conclusion: -- there is currently no automatic path from “PR tests passed” to “publish official operator image” -- publication depends on a tag push +- there is no automatic path today from “PR tests passed” to “publish official operator image” +- official publication still depends on pushing a release tag ### Where the official operator image is published @@ -282,7 +280,7 @@ Actual push implementation: There is no workflow that automatically creates the release tag after tests or after merge. -Current tag creation is manual or task-driven: +Current tag creation remains manual or task-driven: - `openserverless-operator/Taskfile.yml` task `tag` - `openserverless-operator/Taskfile.yml` task `tag-commit-push` @@ -291,41 +289,61 @@ So the current release path is: 1. someone creates or pushes a numeric version tag 2. tag push triggers `image.yml` -3. `image.yml` runs tests/build and then publishes the official image +3. `image.yml` builds and publishes the official image ## Validation performed during implementation -Static checks executed: +Static checks executed during the implementation work: - `bash -n` on updated shell scripts in `openserverless-testing` - YAML parsing with Ruby for modified workflows in all three repositories -- quick selector parsing checks for: - - `kind-amd` - - `k3s-arm` - - `k8s-arm` +- a real trigger validation using: + - an operator PR + - label `kind-amd` + - successful dispatch into `openserverless-testing` -Not executed: +That real validation confirmed: -- full end-to-end GitHub Actions run on live PRs -- real registry publication test for the final operator release image +- the PR-trigger pipeline works +- `repository_dispatch` reaches `openserverless-testing` +- the operator PR test workflow starts correctly + +But it validated the selector-style contract, not yet the final `<test>-<hash>` contract. ## Open points -### 1. Label naming +### 1. Exact meaning of “tag” + +The latest requirement uses the word “tag” as: + +- `<test>-<hash>` + +This should now become the canonical name everywhere in code and docs. + +### 2. Transition plan + +There are now two conceptual layers: + +- trigger pipeline improvements already implemented and merged +- final naming contract `<test>-<hash>` still to be aligned -The implementation assumes GitHub PR labels are the “tag” mentioned in the process description. +### 3. Next update needed in code -### 2. Multiple valid labels on the same PR +To fully align the codebase, the next round should update: -Current logic picks the first matching valid selector found on the PR. +- trigger parsing +- validation rules +- test-name allowlists +- deploy mapping +- workflow payload field names +- documentation -### 3. Release automation +so everything consistently speaks in terms of: -If desired, a future improvement could connect: +- `test` +- `hash` -- PR test success -- merge to `main` -- release tag creation -- official image publication +and no longer in terms of: -Today these are still separate processes. +- platform +- architecture
