This is an automated email from the ASF dual-hosted git repository. sciabarracom pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/openserverless.git
commit 98df5eed3ef39ca0a3962a0f3e30cbdf7fa5d128 Author: Michele Sciabarra <[email protected]> AuthorDate: Wed Sep 9 13:53:12 2026 +0200 Developer Guide and updated README --- .gitignore | 1 + .gitmodules | 3 + BUILD.md | 410 ----------------------------- DEVEL.md | 862 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 28 +- build | 2 +- site | 1 + streamer | 2 +- 8 files changed, 894 insertions(+), 415 deletions(-) diff --git a/.gitignore b/.gitignore index 0f23818..da8a4ca 100644 --- a/.gitignore +++ b/.gitignore @@ -23,3 +23,4 @@ network-cache/ ops RELEASE rebrander.log +opsroot.orig diff --git a/.gitmodules b/.gitmodules index 3196ba1..cc578e0 100644 --- a/.gitmodules +++ b/.gitmodules @@ -58,3 +58,6 @@ path = testing url = https://github.com/apache/openserverless-testing branch = 0.9.0 +[submodule "site"] + path = site + url = https://github.com/apache/openserverless-site diff --git a/BUILD.md b/BUILD.md deleted file mode 100644 index 4af1473..0000000 --- a/BUILD.md +++ /dev/null @@ -1,410 +0,0 @@ -# BUILD.md — Specification - -> **Status: specification only.** This document describes what the build -> documentation must explain and how the build system is supposed to behave. -> It is not an implementation guide, and nothing here is to be implemented -> as part of writing this document. -> -> **The Taskfiles are expected to change to match this spec.** Where the current -> `Taskfile.yml` of a component disagrees with the contract described here, the -> spec is the target and the Taskfile is what gets updated — not the other way -> round. Section 9 lists the known divergences. - ---- - -## 1. Summary - -OpenServerless is not one build. It is a set of independently versioned -components, each living in its own submodule, each with its own `Taskfile.yml` -and its own GitHub Actions workflow. The final documentation must make the -reader understand three things, in this order: - -1. **The common rules** — how tags drive every build, what a tag looks like, - what `task tag` / `task build` / `task ci` mean, and where images are pushed. -2. **The three build cases** — which one applies to the component being worked on. -3. **The three usage scenarios** — building locally, building from a personal - fork, and consuming what the fork produced. - -### The three build cases - -| Case | Components | Distinguishing feature | -|---|---|---| -| **A — single image** | `operator` (olaris-op), `admin-api`, `streamer`, `devcontainer` | One tag → one image | -| **B — multiple images** | `runtimes` | One tag → a whole family of images, tag selects which family | -| **C — binary** | `cli` (`ops`) | No image; produces release binaries | - -### The three usage scenarios - -| Scenario | What the reader wants | Where artifacts land | -|---|---|---| -| **Local** | Build on my machine, run it against my cluster | Local Docker daemon / kind, nothing pushed | -| **Fork** | Build my own version in CI from my fork | My registry namespace | -| **Consume** | Point an OpenServerless install at what I built | `opsroot.json` / `runtimes.json` overrides | - -### The one rule that governs everything - -**All CI builds are triggered by pushing a git tag.** No branch push builds an -image. Every component workflow is configured as `on: push: tags:` with -`branches-ignore: '*'`. The tag is not metadata attached to a build — the tag -*is* the build trigger, and its text *is* the image tag. - ---- - -## 2. Common rules - -These apply to every component and must be stated once, up front, in the final -documentation. The per-case sections must then only describe deviations. - -### 2.1 The three tasks - -Every component exposes the same conceptual trio. The documentation must -present them as a progression, because the reader will use them in this order: - -| Task | Runs where | Effect | Pushes? | -|---|---|---|---| -| `task tag` | local | Creates (and force-replaces) the git tag that names this build | no | -| `task build` | local | Builds using the current tag, loads into the local Docker daemon | no | -| `task ci` | local, triggers remote | Pushes the tag, which starts the GitHub Actions build | yes, via CI | - -Two points the documentation must make explicitly, because they are the two -things newcomers get wrong: - -- `task tag` **deletes existing local tags** before creating the new one. This - is deliberate — snapshot tags are disposable — but it must be flagged, not - discovered. -- `task ci` does not itself build anything. It pushes a tag. The build happens - in GitHub Actions on the repository that received the tag. This is why forking - matters: the tag lands in *your* fork, so *your* Actions run, using *your* - secrets, pushing to *your* registry. - -These three names are a **contract**: every component must expose `tag`, -`build`, and `ci` with exactly these semantics, regardless of what it builds. -Components that currently use different names or a different split must be -aligned to it (see section 9). - -A manual trigger path also exists: the workflow can be started from the GitHub -Actions UI, in which case it creates the tag itself. The documentation must -mention this as the alternative for people who cannot or do not want to push -tags from the command line. - -### 2.2 Tag format - -Tags are timestamps in a compacted, sortable form. The documentation must -explain the encoding, because the tags are otherwise unreadable: - -``` -26i4k36-SNAPSHOT -││││││└─ minutes -│││││└── hour, encoded as a letter: a=00 … x=23 -││││└─── day of month -││└───── month, encoded as a letter: a=January … l=December -└─────── year (two digits) -``` - -So `26i4k36` is 2026, September (`i` = 9th letter), day 4, hour 10 (`k` = 11th -letter, zero-based → 10), minute 36. - -The `-SNAPSHOT` suffix marks a non-release build. The documentation must state -what its presence and absence mean for consumers. - -Case B (runtimes) prefixes the timestamp with the runtime family: - -``` -<runtime>_26i4k36-SNAPSHOT -``` - -for example `python_26i4k36-SNAPSHOT` or `nodejs_26i4k36-SNAPSHOT`. The prefix -is not cosmetic: CI parses it to decide which subset of images to build. This -must be stated as a rule, not shown as an example. - -Components carry their own base version prefix where applicable (e.g. the CLI -produces `v0.9.0-<timestamp>.SNAPSHOT`). The documentation must show one real -tag per case rather than describing the format abstractly. - -### 2.3 Where images go — no configuration required - -The destination registry and the credentials used to push are **derived -automatically from the repository the tag was pushed to**. Nothing has to be -configured for the normal case, and the documentation must lead with that fact -rather than with a configuration table. - -The rule: - -| Tag pushed to | Registry | Namespace | Credentials | -|---|---|---|---| -| the Apache repository | `docker.io` | `apache` | secrets `DOCKERHUB_USER` / `DOCKERHUB_TOKEN` | -| any fork | `ghcr.io` | the fork owner's GitHub user | the current GitHub actor + the built-in `GITHUB_TOKEN` | - -So a contributor who forks and pushes a tag gets images at -`ghcr.io/<their-user>/…` with **no secrets to create and no `.env` to write** — -the workflow authenticates as the user running it, using the token GitHub -provides to every Actions run. This must be stated as the default path, not as a -fallback. - -### 2.4 Overriding the registry (optional) - -An explicit registry can be selected by setting `DOCKERHUB_REGISTRY` together -with the secrets `DOCKERHUB_USER` and `DOCKERHUB_TOKEN`. The documentation must -present this as an override for people who want their images somewhere other -than their own GHCR namespace — **not** as a required setup step. - -### 2.5 Local builds — defaults and `MY_*_IMAGE` overrides - -Everything above describes CI. When building **locally**, no registry is -involved by default: each component builds its image under the same default name -it would use in CI (section 2.3), and the result exists only in the local Docker -daemon. - -That default can be overridden per component by exporting a `MY_<COMPONENT>_IMAGE` -environment variable, normally set in a local `.env` file. The variable holds the -**full image name without the tag** — registry, namespace and repository — and -the tag continues to be computed as described in section 2.2. - -The variables, one per single-image component: - -| Component | Variable | Example value | -|---|---|---| -| operator | `MY_OPERATOR_IMAGE` | `ghcr.io/<your-user>/openserverless-operator` | -| devcontainer | `MY_DEVCONTAINER_IMAGE` | `ghcr.io/<your-user>/openserverless-devcontainer` | -| admin-api | `MY_ADMINAPI_IMAGE` | `ghcr.io/<your-user>/openserverless-admin-api` | -| streamer | `MY_STREAMER_IMAGE` | `ghcr.io/<your-user>/openserverless-streamer` | - -Each component repository must ship a `.env.dist` listing these variables with -their intended default form, so that copying it to `.env` and filling in the -GitHub user is the whole of the setup. `.env` itself is local and must stay -untracked. - -The documentation must present this as the **local** counterpart of section 2.3: -the automatic rule still decides the default, and `MY_*_IMAGE` only replaces it -when someone wants their local build named for their own registry. - -### 2.6 The exception: repositories publishing multiple images - -Two repositories publish images that do **not** correspond to the repository -itself: - -| repository | images published | -|---|---| -| `openserverless-build` | `openwhisk2/controller`, `openwhisk2/invoker`, `openwhisk2/scheduler`, `openwhisk2/standalone`, `openwhisk2/scala` | -| `openserverless-runtimes` | `openserverless-runtime-<rt>` — one per language version | - -This is the single place where the automatic path of section 2.3 is not -sufficient, and the documentation must call it out explicitly rather than -leaving it to be discovered from a failed build. - -The built-in `GITHUB_TOKEN` is scoped to packages **owned by the repository -running the workflow**. GHCR links a package to its creating repository on first -push and grants write access only to that repository. Because these two builds -publish under image names that do not match the repository, `GITHUB_TOKEN` -carries no permission for them and the push is refused. - -Therefore, for `openserverless-build` and `openserverless-runtimes` only, when -pushing to **ghcr.io**: - -- a **`GHCR_TOKEN`** secret **must** be set on the fork, and -- it must be a GitHub Personal Access Token (classic) with the - **`write:packages`** scope — write permission on the target images, not merely - on the packages of the current repository. - -To create it: - -1. Go to <https://github.com/settings/tokens> → *Generate new token (classic)*. -2. Select the **`write:packages`** scope (this implies `read:packages` and - `repo`). -3. Copy the generated token. -4. In your fork: *Settings → Secrets and variables → Actions → New repository - secret*, name it exactly **`GHCR_TOKEN`**, and paste the value. - -When pushing to Docker Hub instead, the equivalent requirement is a -`DOCKERHUB_TOKEN` with write permission on the target images. - -The documentation must state the symptom this produces when skipped — a -`denied: permission_denied: write_package` error at push time, after a long and -otherwise successful build — so the cause is recognisable from the error alone. -The workflows fail fast with these instructions when the secret is missing, -rather than building first and failing at the push. - ---- - -## 3. Case A — single image - -Applies to: `operator` (`olaris-op`), `admin-api`, `streamer`, `devcontainer`. - -One tag produces one image. This is the simplest case and should be documented -first, in full, so the later cases can be described as deltas. - -The documentation must cover: - -- **Prerequisites** — Docker with buildx, Task, and (for CI) a fork. -- **Local flow** — `task tag`, then `task build`. State that the resulting image - is tagged with the timestamp and exists only in the local Docker daemon. -- **CI flow** — `task ci` pushes the tag; the `image` workflow builds multi-arch - (`linux/amd64`, `linux/arm64`) and pushes. -- **Fork setup** — nothing to do. Per section 2.3 the images go to - `ghcr.io/<your-user>/…` automatically, authenticated with the built-in - `GITHUB_TOKEN`. Mention the `DOCKERHUB_REGISTRY` override only as an aside. -- **What CI does beyond building** — license check (skywalking-eyes) and unit - tests run before the image is built. A build can fail for reasons unrelated to - the image; the documentation must say so, so failures are not mysterious. -- **Verification** — how to confirm the image exists, locally and in the registry. - ---- - -## 4. Case B — multiple images (runtimes) - -Applies to: `runtimes`. - -Structurally the same as Case A, but with three differences that the -documentation must call out as the *entire* reason this case is separate: - -1. **The tag selects the work.** `task tag RT=<runtime>` requires the runtime - family. CI parses the tag prefix and builds only the matching family — - `python_…` builds the Python runtimes, `common-…` builds the shared base - image. The documentation must list the recognized prefixes and state what - happens when a tag matches none of them. - -2. **One tag, many images.** A single build produces one image per language - version in the family (e.g. several Python versions). The documentation must - show the resulting image naming scheme: - - ``` - <registry>/<namespace>/openserverless-runtime-<rt>:<ver>-<tag> - ``` - -3. **A token must be supplied.** This is the one case where the automatic - credentials of section 2.3 are not enough: the images published are not the - current repository's own packages, so the built-in `GITHUB_TOKEN` lacks the - permission. Per section 2.6 a **`GHCR_TOKEN`** secret with the - `write:packages` scope must be set in the fork (Settings → Secrets and - variables → Actions) when pushing to ghcr.io, or a `DOCKERHUB_TOKEN` with - write access to the target images when pushing to Docker Hub — optionally - alongside `DOCKERHUB_REGISTRY` and `DOCKERHUB_USER` to change the - destination. The documentation must present this as *the* prerequisite for - Case B, and note that `openserverless-build` carries the same requirement for - the same reason. - -The documentation must also explain the relationship between the **common base -image** and the language runtimes: the base is built and tagged separately, and -language runtimes depend on it. Building a language family without a matching -base is a documented failure mode, not a bug to be discovered. - -Finally, this section must explain how the built runtimes are made visible to an -installation — the generated `runtimes.json` / `runtimes.env` and how they are -consumed. - ---- - -## 5. Case C — binary (cli) - -Applies to: `cli`, which produces the `ops` binary. - -No Docker image is involved. The documentation must cover: - -- **Local flow** — `task build` compiles `ops` into the repository root. Note - that `task build` alone is enough for local use; tagging is only needed when - the embedded version string matters. -- **Tagging** — `task tag` writes `version.txt` and `branch.txt`, commits them, - and creates the tag. The documentation must state that this **creates a - commit**, which is a meaningful difference from the other cases. -- **CI flow** — `task trigger` (the CLI's equivalent of `task ci`) refuses to run - with a dirty working tree, then tags and pushes. The release workflow builds - and publishes binaries for all supported platforms. -- **Installation** — how to put the built `ops` on the `PATH`, and how to verify - which version is running. - ---- - -## 6. The Apache image - -The documentation must include a dedicated section on building the Apache -OpenWhisk-derived images (controller, invoker, scheduler, standalone, scala -base), because it does not follow the pattern of the three cases above. - -It must explain: - -- **Why it is different** — it lives in the `build` submodule, has its own - Taskfile, vendors an OpenWhisk source tree, and builds through a dedicated - devcontainer rather than on the host. -- **The toolchain constraint** — the OpenWhisk Gradle build requires Java 11; - building on a host with a newer JDK fails with `Unsupported class file major - version`. The build therefore runs inside `ops ide devcontainer`. This must be - stated as a rule with its symptom, so the error is self-diagnosing. -- **The local flow** — `task build` in the `build` directory, and what it does - end to end: build images, update `opsroot.json`, load images into kind, roll - out the controller. -- **The two-build-systems caveat** — the root `Taskfile.yml` and - `build/Taskfile.yml` are separate. The root one builds CLI, operator, - runtimes, streamer, admin-api and updates only `operator`, `streamer`, - `systemapi` in `opsroot.json`. It does **not** build or update the OpenWhisk - images. The documentation must state this explicitly; it is the single most - common wrong assumption. -- **Verification** — how to confirm the running pod uses the freshly built - image, not a cached upstream one. - ---- - -## 7. Consuming your own build - -A section the reader reaches after a successful build, answering: *I built it — -now how do I actually run it?* - -It must cover: - -- Overriding component images in `~/.ops/<version>/olaris/opsroot.json`. -- Overriding runtimes via the generated runtimes configuration. -- Loading locally built images into kind (they are not automatically visible to - the cluster — this must be stated, with the symptom: the pod pulls the old - upstream image instead). -- Confirming the override took effect at the pod level, not just in the custom - resource. A change visible in the CR but not in the resulting workload is a - known failure mode and must be documented as such. - ---- - -## 8. Requirements for the final document - -- **Summary first.** The common rules and the two 3×3 tables come before any - component-specific instruction. -- **Cases are separated.** No section may mix Case A, B, and C instructions. - Cases B and C are written as deltas from A, not as full restatements. -- **Every command is shown in context** — which directory it runs in, and what - it produces. -- **Every failure mode listed above appears with its symptom**, so the reader - can match an error message to a cause. -- **Scenario-oriented, not task-oriented.** The reader arrives with a goal - ("build my own runtime image and test it"), not with a task name. - ---- - -## 9. Taskfile alignment - -The contract in section 2 is the target state. The Taskfiles do not all -implement it today, and updating them is in scope for the work this spec -governs. The known divergences: - -| Component | Divergence | Required change | -|---|---|---| -| `cli` | The CI trigger is called `trigger`, not `ci` | Expose `ci` (keeping `trigger` as an alias if desired) | -| `streamer` | The tagging task is called `image-tag`, not `tag` | Expose `tag` | -| `streamer`, `admin-api` | `ci` is absent — the tag is pushed by hand | Add `ci`: create the tag and push it | -| `runtimes` | The CI entry point is `ci-build`, not `ci`; `tag` prints the new tag but does not create it | Expose `ci`; make `tag` actually create the tag | -| `runtimes` | `ci-build` calls `build-lang` without `task`, so the language path is broken | Fix the invocation | -| `admin-api` | Tag suffix is `-SNAPSHOT`; `streamer` has no suffix | Make the suffix consistent across single-image components | -| all single-image | Tag timestamp is `date +%y%m%d%H%M`, not the letter-encoded form of section 2.2 | Adopt one encoding everywhere, and document which | - -Two decisions must be made explicitly before the Taskfiles are touched, because -the documentation cannot be written without them: - -1. **One tag encoding or two.** Either every component uses the letter-encoded - timestamp, or the plain numeric one. The current mix must not be documented - as if it were intentional. -2. **Convergence on the automatic registry rule.** Section 2.3 is the target: - registry, namespace, and credentials derived from the repository owner, with - no configuration in the normal case. The single-image Taskfiles currently - require `REGISTRY` and `NAMESPACE` in a local `.env` and fail without them; - they must be changed to fall back to the automatic behaviour and treat - `DOCKERHUB_REGISTRY` / `DOCKERHUB_USER` / `DOCKERHUB_TOKEN` as an override. - The runtimes exception (2.6) stays. - -Until these are settled, the documentation must not describe the current -behaviour as stable. diff --git a/DEVEL.md b/DEVEL.md new file mode 100644 index 0000000..7d9c0e1 --- /dev/null +++ b/DEVEL.md @@ -0,0 +1,862 @@ +# Developer guide + +OpenServerless is a complete serverless development environment. + +It allows you to build full stack, scalable, cloud-native applications with backend and +frontend, using services like PostgreSQL (SQL), FerretDB (NoSQL) and Milvus (vector) +databases, Redis cache and S3-compatible object storage, with minimum effort. Check the +tutorial on <https://openserverless.apache.org> for a complete guide on how to use it. + +**This guide focuses on how to develop the system itself, not on how to use it.** + +## TL;DR + +The short version, for the impatient. Everything below this section explains *why* each +step is what it is; this is the sequence that gets you to a running local build. + +**1. Clone with submodules.** Most of the code is in submodules, so this is not optional: + +```bash +git clone --recurse-submodules https://github.com/apache/openserverless +cd openserverless +``` + +**2. Build the CLI first.** Nothing else works until `./ops` exists: + +```bash +task cli # compiles cli/ into ./ops +``` + +The task files it runs are the ones in [`oplugins/`](oplugins/) — plain YAML, editable in +place. + +> [!WARNING] +> **`./ops` behaves differently depending on where you are.** Run it from this directory +> and it uses your local [`oplugins/`](oplugins/). Run it from anywhere else and it +> **downloads** the task files from GitHub and uses those instead — your local edits are +> silently ignored. Always check with `./ops -info` that `OPS_ROOT` points inside your +> working copy. See [How `ops` finds its tasks](#how-ops-finds-its-tasks). + +**3. Know where the images come from.** [`oplugins/opsroot.json`](oplugins/opsroot.json) +holds, under `.config.images`, the pointer to every image that gets deployed. Read it +before you wonder why the cluster is running something other than what you built — see +[`opsroot.json`](#opsrootjson). + +**4. Build only what you need, then load it into kind.** Each component has its own task, +and each one builds the image *and* saves it so kind can pick it up: + +```bash +task operator # or: task openwhisk, task admin-api, task streamer, task runtimes +task opsroot # rewrite opsroot.json to point at what you just built +ops setup docker # create the kind cluster and load your local images into it +``` + +`task build` does all of them at once — useful the first time, rarely what you want after. +Nothing is pulled from a registry: the images are saved to disk and imported straight into +the kind node. See [The development loop, end to end](#the-development-loop-end-to-end). + +**5. Pick the submodules you actually work on.** You generally do not need to build +everything. Work inside a single subrepo, rebuild just that component, redeploy — see +[Iterating on a single component](#iterating-on-a-single-component). + +**6. To publish images, fork and push a tag.** CI builds are triggered by tags, and they +run in whichever repository received the tag — so the tag must land in *your* fork: + +```bash +# fork the repo on GitHub, then, in the component's directory: +git remote set-url origin https://github.com/<your-org>/<the-fork> +task tag # create the timestamp tag (this deletes existing local tags) +task ci # push the tag → GitHub Actions builds and pushes the image +``` + +See [Publishing images](#publishing-images). + +## Components + +The key components, each one a separate git repository checked out as a subdirectory of +this one: + +| Directory | Repository | Role | +|-----------------|-------------------------------------|------| +| `cli` | `apache/openserverless-cli` | the `ops` CLI: controls the operator and implements the development environment | +| `oplugins-op` | `apache/openserverless-operator` | the operator, installing all the components including the integrated services | +| `admin-api` | `apache/openserverless-admin-api` | the admin API, complementing the operator | +| `streamer` | `apache/openserverless-streamer` | streaming support for AI applications | +| `runtimes` | `apache/openserverless-runtimes` | the runtimes to run applications in Python/Node.js/Go/Java using the integrated services | +| `build` | `apache/openserverless-build` | the OpenWhisk fork: controller and invoker images | +| `devcontainer` | `apache/openserverless-devcontainer`| the development container image | +| `oplugins` | `apache/openserverless-task` | the task files (the "plugins") that `ops` executes | +| `testing` | — | the end-to-end test suite | + +## Start here: build the CLI + +Everything else in this guide goes through `ops`, so build it first. From the root of this +repository: + +``` +task cli +``` + +That compiles the Go sources in [`cli/`](cli/) and leaves the binary in the repository root +as `./ops`. Nothing else is needed to start — no prerequisites to install, no configuration +to write. + +Now ask it where it is: + +``` +./ops -info +``` + +This prints the environment the CLI resolved for itself — which task files it will run, +where it caches things, which version it is. It is the first thing to check whenever +something behaves unexpectedly, and it is explained field by field in +[How `ops` finds its tasks](#how-ops-finds-its-tasks). + +Then ask it what it can do: + +``` +./ops -t +``` + +**The first run does real work.** `ops` ships with no task files and no external tools, so +before it can list anything it: + +1. **downloads the tasks** — a git clone of `$OPS_REPO` + (default `apache/openserverless-task`) into `~/.ops/$OPS_BRANCH/oplugins`, unless it + finds them locally first. You will see `Cloning tasks...` / `Tasks downloaded + successfully`. In this repository the checked-out [`oplugins/`](oplugins/) is used + instead, so nothing is downloaded. +2. **downloads the prerequisites** — the external binaries the tasks declare in + `prereq.yml`, cached under `~/.ops/<os>-<arch>/bin`: `kubectl`, `helm`, `kind`, `k3sup`, + `yq`, `rg`, `uv`, `bun`, `7zz`, `coreutils`. They are fetched once, on demand, and pinned + by version. + +Expect the first `ops -t` to pause while that happens; later runs are immediate. When it +finishes you get the banner and the list of top-level tasks: + +``` +$ ./ops -t +Welcome to ops, the all-mighty, extensibile apache OPenServerless CLI Tool. +... +***** +Tasks +***** + +Type ops <task> to see usage and subtasks. + +----------------------------------- +OpenServerless Administration Tasks +----------------------------------- + + admin Manage additional users in OpenServerless + config Manage the Apache OpenServerless configuration + setup Setup the Apache OpenServerless platform on multiple environments + debug Debug utilities for the Apache OpenServerless platform + cloud OpenServerless setup utilities for supported Deployment models on Cloud Providers + util Utilities + ... +``` + +`ops <task>` with no arguments shows that task's own usage and subtasks, so you can explore +downwards from here. + +If a download goes wrong, `ops -update` re-fetches the tasks and prerequisites, and +`ops -reset` clears `~/.ops` entirely so the next run starts from scratch. + +## The CLI + +The system is controlled by `ops`, a single self-contained Go binary. It is essentially a +**task executor**: it is based on [Task](https://taskfile.dev) but extends it in ways that +turn it into a complete portable scripting environment. + +### It is Task, extended + +1. **It downloads prerequisite binaries.** Task files declare what they need in a + `prereq.yml`; `ops` fetches and caches the missing binaries on first use (see + [`cli/prereq.go`](cli/prereq.go)). You do not have to install them yourself. + +2. **It embeds tools.** `wsk`, `wskdeploy`, `jq`, `sh`, `awk`, `envsubst`, `task` and a set + of small purpose-built helpers are compiled into the binary itself (see + [`cli/_patches/`](cli/_patches/) and [`cli/tools/`](cli/tools/)). They are invoked with a + `-` prefix and need no download: + + ``` + ops -wsk action list + ops -jq . file.json + ops -h # list all the embedded tools + ``` + +The result is that a single binary gives you a complete, portable scripting environment — +the same on Linux, macOS and Windows. + +### General syntax + +``` +ops -<tool> <args>... # run an embedded tool +ops <task> <args>... # run a task from the task files +``` + +Essential flags: + +``` +-h | -help list embedded tools +-v | -version current version (always mention it when asking for help) +-t | -tasks list top level tasks (downloads them if needed) +-i | -info show the CLI environment +-u | -update download the latest tasks and prerequisites +-c | -config manage the server configuration +-l | -login access the system +-reset clean the downloads (if nothing works, try this) +``` + +### How `ops` finds its tasks + +When you run a task, `ops` locates its **root** — the directory holding `opsfile.yml` and +`opsroot.json` — in this order (see `locateOpsRoot` in +[`cli/prepare.go:171`](cli/prepare.go#L171)): + +1. `$OPS_ROOT`, if set. +2. Searching **upwards** from the current directory for a folder containing both + `opsfile.yml` and `opsroot.json`. +3. A `oplugins` (or `olaris`) subfolder **of the current directory**. This is what happens + when you work in this repository: the checked-out [`oplugins/`](oplugins/) is used + directly. +4. `~/.ops/$OPS_BRANCH/oplugins`. If it is not there, `ops` **downloads** it with a git + clone from `$OPS_REPO` (default `http://github.com/apache/openserverless-task`) at the + current branch. + +So in normal use `ops` self-provisions its task files; in development it picks up your +local `oplugins` instead. Use `ops -info` to know which values are actually in effect: + +``` +$ ops -info +OPS & OPS_CMD: /Users/me/Projects/openserverless/ops +OPS_VERSION: 0.9.0-2609031223.SNAPSHOT +OPS_BRANCH: +OPS_BIN: /Users/me/.ops/darwin-arm64/bin +OPS_TMP: /Users/me/.ops/tmp +OPS_HOME: /Users/me/.ops +OPS_ROOT: /Users/me/Projects/openserverless/oplugins +OPS_REPO: http://github.com/apache/openserverless-task +OPS_PWD: /Users/me/Projects/openserverless +OPS_TASKS: 7e523adb9af3f17ae221220e9c17bce31fd92090 +OPS_ROOT_PLUGIN: /Users/me/Projects/openserverless +``` + +`OPS_ROOT` pointing inside your working copy is the sign that you are running against your +local task files. + +## The configuration files + +### `opsroot.json` + +[`oplugins/opsroot.json`](oplugins/opsroot.json) is **the glue of the whole system**. It +declares which images to deploy and other information; `.config.images` is the part that +matters most during development: + +```json +{ + "config": { + "images": { + "operator": "docker.io/apache/openserverless-operator:0.9.0-incubating.26i08j16-snapshot", + "controller": "docker.io/apache/openserverless-wsk-controller:0.9.0-incubating.26i07r51-snapshot", + "invoker": "docker.io/apache/openserverless-wsk-invoker:0.9.0-incubating.26i07r51-snapshot", + "streamer": "docker.io/apache/openserverless-streamer:...", + "systemapi": "docker.io/apache/openserverless-admin-api:...", + "devcontainer":"docker.io/apache/openserverless-devcontainer:...", + "couchdb": "docker.io/apache/couchdb:2.3", + "redis": "docker.io/bitnamilegacy/redis:8.2.1", + "...": "... third-party service images ..." + }, + "ops": { } + }, + "version": "..." +} +``` + +#### `.config` is a source of environment variables + +The `config` object is not read by the tasks as JSON. **It is flattened into environment +variables** that are then passed to every task. The rule (see `flatten` in +[`cli/config/config_map.go:156`](cli/config/config_map.go#L156)) is: walk the tree, join the +keys with `_`, uppercase the result. So + +``` +.config.images.operator → IMAGES_OPERATOR +``` + +and a task can simply use `$IMAGES_OPERATOR`. `ops -config -d` shows the whole set — see +[Inspecting and editing the configuration](#inspecting-and-editing-the-configuration-ops--config) +below. + +This is why `opsroot.json` is the glue: changing an image there changes what every task +deploys, with no code change anywhere. + +The flattening merges three sources, in increasing order of precedence +([`cli/main.go:460`](cli/main.go#L460)): + +1. `opsroot.json` from the task root — the defaults shipped with the release; +2. `config.json` — the user's own configuration, written by `ops -config`; +3. the `opsroot.json` of any installed plugin, under its own key (a plugin whose name + collides with an existing key is ignored, with a warning). + +#### Inspecting and editing the configuration: `ops -config` + +`ops -config` is how you read and change that merged configuration without editing JSON by +hand ([`cli/config/config_tool.go`](cli/config/config_tool.go)). + +**Dump every variable** — `-d` (or `--dump`) prints the whole flattened set, exactly as the +tasks will see it. This is the quickest way to answer "what is this task actually going to +use?": + +``` +$ ops -config -d +IMAGES_OPERATOR=docker.io/apache/openserverless-operator:0.9.0-incubating.26i08j16-snapshot +IMAGES_CONTROLLER=docker.io/apache/openserverless-wsk-controller:0.9.0-incubating.26i07r51-snapshot +OPERATOR_CONFIG_APIHOST=miniops.me +OPERATOR_COMPONENT_POSTGRES=true +POSTGRES_CONFIG_REPLICAS=1 +... +``` + +> The dump includes the generated passwords and API keys (`SECRET_*`, +> `REGISTRY_CONFIG_SECRET_*`). Redact it before pasting into a bug report. + +**Read one value** — pass a bare key, with no `=`. Several keys print several values: + +``` +$ ops -config IMAGES_OPERATOR +docker.io/apache/openserverless-operator:0.9.0-incubating.26i08j16-snapshot +``` + +**Set a value** — pass `KEY=VALUE`. Several pairs at once are fine: + +``` +ops -config OPERATOR_CONFIG_APIHOST=my.host.example +ops -config IMAGES_STREAMER=ghcr.io/me/openserverless-streamer:mytag POSTGRES_CONFIG_REPLICAS=3 +``` + +**Remove a value** — `-r` (or `--remove`) with the keys to drop: + +``` +ops -config -r POSTGRES_CONFIG_REPLICAS +``` + +All writes go to `config.json` in `$OPS_HOME` (`~/.ops/config.json`), which is layer 2 of +the merge above — so a value you set there **overrides** `opsroot.json` and survives the +task files being re-downloaded. + +That layering has two consequences worth knowing: + +- **`-r` removes a key from `config.json` only.** It cannot delete anything that came from + `opsroot.json`; removing your override just lets the underlying default show through + again. Removing a key that only exists in `opsroot.json` fails with + `invalid key: '<KEY>' - key does not exist in config.json`. +- **You cannot blank a value.** `ops -config -h` claims that `KEY=""` disables an + `opsroot.json` value, but the parser rejects an empty right-hand side + ([`cli/config/config_tool.go:161`](cli/config/config_tool.go#L161)) and the command fails + with `invalid key-value pair: "KEY="`. To neutralise an inherited value today, set it to + something the task treats as off (`false`, `0`) — or edit `oplugins/opsroot.json` + directly, as described under [Overriding a single image](#overriding-a-single-image). + +#### `.version` is the required CLI version + +The top-level `version` field is **the version of `ops` that these task files require**. On +every run the CLI compares its own version against it as semver +([`cli/prepare.go:129`](cli/prepare.go#L129)). If `ops` is older, it says so and updates +itself automatically: + +``` +Your ops version (0.9.0) is older than the required version (0.9.1). +``` + +In development your CLI version is usually not valid semver (e.g. +`0.9.0-2609031223.SNAPSHOT`), so the check is skipped with a warning and you are never +force-updated out of your own build. + +### `runtimes.json` + +[`oplugins/runtimes.json`](oplugins/runtimes.json) points to the runtime images to use. It +is **a standard OpenWhisk runtimes manifest** — the same format OpenWhisk itself consumes, +with `runtimes`, `blackboxes` and `description` at the top level: + +```json +{ + "runtimes": { + "nodejs": [ + { + "kind": "nodejs:24", + "default": false, + "image": { + "prefix": "docker.io/apache", + "name": "openserverless-runtime-nodejs", + "tag": "v24-26i07r17-snapshot" + }, + "deprecated": false, + "attached": { "attachmentName": "codefile", "attachmentType": "text/plain" } + } + ], + "python": [ ], "go": [ ], "java": [ ] + }, + "blackboxes": [ ], + "description": "..." +} +``` + +It is generated in the [`runtimes/`](runtimes/) repository and copied over by +`task sync-runtimes`. Because it is the standard format, the CLI also hands it to the +embedded `wsk` through `WSK_RUNTIMES_JSON` ([`cli/main.go:291`](cli/main.go#L291)). + +## How to build + +The system runs on Kubernetes and can be deployed in many environments — most notably k3s, +microk8s, and cloud offerings like Amazon EKS, Azure AKS and Google GKE. For development +you can use Docker, and OpenServerless will install `kind` for you. + +Build everything locally with: + +``` +task build +``` + +This builds all the required images. It runs, in order (see +[`Taskfile.yml:231`](Taskfile.yml#L231)): + +| Step | What it does | +|------|--------------| +| `gitinit` | initializes git in the subrepos if missing (tags are the build's currency, so a repo without git cannot be built) | +| `opslink` | builds `ops`, symlinks it into `~/.local/bin`, links `oplugins` into `~/.ops/$OPS_BRANCH` | +| `openwhisk` | builds the controller and invoker images | +| `operator` | builds the operator image | +| `runtimes` | builds `runtimes.json` | +| `streamer` | builds the streamer image | +| `admin-api` | builds the admin-api image | +| `opsroot` | **rewrites `oplugins/opsroot.json` with the tags just built** | + +You compile `ops` first, then `ops` builds the system. + +### Everything is driven by tags + +**All the compilations are driven by the git tag** of each component repository. A build +does not take a version from a variable — it reads the tag out of git, tags the image with +it, and writes that same tag into `opsroot.json`. There are two kinds of tag, for two +different purposes. + +**There is no way around this, not even the manual runs.** Starting a workflow by hand from +the GitHub Actions UI (`workflow_dispatch`) looks like it bypasses tagging, but under the +hood it does exactly what you would have done locally: the job runs `task tag` and +`task ci` on the runner, and the following step reads the tag back with +`git describe --tags --abbrev=0` to name the image +([`oplugins-op/.github/workflows/image.yml:54`](oplugins-op/.github/workflows/image.yml#L54)). +So **every image that exists corresponds to a tag** — there is no such thing as an untagged +build, and given an image tag you can always find the commit it was built from. + +#### Development tags: the operator ref + +The main build derives its tag from the **git ref of the operator** +([`Taskfile.yml:25`](Taskfile.yml#L25)): + +```yaml +RELEASE: + sh: git -C oplugins-op rev-parse --short HEAD +OPERATOR_TAG: "{{.RELEASE}}" +``` + +Every other component inherits it — `STREAMER_TAG`, `ADMIN_API_TAG`, `OPENWHISK_TAG`, +`DEVCONTAINER_TAG` and `OPS_BRANCH` are all `{{.OPERATOR_TAG}}`. One short commit hash +labels the whole build, so a local build is always internally consistent. + +**These tags are meant for private development.** They never leave your machine: the images +are saved locally (under `~/.ops/<os>-<arch>/images/kind/`) and loaded into `kind`. + +#### Publication tags: the timestamp + +To publish images you define a tag explicitly with `task tag`, in the component's own +directory. These are the tags meant to publish images, and they are covered in full in +[Publishing images](#publishing-images) below. + +### Keeping `opsroot.json` in sync + +Two tasks write `opsroot.json`, and it is worth knowing which one you are running: + +- **`task opsroot`** (part of `task build`) writes the tags of the images **you just built + locally** — the operator-ref tags. +- **`task sync-opsroot`** writes the **latest published tag of each component**, read from + git: + + ```yaml + OPENWHISK_TAG: + sh: git -C build for-each-ref --sort=-creatordate --count=1 --format='%(refname:short)' refs/tags + ``` + +Both keep a pristine copy in `opsroot.orig` on first run and regenerate from it, so +repeated runs are idempotent. `task sync-images` runs the runtimes and opsroot syncs +together. + +> **Gotcha.** `sync-opsroot` reads whatever tags are **local**. A freshly cloned subrepo has +> no tags, `for-each-ref` returns the empty string, and you silently get a tagless image +> reference such as `docker.io/apache/openserverless-wsk-controller:`. If images fail to +> pull with an empty tag, run `git fetch --tags` in the offending subdirectory and re-run +> the task. + +## Developing against a local build + +In development you set `OPS_ROOT` temporarily to your local task files, so that `ops` uses +the `oplugins` in your working copy rather than the downloaded ones. This is what +[`Taskfile.yml`](Taskfile.yml) does when running the tests: + +```bash +export PATH="$PWD:$PATH" +export OPS_ROOT="$PWD/oplugins" +export OPS_BRANCH="<the branch>" +which ops +ops -info +``` + +Always confirm with `ops -info` that `OPS_ROOT` and `OPS_BIN` are what you expect before +concluding that a change did not work. + +`task opslink` makes this semi-permanent: it symlinks `./ops` into `~/.local/bin` and your +`oplugins` into `~/.ops/$OPS_BRANCH/oplugins`, and appends `OPS_BRANCH` and the `PATH` +entry to `~/.profile`. + +### The development loop, end to end + +Building for development means producing images that exist **only on your machine**, tagged +with your operator ref, and pointing a local `kind` cluster at them. The full cycle: + +```bash +# 1. build everything: ops, the images, and a matching opsroot.json +task build + +# 2. make ops and your local task files the ones in use +export PATH="$PWD:$PATH" +export OPS_ROOT="$PWD/oplugins" + +# 3. verify you are running against your own build +ops -info # OPS_ROOT must point inside your working copy +ops -t # list the tasks that will run + +# 4. create the cluster and deploy +ops setup docker +``` + +The step that ties it together is invisible unless you go looking for it. Because these +images are never pushed to a registry, `task build` **saves each one to disk** as it is +built — `task image-save` writes it to +`~/.ops/<os>-<arch>/images/kind/<base64-of-the-image-name>` +([`Taskfile.yml:75`](Taskfile.yml#L75)). Later, `ops setup docker` calls +`ops util freeze kind-load`, which walks that directory, decodes each filename back into an +image name, and imports the layers straight into the kind node's containerd +([`oplugins/util/freeze/opsfile.yml:176`](oplugins/util/freeze/opsfile.yml#L176)). + +So the chain from a code change to a running pod is: + +``` +your commit → operator ref = the tag → images built with that tag + → saved under ~/.ops/.../images/kind/ + → opsroot.json rewritten with that tag (task opsroot) + → IMAGES_* env vars (the CLI, on every run) + → kind-load imports them into the cluster (ops setup docker) + → the manifests reference exactly those images +``` + +Nothing is pulled from Docker Hub, because the tag in `opsroot.json` matches an image +already present in the node. + +### Iterating on a single component + +A full `task build` is rarely what you want after the first one. Each component has its own +task, so rebuild just the one you touched and redeploy: + +```bash +task streamer # or: operator, admin-api, openwhisk, cli, runtimes +task opsroot # rewrite opsroot.json with the new tags +ops setup docker # reload the images and roll out +``` + +Two things to keep in mind: + +- **`task cli` is a no-op once `./ops` exists** — it carries `status: ! test -e ../ops` + ([`Taskfile.yml:95`](Taskfile.yml#L95)), and `task opslink` and `task build` both depend + on it, so neither will rebuild the binary either. After changing the CLI, `rm ./ops` (or + `task clean`) before rebuilding, or you will keep running the old one. `ops -info` + showing a stale `OPS_VERSION` is the tell. +- **The tag only changes when the operator ref changes.** `RELEASE` is + `git -C oplugins-op rev-parse --short HEAD`, so rebuilding the streamer without + committing in `oplugins-op` produces an image with the *same* tag as before. That is + usually what you want — but it means kind may keep the previously imported layers, so + delete the cluster (`ops setup docker delete`) if a change stubbornly refuses to appear. + +### Overriding a single image + +You do not have to rebuild anything to try a different image. Because `.config` is just a +source of env vars, editing `oplugins/opsroot.json` is enough: + +```bash +jq '.config.images.streamer = "docker.io/apache/openserverless-streamer:some-other-tag"' \ + oplugins/opsroot.json > tmp && mv tmp oplugins/opsroot.json +ops setup docker +``` + +`task opsroot` and `task sync-opsroot` will overwrite this on their next run — they +regenerate from `opsroot.orig` — so treat it as a temporary probe, not a durable change. + +### Starting from a clean slate + +When the state gets confusing, the reset ladder, from cheapest to most drastic: + +```bash +ops setup docker delete # destroy the kind cluster +task clean # remove ./ops and the cached kind images +ops -reset # remove ~/.ops entirely (asks for confirmation) +ops -update # re-download tasks and prerequisites +``` + +## Publishing images + +Everything above builds on your machine and stays there. Publishing is a different flow, +and it is governed by one rule: + +> **All CI builds are triggered by pushing a git tag.** No branch push ever builds an +> image. Every component workflow is declared `on: push: tags:` with `branches-ignore: '*'` +> — the tag is not metadata attached to a build, the tag **is** the trigger, and its text +> **is** the image tag. + +### The three tasks + +Every component — the operator, admin-api, streamer, devcontainer, the OpenWhisk build, the +runtimes and the CLI — exposes the same trio, with the same meaning. You use them in this +order: + +| Task | Runs where | Effect | Pushes? | +|------|------------|--------|---------| +| `task tag` | local | creates (force-replacing) the git tag that names this build | no | +| `task build` | local | builds using the current tag, into the local Docker daemon | no | +| `task ci` | local, triggers remote | pushes the tag, which starts the GitHub Actions build | yes, via CI | + +Run them **in the component's own directory**, not in the root — each subrepo has its own +`Taskfile.yml` and its own workflow. + +Two things newcomers consistently get wrong, so they are worth stating outright: + +- **`task tag` deletes the existing local tags** before creating the new one. This is + deliberate — snapshot tags are disposable — but it is destructive, and it is not what + `git tag` normally does. +- **`task ci` does not build anything.** It runs `git push --tags`. The build happens in + GitHub Actions *on the repository that received the tag*. This is the whole reason + forking matters: the tag lands in **your** fork, so **your** Actions run, with **your** + credentials, pushing to **your** registry. + +There is also a manual path: every workflow declares `workflow_dispatch`, so it can be +started from the GitHub Actions UI, where it creates the tag itself. Use it when you cannot +or do not want to push tags from the command line. + +The CLI is the one component with a wrinkle: `task ci` (alias `task trigger`) refuses to +run on a dirty tree (`git diff --exit-code`), and its `task tag` **creates a commit** — it +writes `version.txt` and `branch.txt` and commits them, because the CLI embeds its own +version string. Every other component only moves a tag. + +### The tag format + +Tags are timestamps in a compacted, sortable form. The encoding is uniform across all +components, and it is worth learning because the tags are otherwise unreadable: + +``` +0.9.0-incubating.26i07r51-snapshot +└────┬─────────┘ ││││││└┬┘└───┬───┘ + BASETAG ││││││ │ └───── SUFFIX: -snapshot marks a non-release build + ││││││ └─────────── minute + │││││└───────────── hour, a letter: a=00 … x=23 + │││└┴────────────── day of month + ││└──────────────── month, a letter: a=January … l=December + └┴───────────────── year, two digits +``` + +So `26i07r51` is 2026, September (`i` = 9th letter), day 07, hour 17 (`r` = 18th letter, +zero-based → 17), minute 51. + +Because the encoding is lexicographically sortable and the tags are created in order, +"the newest tag" and "the latest build" are the same thing — which is exactly what +`task sync-opsroot` relies on when it picks a tag with +`git for-each-ref --sort=-creatordate --count=1`. + +The **runtimes** are the exception: their tag carries a prefix naming the family to build, +and CI parses it to decide what work to do (see below). + +### Where the images go + +The destination registry and the credentials are **derived automatically from the +repository the tag was pushed to**. In the normal case there is nothing to configure: + +| Tag pushed to | Registry | Namespace | Credentials | +|---------------|----------|-----------|-------------| +| the Apache repository | `docker.io` | `apache` | secrets `DOCKERHUB_USER` / `DOCKERHUB_TOKEN` | +| any fork | `ghcr.io` | the fork owner's GitHub user | the current GitHub actor + the built-in `GITHUB_TOKEN` | + +So a contributor who forks and pushes a tag gets images at `ghcr.io/<their-user>/…` with +**no secrets to create and no `.env` to write** — the workflow authenticates as the user +running it, with the token GitHub hands to every Actions run. + +To send images somewhere else, set the `DOCKERHUB_REGISTRY` secret together with +`DOCKERHUB_USER` and `DOCKERHUB_TOKEN`. That is an override for people who want a specific +destination, not a setup step. + +#### The exception: repositories that publish other repositories' images + +Two repositories publish images whose names do **not** match the repository itself: + +| Repository | Images published | +|------------|------------------| +| `openserverless-build` | `openwhisk2/controller`, `openwhisk2/invoker`, `openwhisk2/scheduler`, `openwhisk2/standalone`, `openwhisk2/scala` | +| `openserverless-runtimes` | `openserverless-runtime-<rt>` — one per language version | + +The built-in `GITHUB_TOKEN` is scoped to packages **owned by the repository running the +workflow**: GHCR links a package to its creating repository on first push and grants write +access only to that repository. Since these two publish under names that do not match, +`GITHUB_TOKEN` carries no permission and the push is refused. + +For these two repositories only, when pushing to ghcr.io, a **`GHCR_TOKEN`** secret must be +set on the fork — a classic Personal Access Token with the **`write:packages`** scope: + +1. <https://github.com/settings/tokens> → *Generate new token (classic)*. +2. Select **`write:packages`** (this implies `read:packages` and `repo`). +3. In your fork: *Settings → Secrets and variables → Actions → New repository secret*, + named exactly **`GHCR_TOKEN`**. + +When pushing to Docker Hub instead, the equivalent is a `DOCKERHUB_TOKEN` with write +permission on the target images. + +The symptom when this is missing is `denied: permission_denied: write_package` at push +time, after a long and otherwise successful build. Both workflows now **fail fast** with +these instructions in the job summary rather than building first and failing at the end. + +### Naming your local build + +Locally no registry is involved: each component builds under the same default image name it +would use in CI, and the result exists only in your Docker daemon. To name it for your own +registry instead, export `MY_<COMPONENT>_IMAGE` — the **full image name without the tag**; +the tag is still computed as above. Each component ships a `.env.dist` to copy to `.env` +(which stays untracked): + +| Component | Variable | +|-----------|----------| +| operator | `MY_OPERATOR_IMAGE` | +| streamer | `MY_STREAMER_IMAGE` | +| admin-api | `MY_ADMINAPI_IMAGE` | +| devcontainer | `MY_DEVCONTAINER_IMAGE` | +| OpenWhisk build | `MY_CONTROLLER_IMAGE`, `MY_INVOKER_IMAGE`, `MY_SCHEDULER_IMAGE`, `MY_STANDALONE_IMAGE` | + +### What each component publishes + +The trio above is identical everywhere; what differs is what one tag produces. + +**One tag → one image** — `operator`, `admin-api`, `streamer`, `devcontainer`. The plain +case: + +```bash +cd streamer +task tag # 0.9.0-incubating.26i07r51-snapshot +task build # local image, nothing pushed +task ci # push the tag → CI builds linux/amd64 + linux/arm64 and pushes +``` + +CI also runs the license check (skywalking-eyes) and the unit tests **before** building the +image, so a red build is not necessarily an image problem. + +**One tag → a family of images** — `runtimes`. The tag selects the work: + +```bash +cd runtimes +task tag RT=python # python_26i07r51-snapshot +task tag # RT defaults to "all" +``` + +CI parses the prefix (`all`, `common`, `experimental`, or a single language) and builds +only that family, producing one image per language version: + +``` +<registry>/<namespace>/openserverless-runtime-<rt>:<ver>-<tag> +``` + +Every runtime image is `FROM` the **common base**, so the common job runs first and the +language matrix depends on it; building a language family against a missing or mismatched +base is a failure mode, not a bug. This repository needs `GHCR_TOKEN` (see above). The +resulting `runtimes.json` is what reaches an installation, via `task sync-runtimes`. + +**One tag → binaries, no image** — `cli`. `task build` compiles `ops` into the repository +root and is all you need locally; tagging matters only when the embedded version string +does. `task ci` tags, commits, and pushes, and the release workflow publishes binaries for +every supported platform. + +**The OpenWhisk images** — `build`. This one does not follow the pattern of the others and +deserves its own note: + +- It vendors an OpenWhisk source tree and builds through a devcontainer rather than on the + host, because the OpenWhisk Gradle build **requires Java 11**. On a host with a newer JDK + it fails with `Unsupported class file major version`. Run it inside + `ops ide devcontainer`. +- Its `task build` does the whole local round trip: build the images, update + `opsroot.json`, load into kind, roll out the controller. +- **The root `Taskfile.yml` and `build/Taskfile.yml` are separate build systems.** This is + the single most common wrong assumption. The root `task build` does invoke `task + openwhisk`, but the two Taskfiles maintain their own tags and their own `opsroot.json` + edits — when in doubt about which controller image is deployed, read + `.config.images.controller` rather than inferring it. + +### Consuming a build you published + +Once CI has pushed your images, point an installation at them by editing the image +references in `opsroot.json` — either your working copy's `oplugins/opsroot.json`, or +`~/.ops/$OPS_BRANCH/oplugins/opsroot.json` for an installed CLI: + +```bash +jq '.config.images.streamer = "ghcr.io/<your-user>/openserverless-streamer:<your-tag>"' \ + oplugins/opsroot.json > tmp && mv tmp oplugins/opsroot.json +ops setup docker +``` + +Runtimes are overridden the same way through `runtimes.json`. + +Two failure modes worth recognising: + +- **Locally built images are not automatically visible to kind.** If you skip the + `kind-load` step described earlier, the pod silently pulls the old upstream image + instead. The symptom is a deployment that succeeds while running the wrong code. +- **A change visible in the custom resource is not proof it reached the workload.** Confirm + at the pod level: + + ```bash + kubectl -n openserverless get pods -o jsonpath='{.items[*].spec.containers[*].image}' + ``` + +## Testing + +Run the whole suite (defaults to `kind`): + +``` +task test +``` + +Run a single test — with an empty or unknown `TEST` it lists the available ones: + +``` +task test-one TEST=<name>.sh +task test-one TEST= +``` + +Both live in [`testing/tests/`](testing/tests/) and export `OPS_ROOT` to the local +`oplugins`, so they always exercise your working copy. + +## Other useful tasks + +| Task | Purpose | +|------|---------| +| `task` | list all the tasks and print the current release | +| `task clean` | remove `./ops` and the cached kind images | +| `task license` | check the license headers with `license-eye` (`CMD=fix` to fix them) | +| `task sync-images` | refresh `runtimes.json` and `opsroot.json` from the latest published tags | +| `task release VER=<v> KEY=<gpg-key>` | build the signed source release tarball; the source must be in a version folder and you need a GPG key to sign | + +See also [VERIFY.md](VERIFY.md) for verifying a release, and +[CONTRIBUTING.md](CONTRIBUTING.md). diff --git a/README.md b/README.md index ba14dc3..1018054 100644 --- a/README.md +++ b/README.md @@ -2,16 +2,28 @@ Welcome to [Apache OpenServerless](https://openserverless.apache.org), an incubating project at the [Apache Software Foundation](https://www.apache.org). -- If you want to **install** Apache OpenServerless in the cloud go [here](https://openserverless.apache.org/docs/installation/). +- If you want to **install** Apache OpenServerless go [here](https://openserverless.apache.org/docs/installation/). - If you want to **understand** what this project is check the [original proposal](https://cwiki.apache.org/confluence/display/INCUBATOR/OpenServerlessProposal). - If you want to **discuss** with us, join our mailing list by sending an email to `[email protected]` -- If you want to **contribute** to the project, read [this guide](CONTRIBUTING.md) +- If you want to **contribute** to the project, read [this the contribution guide](CONTRIBUTING.md) and the [developer guide](DEVEL.md). -## Build and test from sources + +> [!IMPORTANT] +> **The rest of this document covers building Apache OpenServerless from source, +> for developers of the project itself.** +> +> If you are an end user and just want to *use* OpenServerless, you do not need +> any of this — start from the +> [installation guide](https://openserverless.apache.org/docs/installation/) +> instead. It walks you through installing the `ops` CLI and creating a local +> OpenServerless with `ops setup mini`. > [!WARNING] > Building from the latest sources in git is **not recommended for production > use**. The `main` branch may contain unstable, untested, or incomplete > changes. For production deployments, use an official release tarball instead. +## Build and test from sources + + Download a release tarball from the Apache distribution area, untar and cd to it: ``` @@ -70,6 +82,9 @@ You have to install [lima](https://lima-vm.io) (example: `brew install lima`). E `./build-and-test-mac.sh` +Each script builds and tests inside a disposable Linux VM or WSL distribution, so +the build never touches your host toolchain. + ## Development setup ### Prerequisites @@ -140,6 +155,9 @@ cd openserverless ``` To work on a release branch instead of `main`, add `--branch <branch>`. +Remember that most of the code lives in git submodules, so `--recurse-submodules` +is required; if you already cloned without it, run +`git submodule update --init --recursive`. You can build with: `task build` @@ -149,6 +167,10 @@ You can check all the files have the license header: `task license` Read the task files (that are basically shell scripts wrapped in a yaml environment) to learn all the build procedures. +For a guided tour of how the system is put together — the cli, the `opsroot.json` +configuration, how tags drive the builds, the development loop and how images are +published — read the [developer guide](DEVEL.md). + ## License Apache OpenServerless is licensed under the [Apache License, Version 2.0](LICENSE). diff --git a/build b/build index bf99b08..65b9348 160000 --- a/build +++ b/build @@ -1 +1 @@ -Subproject commit bf99b0878cb0755f27b223bda7e8b084f286ab5d +Subproject commit 65b9348cd8c30dec3a7ea21b08966a528cde0df6 diff --git a/site b/site new file mode 160000 index 0000000..5011a9e --- /dev/null +++ b/site @@ -0,0 +1 @@ +Subproject commit 5011a9e394ea19f597f064f69187619fc1019dfe diff --git a/streamer b/streamer index 65a1451..efac38f 160000 --- a/streamer +++ b/streamer @@ -1 +1 @@ -Subproject commit 65a1451eac18d019c0999c90f8095fa880d935c8 +Subproject commit efac38f5297c8a30b9019b6ed316cff3a4c038c9
