This is an automated email from the ASF dual-hosted git repository. sciabarracom pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/openserverless.git
commit ade3ddef1a4ae351e91ff3a2049b8b5ac82fd8da Author: Michele Sciabarra <[email protected]> AuthorDate: Thu Sep 10 12:21:28 2026 +0200 adding syncimages to update references --- syncimages.py | 465 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 465 insertions(+) diff --git a/syncimages.py b/syncimages.py new file mode 100755 index 0000000..68c69cc --- /dev/null +++ b/syncimages.py @@ -0,0 +1,465 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +r"""Point oplugins/runtimes.json and oplugins/opsroot.json at the latest images. + +The remote counterpart of `task sync-opsroot` / `task sync-runtimes`: those +read whatever tags happen to be **local** in the subrepos, and so hit the +gotcha documented in DEVEL.md - a freshly cloned subrepo has no tags and +you silently get a tagless `...openserverless-wsk-controller:` reference. +This reads the tags from the remotes instead, so a clone without tags and +a clone that never fetched give the same, correct answer. + +Takes a `<user>` (default `apache`), which selects both where to look for +the tags and what registry the images are pulled from. This mirrors the +publication rule of DEVEL.md ("Where the images go"): a tag pushed to the +Apache repository builds to `docker.io/apache`, a tag pushed to any fork +builds to `ghcr.io/<the fork owner>`. So a contributor syncs their own +published builds with + + ./syncimages.py myuser + +Tags are read with `git ls-remote` - nothing is cloned, and no credentials +are needed for public repos. + +**Nothing is written unless you pass `--write=yes`.** The default run reports +the images it resolved, each line prefixed `[dry-run]`, and exits without +touching the files. This is deliberate: the tags come from whatever has +been published at this moment, so you want to see what you are about to +pin before you pin it. + +Both arguments tolerate an empty value, so a task file can pass them +through without having to branch on whether they are set - `{{.USER}}` +and `--write={{.WRITE}}` interpolate to the empty string when undefined: + + ./syncimages.py # dry run, apache + ./syncimages.py "" --write= # the same, spelled by a task file + ./syncimages.py myuser --write=yes + +`--write` also accepts a bare `--write` (meaning yes), and no/false/0/off +as well as yes/true/1/on. Anything else is an error rather than a silent +no, because writing is the half of this script that changes your files. + +A repo that cannot be read, or that carries no publication tag, is +**warned about and skipped**, not fatal - a fork usually has only some of +the components, and the useful answer is the tags of the ones it does +have. Those images keep whatever they already reference, every warning is +repeated in a summary at the end, and the exit status is non-zero so a +script chaining this notices that the sync was partial. + +# The tag format + +Per DEVEL.md ("The tag format"), every component tags a publication build +with a timestamp in a compacted, sortable form: + + 0.9.0-incubating.26i07r51-snapshot + └────┬─────────┘ ││││││└┬┘└───┬───┘ + BASETAG ││││││ │ └───── SUFFIX: -snapshot marks a non-release build + ││││││ └─────────── minute + │││││└───────────── hour, a letter: a=00 … x=23 + │││└┴────────────── day of month + ││└──────────────── month, a letter: a=January … l=December + └┴───────────────── year, two digits + +so `26i07r51` is 2026, September (`i`, 9th letter), day 07, hour 17 (`r`, +18th letter, zero-based), minute 51. + +DEVEL.md notes the encoding is lexicographically sortable, which is what +makes `--sort=-creatordate` and "the newest tag" agree. We do not rely on +that: `ls-remote` gives no dates to sort by, and the repos still carry +older tags in a plain-decimal `YYMMDDHHMM` form that does *not* sort +against the letter form. So the timestamp is decoded into its real date +before comparing, both forms are understood, and a tag that decodes as +neither is skipped rather than guessed at. + +# runtimes.json + +The runtimes are the exception DEVEL.md calls out: their tag carries a +prefix naming the family to build, which CI parses to decide what work to +do. They are built and published together under a single `all_<timestamp>` +tag, so one tag drives every entry. Each `image` subsection becomes + + { + "prefix": "<registry>/<user>", + "name": "openserverless-runtime-java", + "tag": "<version>-<timestamp>" + } + +keeping the existing `<version>-` prefix of the tag - that is the language +version (`v1.27-`, `sys-`), it names what is inside the image and is not +ours to change. + +# opsroot.json + +Each image under `config.images` that we build ourselves is named by the +whole tag of its own repo - `BASETAG.<timestamp><SUFFIX>`, exactly the +string the component was tagged with, which is what `task sync-opsroot` +writes and what makes an image traceable back to its tag: + + <origin>-operator -> operator + <origin>-admin-api -> systemapi + <origin>-build -> controller, invoker + <origin>-streamer -> streamer + <origin>-devcontainer -> devcontainer + +Only the `:<tag>` part of the value is replaced, so third-party images +(couchdb, redis, ingress, ...) keep their own registry and tag - those are +not ours to retag. + +Every resolved image is printed, one per line, as `<image>:<tag>`. +""" + +import argparse +import json +import re +import subprocess +import sys + +# month, a=January ... l=December - and hour, a=00 ... x=23 +MONTHS = "abcdefghijkl" +HOURS = "abcdefghijklmnopqrstuvwx" + +# 26i07r51, the form documented in DEVEL.md +LETTER_TS_RE = re.compile( + r"^(?P<yy>\d\d)(?P<month>[a-l])(?P<dd>\d\d)(?P<hour>[a-x])(?P<mm>\d\d)$" +) +# 2409121919, the plain-decimal form of the older tags +DIGIT_TS_RE = re.compile(r"^(?P<yy>\d\d)(?P<month>\d\d)(?P<dd>\d\d)(?P<hour>\d\d)(?P<mm>\d\d)$") + +# everything skipped, to be summarised at the end: a warning scrolled past +# half an hour ago is a warning nobody acts on +WARNINGS = [] + +# whether we are actually writing; only `report` needs to know, and +# threading it through every call site would not make it clearer +WRITING = False + +# the repos we build ourselves, and the opsroot.json keys they feed +OPSROOT_REPOS = [ + ("operator", ["operator"]), + ("admin-api", ["systemapi"]), + ("build", ["controller", "invoker"]), + ("streamer", ["streamer"]), + ("devcontainer", ["devcontainer"]), +] + + +# what counts as yes and no for --write, so that `--write=$(VAR)` works +# whatever spelling the caller's shell or task file happens to use +TRUE = ("yes", "y", "true", "1", "on") +FALSE = ("", "no", "n", "false", "0", "off") + + +def parse_flag(value): + """Read --write, rejecting anything that is not plainly yes or no. + + An unrecognised value is an error rather than a silent false: writing + is the destructive half of this script, and `--write=maybe` quietly + doing nothing is worse than a message. + """ + normalised = value.strip().lower() + if normalised in TRUE: + return True + if normalised in FALSE: + return False + raise SystemExit( + f"--write: expected one of {', '.join(TRUE)} or {', '.join(FALSE[1:])} " + f"(or empty, meaning no), not {value!r}" + ) + + +def report(path, image): + """Print a resolved image, saying plainly whether it is being written.""" + print(f"{'' if WRITING else '[dry-run] '}{path}: {image}") + + +def warn(message): + """Report something we are skipping, on stderr so stdout stays pipeable.""" + print(f"warning: {message}", file=sys.stderr) + WARNINGS.append(message) + + +def decode_timestamp(stamp): + """Return the date a build timestamp encodes, or None if it is not one. + + The result is a tuple, to be compared rather than read: it is the real + (year, month, day, hour, minute) of the build, so tags in the letter + form and in the older decimal form order correctly against each other. + + The SUFFIX (`-snapshot`, `-RC`, ...) is not part of the timestamp but + does distinguish two tags of the same minute, so it is kept as the + last element to make the order total rather than arbitrary. + """ + base, _, suffix = stamp.partition("-") + + match = LETTER_TS_RE.match(base) + if match: + month = MONTHS.index(match.group("month")) + 1 + hour = HOURS.index(match.group("hour")) + else: + match = DIGIT_TS_RE.match(base) + if not match: + return None + month = int(match.group("month")) + hour = int(match.group("hour")) + + return ( + int(match.group("yy")), + month, + int(match.group("dd")), + hour, + int(match.group("mm")), + suffix, + ) + + +def remote_tags(origin): + """The tag names of a remote repo, or None if it cannot be read. + + A repo we cannot reach is reported and skipped rather than fatal: a + fork typically has only some of the components, and the useful answer + is the tags of the ones it does have, not nothing at all. + """ + try: + out = subprocess.run( + ["git", "ls-remote", "--tags", origin], + check=True, + capture_output=True, + text=True, + ).stdout + except FileNotFoundError: + raise SystemExit("git not found in PATH: it is needed to read the remote tags") + except subprocess.CalledProcessError as exc: + # git is chatty on failure and the last line is the useful one + detail = (exc.stderr.strip().splitlines() or ["?"])[-1] + warn(f"cannot list the tags of {origin}: {detail}") + return None + + tags = set() + for line in out.splitlines(): + _, _, ref = line.partition("\t") + if ref.startswith("refs/tags/"): + # annotated tags are listed twice, the second as <tag>^{} + tags.add(ref[len("refs/tags/"):].removesuffix("^{}")) + return tags + + +def latest_tag(origin, timestamp_of): + """The most recent publication tag of `origin`, and its timestamp. + + `timestamp_of` locates the timestamp inside a tag name, and returns + None for the tags that are not publication tags at all - the runtimes + repo also carries `common1.18.4`, `experimental-...` and friends. + + Returns the tag whose timestamp is the latest, paired with that + timestamp: opsroot.json wants the whole tag, runtimes.json rebuilds + its own tags around the timestamp. Returns None, after warning, when + the repo cannot be read or carries no publication tag - the image + then keeps whatever it already references. + """ + tags = remote_tags(origin) + if tags is None: + return None + + candidates = {} + for tag in tags: + stamp = timestamp_of(tag) + if stamp and decode_timestamp(stamp): + candidates[tag] = stamp + + if not candidates: + warn( + f"no publication tag found in {origin}" + + (f" (it has {len(tags)} tag(s), none timestamped)" if tags else " (it has no tags)") + ) + return None + + tag = max(candidates, key=lambda tag: decode_timestamp(candidates[tag])) + return tag, candidates[tag] + + +def sync_runtimes(path, prefix, timestamp, write): + """Retag every runtime image in runtimes.json, keeping its version prefix.""" + if timestamp is None: + warn(f"{path} left alone: no runtimes tag to sync to") + return + + with open(path) as file: + runtimes = json.load(file) + + for kinds in runtimes["runtimes"].values(): + for kind in kinds: + image = kind["image"] + # the tag is <language version>-<timestamp>: keep the version + version, sep, _ = image["tag"].partition("-") + if not sep: + warn( + f"{path}: the tag of {image['name']} ({image['tag']}) has no " + "<version>- prefix to keep, leaving it alone" + ) + continue + image["prefix"] = prefix + image["tag"] = f"{version}-{timestamp}" + report(path, f"{image['prefix']}/{image['name']}:{image['tag']}") + + if write: + write_json(path, runtimes) + + +def sync_opsroot(path, prefix, tags, write): + """Repoint, in opsroot.json, each image we build ourselves. + + Both halves move: the tag, and the `<registry>/<user>` the image is + pulled from. A fork publishes to its own registry, so pinning its tag + onto `docker.io/apache` would name an image that does not exist. + Third-party images (couchdb, redis, ...) are not in OPSROOT_REPOS and + so keep both their registry and their tag. + """ + with open(path) as file: + opsroot = json.load(file) + + images = opsroot["config"]["images"] + for repo, keys in OPSROOT_REPOS: + tag = tags[repo] + for key in keys: + if tag is None: + # the warning naming the repo has already been issued + warn(f"{path}: config.images.{key} left at {images.get(key, '?')}") + continue + if key not in images: + warn(f"{path}: no config.images.{key} to update, skipping") + continue + image, sep, _ = images[key].rpartition(":") + if not sep: + warn( + f"{path}: config.images.{key} ({images[key]}) has no :<tag> " + "to replace, leaving it alone" + ) + continue + images[key] = f"{prefix}/{image.rpartition('/')[2]}:{tag}" + report(path, images[key]) + + if write: + write_json(path, opsroot) + + +def write_json(path, data): + """Rewrite a json file in place, preserving its indentation.""" + with open(path) as file: + # the two files do not agree on the indent, and reindenting one of + # them would bury the retag in a diff nobody can review + indent = 4 if file.read().startswith("{\n ") else 2 + + with open(path, "w") as file: + json.dump(data, file, indent=indent) + file.write("\n") + + +def main(argv=None): + parser = argparse.ArgumentParser( + description=__doc__.splitlines()[0], + epilog="By default nothing is written: the run only reports what it would do. " + "Pass --write=yes to apply it. Both arguments accept an empty value, so a " + "task file can pass them through unconditionally: an empty <user> means " + "apache, and an empty --write means no.", + ) + parser.add_argument( + "user", + nargs="?", + default="", + help="the github user and registry namespace to sync from " + "(default, and for an empty value: apache)", + ) + parser.add_argument( + "--write", + nargs="?", + const="yes", + default="", + metavar="yes|no", + help="rewrite the json files (default: dry run, write nothing). " + "Accepts yes/no/true/false/1/0, and an empty value meaning no", + ) + parser.add_argument( + "--runtimes-json", + default="oplugins/runtimes.json", + help="path to runtimes.json (default: %(default)s)", + ) + parser.add_argument( + "--opsroot-json", + default="oplugins/opsroot.json", + help="path to opsroot.json (default: %(default)s)", + ) + args = parser.parse_args(argv) + + # an empty value is how a task file passes "not set": `{{.USER}}` and + # `--write={{.WRITE}}` both interpolate to the empty string when the + # variable is undefined, and neither should be an error + user = args.user or "apache" + write = parse_flag(args.write) + + global WRITING + WRITING = write + + registry = "docker.io" if user == "apache" else "ghcr.io" + origin = f"https://github.com/{user}/openserverless"; + prefix = f"{registry}/{user}" + + # every tag is resolved before anything is written, so that a repo we + # cannot reach cannot leave the files half synced + # + # the runtimes tag is the family prefix plus the timestamp, and only + # the timestamp is reused - each entry rebuilds its own tag around it + runtimes = latest_tag( + f"{origin}-runtimes", + lambda tag: tag[len("all_"):] if tag.startswith("all_") else None, + ) + # the others are tagged BASETAG.<timestamp><SUFFIX>, and the whole tag + # is what names the image + tags = {} + for repo, _ in OPSROOT_REPOS: + found = latest_tag( + f"{origin}-{repo}", lambda tag: tag.rpartition(".")[2] or None + ) + tags[repo] = found[0] if found else None + + sync_runtimes(args.runtimes_json, prefix, runtimes[1] if runtimes else None, write) + sync_opsroot(args.opsroot_json, prefix, tags, write) + + # stdout is block-buffered when piped, so without this the stderr + # summary below would appear *before* the lines it summarises + sys.stdout.flush() + + if WARNINGS: + print( + f"\n{len(WARNINGS)} warning(s), the images above are all that could be " + "resolved:", + file=sys.stderr, + ) + for message in WARNINGS: + print(f" - {message}", file=sys.stderr) + + if not write: + print("\ndry run: nothing written, pass --write=yes to apply", file=sys.stderr) + + # something unresolved is a real failure even though we carried on, so + # a caller chaining this in a script notices + return 1 if WARNINGS else 0 + + +if __name__ == "__main__": + sys.exit(main())
