GitHub user aicam created a discussion: Proposal: Runtime images — bring your own image for a computing unit
## Summary A **runtime image** is the image a computing unit runs, built from a Dockerfile the user owns. A computing unit started from one runs that image instead of the deployment's default, so a workflow can use interpreters, system packages and compiled libraries the default image does not have. We have a working prototype and would like opinions before it goes any further — especially on the questions at the end. This is the same problem area as #7475, approached from the computing unit rather than the operator. ## The problem Today a user's only way to add a dependency is a **Python virtual environment**: a list of pip specs installed into a venv on the computing-unit image's own interpreter. Three common needs fall outside that definition, and no amount of effort within the mechanism reaches them: | Need | Why a venv cannot | An image can | | --- | --- | --- | | A different Python version | `venv` is built from the image's interpreter | `FROM` a different base | | A system package (`hmmer`, `ffmpeg`, a driver) | a venv holds Python packages only | `apt-get install` as root, at build time | | A package that compiles from source | there is no build step, only `pip install <spec>` | any `RUN` you like | A shell on a running pod does not close the gap either. That shell is the unprivileged `texera` user, so `apt install` fails, and anything it did install would live in the container's writable layer and vanish on the next restart. ### The case that forced it: AlphaFold 3 [AlphaFold 3](https://github.com/google-deepmind/alphafold3) is a sharp example because it fails a venv in all three ways at once — it needs **Python ≥ 3.12** (the engine's is 3.10), it is **not on PyPI and compiles a C++ extension**, and it calls **`jackhmmer`/`nhmmer`**, which are executables, not Python packages. This is not a hypothetical dependency. It is the shape of most scientific tooling our users bring, and each row above is a hard stop rather than an inconvenience. ## The prototype **Runtime Images** in the sidebar → **New runtime image** → the editor opens pre-filled with the computing-unit image's own Dockerfile, so the starting point is what already exists rather than a blank file. Saving starts a build; **Logs** is readable during it and after. When creating a computing unit, the runtime image is picked from a dropdown — only `READY` ones appear. ```mermaid flowchart LR D["user Dockerfile"] --> J["BuildKit Job<br/>(rootless)"] J -- "push <registry>/<id>:<n>" --> R["in-cluster registry"] R -- "kubelet pulls" --> P["computing unit pod"] ``` Four decisions worth commenting on: - **BuildKit, rootless.** Kaniko was archived in June 2025. Rootless because a build runs arbitrary user-supplied instructions, and a privileged builder would make the cluster's isolation depend on the Dockerfile being well behaved. - **The tag carries a build number.** A rebuild publishes `:n+1` rather than overwriting `:n`, so a computing unit already running an earlier build is untouched. - **Status is reconciled on read.** A build finishes on the cluster, not in the service, so a row learns its outcome when someone lists or opens it — no background threads, no leader election. - **Nothing is migrated.** Python virtual environments still exist and still work for per-UDF selection. ### It runs AlphaFold 3's data pipeline, on minikube, 2 CPU / 4 GiB, no GPU — AF3 3.0.5 built from source, calling the real `jackhmmer` through `alphafold3.data.tools.jackhmmer`, on Python **3.12.14**, from an engine whose own interpreter is 3.10: | protein | residues | msa_depth | search_seconds | | --- | --- | --- | --- | | ubiquitin | 76 | 466 | 0.075 | | NEDD8 | 81 | 459 | 0.274 | | SUMO1 | 101 | 322 | 0.297 | Flipping the UDF back to the default Python environment fails the identical workflow with `No module named 'alphafold3'`, which is the point — the selection is load-bearing, and what it selects is something no venv could have built. (Structure prediction is out of scope here: no GPU, no model parameters, no 630 GB databases. This is the dependency problem, not an AF3 port.) ## Known limits of the prototype - The registry is **plain HTTP with no authentication**; it works only because container runtimes treat the Service CIDR as insecure. - **Nothing constrains what a Dockerfile may do.** An image that omits `USER texera` runs as **root** in the pod, because the pod spec pins no `securityContext` of its own. - **No quota** on how many runtime images a user creates, how large one may be, or how long old tags are kept. - Deleting a runtime image leaves its pushed layers in the registry. ## Questions 1. **Is the computing unit the right granularity?** #7475 asks for a container *per operator*. Per-unit is much cheaper to build and reason about, but one workflow then cannot mix two incompatible dependency sets. Is that limit acceptable, or is per-operator the real target? 2. **Should users be able to name an existing image** (one on Docker Hub, or any bare image reference) instead of authoring a Dockerfile we build? That skips the builder entirely, at the cost of any control over what is inside. 3. **Who is allowed to create one?** A build is arbitrary code execution with network access on the cluster. Admins only, an allowlist, or every user with quotas? 4. **Do Python virtual environments stay?** They are strictly weaker, but simpler for the common "just add a pip package" case, and they select per-UDF rather than per-unit. 5. **Sharing.** Runtime images are currently private to their owner. Should they be shareable like datasets and workflows, or published to a curated library? GitHub link: https://github.com/apache/texera/discussions/7812 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
