GitHub user sshiv012 added a comment to the discussion: Running an operator
inside a container
Adding details on the use case and what we built for it.
**The use case: isolation** Ha's pipeline is three Python steps that each pin
their own dependencies. Running them in one computing unit means one Python
environment, and a single environment can hold only one version of a shared
library. The steps may pin different numpy versions (1.26.4 / 2.1.0 / 2.0.2),
one of them needs `pysam`, another `scikit-image`, and in practice any two bio
tools disagree on something. The failure mode is not performance; it is "this
workflow cannot be installed." Nextflow solves this by giving each process its
own container image, which is why the pipeline existed as a Nextflow workflow
first. The ask was: let a Texera operator do the same.
**How it looks from the UI.** On top of Chris's (@kunwp1) rented-compute
prototype we made the image a per-operator property. In the operator's property
panel there is an Offload toggle; turning it on reveals *Instance type* (a
catalog entry such as `local-2g`, which sets the container's memory cap) and
*Image* (any Docker tag). Leave *Image* blank and the operator runs on the
platform's default worker image; set it to `myorg/qc-step:1.0` and that image
is used for that operator only. When the workflow runs, the master issues
`docker run <image>` per offloaded operator, which pulls the tag from Docker
Hub (or any registry the daemon is logged into) if it is not already local,
exactly like Nextflow's `container = 'myorg/qc-step:1.0'` directive. The one
constraint today: the image must be built `FROM texera-computing-unit-worker`,
because the container has to start a Texera worker and join the cluster. The
step's own libraries go on top of that base, so each operator gets its own envir
onment.
```mermaid
flowchart LR
subgraph UI["Workflow editor"]
P["Operator property panel<br/>Offload: on<br/>Instance type:
local-2g<br/>Image: myorg/qc-step:1.0"]
end
subgraph CU["Computing-unit master (JVM, Pekko cluster seed :2552)"]
O["RuntimeOffloadOrchestrator<br/>rent one container per offloaded
operator<br/>pin the operator's workers to it"]
end
subgraph D["Docker daemon"]
C1["container: qc<br/>image qc:1.0 (numpy 1.26)<br/>Texera worker :2560"]
C2["container: plot_qc<br/>image plot-qc:1.0 (numpy 2.1)<br/>Texera worker
:2561"]
C3["container: interactive<br/>image interactive:1.0 (numpy 2.0)<br/>Texera
worker :2562"]
end
P -->|run| O
O -->|docker run --memory=2g image| C1 & C2 & C3
C1 & C2 & C3 <-->|"Pekko Artery TCP (Kryo tuples + control)"| CU
```
**Same design as Nextflow, different transport.** In both systems the unit of
isolation is "one container per step, each with its own image." The difference
is what crosses the container boundary. Nextflow stages input files into the
task's work directory and runs a shell command; the container is a process
sandbox and data moves as files. In Texera the container runs a full worker
that joins the master's Pekko cluster over Artery TCP, and the operator's input
and output tuples are streamed to it the same way they are streamed between
in-process workers. Nothing in the operator changes; placement is the only
difference, and the dataflow engine keeps working across the boundary
(backpressure, pause/resume, stats).
GitHub link:
https://github.com/apache/texera/discussions/7475#discussioncomment-18807240
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
