GitHub user aicam edited a comment on the discussion: Proposal - Supporting
user-provided ML models in workflows
## Design update
### Measurements (GeeseFS)
| What we measured | Result |
| --- | --- |
| Mounting one model version twice | Data transferred twice. Caches are
per-mount at two layers (GeeseFS file cache; content fetched via file-service's
S3 proxy) — neither is shared. *N* mounts = *N* × (transfer + cache). |
| Cost of a live mount | One GeeseFS process per mount for its lifetime: ≈30 MB
RSS, ≈10 threads. |
Implication: mount only what a run actually uses. Mounting everything a user
can access is not viable, and mount *timing* becomes a design decision.
### Decisions
**1. Mounts are isolated per computing unit** — keyed by `(cuid, repository,
commit)`, not by the model locator alone.
- Mounter mounts at `MOUNT_ROOT/<cuid>/<repo>/<commit>`; a CU pod's `hostPath`
is only its own `<cuid>` subtree.
- Two CUs using the same model version get two mounts, two caches — the cost
above, accepted for three reasons: (a) mounts are read-only today, but
write-back (checkpoints, fine-tuned weights) would leak across CUs on a shared
mount; (b) each mount is authorized with the pod's own user JWT, so a shared
mount would have to pick one identity for several users; (c) teardown is a
per-CU unmount instead of refcounting across pods.
- Isolation is implicit in the path scheme, so it is pinned by tests on both
the mounter and the pod's host path.
**2. CU pods never mount; a privileged per-node DaemonSet does.**
```
CU pod ──POST /mount {cuid, repo, commit, jwt, fileServiceBase}──►
texera-mounter (per node, privileged)
│
geesefs → file-service S3 proxy
▼
MOUNT_ROOT/<cuid>/<repo>/<commit> (read-only)
│ mount
propagation
▼
CU pod sees
/mnt/texera-mounts/<repo>/<commit>
```
No LakeFS credential is held by the pod or the mounter. The mounter is
stateless — the mount path encodes its own identity — and a watcher unmounts a
CU's subtree when its pod is deleted.
**3. Resolution and mounting moved from compile time to run time.** Previously
both happened in `getPhysicalOp`, which the compiling service re-runs on every
workflow edit — a DB round trip per model parameter, hundreds of times, for a
workflow that may never run. The mount path also belongs to the CU that runs
the operator, not to whoever compiled it.
- Compilation injects the selected model version path (complete, type-correct,
free) and validates that a model was selected.
- A new `ExecutionTimeBinding` on `PhysicalOp` resolves the mount path once per
operator when its region starts, memoized across worker configs and executor
init.
- The mount itself happens in the worker, possibly on another node.
- Caveat for future work: the compiling service serializes the physical plan
back to the editor, so a lazy value reachable through a Jackson getter would
defer nothing. Pinned by a test.
**4. The explicit mount step in the CU panel is removed.** Model selection
happens in the operator's property panel via a new UI parameter kind, built on
@carloea2's work (#5912) rather than a parallel mechanism:
```python
model_dir = self.UiParameter("iris_classifier", UiParameterType.MODELS).value
```
The value is the local directory the version is mounted at. The set of models a
run needs is now derived from the workflow, not from CU state.
**5. Sample code is generated per model type**, keyed on the model's
`(framework, format)`: pytorch/torchscript, pytorch/state-dict,
pytorch/safetensors, tensorflow/savedmodel, onnx/onnx, sklearn/joblib,
sklearn/pickle, plus a generic fallback. The snippet loads in `open()` (once
per worker, off the per-tuple path) and joins the mounted directory with the
file's relative path. `framework` and `format` are columns on `model` and
editable after upload.
### Type: three distinct notions, not to be conflated
1. **Resource type** — dataset vs model, from the path label (`/datasets/…`,
`/models/…`), with a one-time migration of paths stored in `workflow.content`.
2. **Model type** — `(framework, format)`, drives the sample code.
3. **UI parameter type** — how the property panel renders the selector (#5912);
`models` is one kind.
Open with @carloea2: whether (2) is a closed enum validated at upload, or
free-form with a known-pairs table (current behaviour).
### Open — model environment requirements
Not yet designed; this is our next investigation.
- **Hardware**: GPU requirement and VRAM. CUs already expose a GPU limit, so
the gap is model-side metadata plus a check when a GPU model runs in a CU
allocated none.
- **Versions**: TorchScript is not guaranteed portable across PyTorch versions;
ONNX opset; CUDA/driver compatibility.
- **OS-level**: how far a model is tied to the OS it was produced on — glibc
version and native extensions pulled in by pickled objects are the realistic
failure modes. `pickle`/`joblib` are worst-case (loading imports whatever the
pickle references); safetensors and TorchScript are better behaved.
Likely direction: record environment requirements as model metadata and match
them against the CU's image and allocation, rather than a per-model environment
mechanism.
### Next
- Report environment measurements here.
- Settle the type question with @carloea2.
- PR plan against the sub-issues of #6494: mounter + isolation → execution-time
binding → `models` UI parameter → snippet generator.
GitHub link:
https://github.com/apache/texera/discussions/6616#discussioncomment-17909726
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
