GitHub user aicam edited a discussion: Mounting datasets and models: three open
questions
## Background
Since #6866, #6896 and #6897, a Python UDF can take a dataset or model version
as a parameter (`value=Resource.DATASET` or `value=Resource.MODEL`) and read it
as a local, read-only directory. When an execution starts, the engine asks
access-control-service to mount each version the workflow names. A mounter
DaemonSet on each node then runs GeeseFS against file-service's read-only S3
proxy, which checks the user's read access on every request. The mount reaches
the computing unit through mount propagation. Files are fetched on demand, so a
UDF can load a multi-GB model or walk a folder of many files without
downloading anything first.
I'd like the community's input on three questions about where this goes next.
### 1. Do we still need the Python library for reading dataset files?
A Python UDF can also read a file with
`pytexera.DatasetFileDocument("/dataset/<owner>/<name>/<version>/<file>").read_file()`.
It gets a presigned URL from file-service and downloads the whole file into
memory. It handles one file at a time and cannot list a folder. A mounted
version covers the same need with plain `open()` and `os.listdir()`, and works
with any library that takes a path (pandas, torch, PIL, ...).
However, mounting is Kubernetes-only and off by default, so single-node
deployments and clusters that enforce the `baseline` or `restricted` Pod
Security Standards can't use it. Should we keep `DatasetFileDocument` as the
fallback, deprecate it once mounting can be on everywhere, or do something else?
### 2. What should mounting look like for native operators?
We have wanted operators that work on many files at once for a long time: a
folder of images, a sharded CSV, a directory of documents. Today the native
scan operators (CSV, JSONL, Arrow, File Scan, ...) each take a single file path
and read it through a presigned URL. None of them accepts a folder or a
pattern, and none uses mounting. File Scan can read many files only when
another operator feeds it file names, or by unpacking a zip. The infrastructure
for reading a whole folder now exists, but only Python UDFs use it.
Some possible directions:
- Scan operators take a dataset version (or a folder in it) plus a pattern such
as `*.csv`, and either concatenate the matching files or emit one row per file.
- A "List files" source that outputs the paths in a mounted version for
downstream operators to read.
- Operators read mounted files in place instead of copying them to temp files.
What would you want the experience to be, and which operators should come first?
### 3. Is the current design secure enough?
How it works today:
- The mounter DaemonSet is privileged (FUSE needs it) and listens on a host
port on every node. It accepts requests only from access-control-service, which
it verifies with a Kubernetes TokenReview.
- Access-control-service requires WRITE access to the computing unit and read
access to the dataset or model version, using the token of the computing unit's
owner.
- The computing-unit pod itself stays unprivileged. It does need a hostPath
volume (its own per-unit folder under the node's mount root) with
HostToContainer propagation, and that hostPath is why mounting is opt-in.
- GeeseFS authenticates to the S3 proxy with the user's JWT as its access key.
The proxy validates the JWT, checks read access on each request, and re-signs
the request with the deployment's LakeFS credentials.
Things I'm unsure about:
- A hostPath in user pods, plus a privileged agent on every node.
- The user's JWT is held by a long-lived GeeseFS process on the node.
- Mounts are released when the pod is deleted, not when an execution ends.
- With the optional disk cache, data stays on the node after access is revoked.
Is this an acceptable trade-off? Would an alternative be better, for example a
CSI driver, a FUSE sidecar in each computing unit, or no FUSE at all with a
streaming client?
@parshimers @chenlica Let me know if the questions are complete or you want to
add anything to them.
@bobbai00 your feedback is welcome here since you developed the library.
GitHub link: https://github.com/apache/texera/discussions/9048
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
