aicam opened a new issue, #6606: URL: https://github.com/apache/texera/issues/6606
## Summary Add the ability to **mount a versioned dataset directly into a computing unit pod's file system**, so operator code can read dataset files from a local path instead of downloading them over HTTP first. The initial consumer is the **Python UDF** operator, which gets a new "Mount dataset" property; the mounted directory is exposed to user code as a path variable. ## Motivation Today, if an operator needs a dataset's files (e.g. a large ML model with multiple shards), the options are to stream them through the storage API or download the whole dataset into the pod before use. For large datasets (multi-GB models) this is slow, uses a lot of pod disk, and forces a full transfer even when the code only reads part of the data. Texera stores datasets in MinIO/S3 with **LakeFS** providing versioning on top. LakeFS exposes an **S3-compatible gateway**, which means a dataset version (a LakeFS repository at a specific commit) can be mounted as a read-only file system using a FUSE S3 client. We use **[GeeseFS](https://github.com/yandex-cloud/geesefs)** for this. Key properties of the approach: - **Lazy / on-demand**: the mount itself transfers zero bytes; file contents are fetched with ranged reads only when the code actually `read()`s them, with read-ahead for throughput. - **Version-pinned & immutable**: a dataset version maps to a LakeFS commit, so the mounted content is content-addressed and never changes underneath the reader — no cache invalidation concerns. - **Storage-backend agnostic**: GeeseFS always talks to the LakeFS S3 gateway, so it works identically whether LakeFS is backed by in-cluster MinIO or an external S3 bucket. This has been prototyped end-to-end: a ~2 GB sharded PyTorch model in one dataset, loaded inside a Python UDF via `torch.load()` from the mounted path, with bit-exact outputs — validated against both in-cluster MinIO and a remote AWS S3 backing store. ## Scope (Kubernetes deployment only) This feature targets the Kubernetes deployment. Mounting requires FUSE inside the pod, which needs a privileged computing-unit pod and `CAP_SYS_ADMIN` on the mount binary. It is therefore gated behind an opt-in flag (`fuseMountEnabled`, default off) and is a no-op when disabled. ## How it works (end to end) 1. Python UDF has a new **"Mount dataset"** property (`/ownerEmail/datasetName/versionName`). 2. At compile time the dataset path is resolved to a `repository:commitHash` locator via LakeFS/DB lookup. 3. The locator flows through `PhysicalOp` → `WorkerConfig` to the Python worker. 4. Before the Python worker starts, a **mount manager** runs GeeseFS against the LakeFS S3 gateway, mounting `repository:commitHash` read-only at a local path in the pod. 5. The mounted path is exposed to UDF code as the variable **`MOUNTED_DATASET_PATH`**. ## Delivery plan (sub-issues / PRs) The feature will land as a series of small, independently reviewable PRs. Each PR before the last is dormant (default-off / no active caller), so nothing changes behavior until the stack is complete. - [ ] **1. Package GeeseFS in the computing-unit image + Helm flag scaffolding** (this stack's first PR) — install the GeeseFS binary + FUSE into the computing-unit image and add the default-off `fuseMountEnabled` Helm flag. No behavior change. - [ ] **2. Computing-unit manager honors the flag** — when enabled, create computing-unit pods privileged and inject LakeFS credentials into them. - [ ] **3. Engine mount manager** — `DatasetMountManager` runs GeeseFS against the LakeFS gateway; carry the dataset locator through `PhysicalOp`/`WorkerConfig` to the Python worker. - [ ] **4. Dataset-version resolution** — resolve `/owner/dataset/version` to a `repository:commitHash` locator. - [ ] **5. Python UDF "Mount dataset" property + runtime exposure** — add the operator property and expose the mounted path to UDF code as `MOUNTED_DATASET_PATH`. Future consumers (other operators, other languages) can reuse the same mount plumbing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
