aicam opened a new pull request, #6867:
URL: https://github.com/apache/texera/pull/6867
### What changes were proposed in this PR?
Build on the mount infrastructure (#6862) to let the platform and users
mount datasets and use them in Python UDFs.
- **Engine mount client** — `DatasetMountManager` asks the node mounter to
mount a dataset for the computing unit; the locator flows through
`PhysicalOp`/`WorkerConfig` to the Python worker, which exposes the resolved
local path.
- **Per-computing-unit mount API** — `GET/POST/DELETE
/computing-unit/{cuid}/mounts` (`ComputingUnitManagingResource` +
`MounterClient`), a stateless proxy to that CU's node mounter; `FileResolver`
resolves dataset paths both ways.
- **"Mount datasets into computing unit" UI** — a per-CU modal to mount /
list / unmount datasets.
- **Python UDF dataset-variable bindings** — bind each mounted dataset to a
variable holding its local path, e.g. `open(f"{A}/file.csv")`.
### Any related issues, documentation, discussions?
Closes #6863 · part of #6606. Stacks on #6862 (mount infrastructure); this
PR's diff becomes clean once #6862 merges.
### How was this PR tested?
`sbt
WorkflowOperator/WorkflowExecutionService/ComputingUnitManagingService/FileService
compile` + production frontend build green; unit tests updated
(`PythonWorkflowWorkerStartupConfigSpec`, `test_run_python_worker.py`).
End-to-end on minikube: mounted a dataset onto a Kubernetes CU via the API,
bound it to a variable in a Python UDF, and read a ~2 GB sharded PyTorch model
from the variable's path via `torch.load`, bit-exact (`model_output_sum ==
expected_sum`).
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
