GitHub user tanishqgandhi1908 created a discussion: Proposal - Supporting user-provided ML models in workflows
## What we're proposing Let users bring their own trained ML models into Texera and use them in workflows, upload a model, keep versions of it, share it, and run inference on your data — just like datasets work today. Tracking issue: #6494. ## What learned from research Studied how other data/ML platforms handle this, and took a few clear lessons: - **Each resource type gets its own table and its own resolver.** Don't force one shared table or one universal path rule to cover everything — give models their own home, separate from datasets. - **Tell resource types apart by a label in the path**, then look up the right table from that label (e.g. `/datasets/...` vs `/models/...`). - **A model is just a folder of files with versions** — the same shape as a dataset. So we can reuse the storage and versioning. - **To run a model, give the worker a real folder of its files** (via a mount), because model loaders expect to open files from a directory. ## The design Models are a **sibling of datasets**: their own tables and their own storage repo, on top of the same storage engine we already use. The path label decides the type (`/datasets/...` vs `/models/...`), and a worker mounts the chosen model version to read its files. **Tables and storage** <img width="1291" height="944" alt="texera-asset-tables-design" src="https://github.com/user-attachments/assets/965f08c0-f1cb-4d31-8616-f6afd79d13aa"; /> **How a model reaches a worker when it runs** <img width="1249" height="796" alt="texera-geesefs-model-mount" src="https://github.com/user-attachments/assets/66414d36-04b6-44e6-9ea3-9ad0a98741dc"; /> ## The plan One small step per PR (see sub-tasks under #6494): add the path label → model tables → model storage & lookup → upload/version/access API → UI → use in a workflow → sharing & docs. We start with PyTorch; more frameworks and a no-code inference operator come later. ## Questions for the community 1. **Path label**: we're making the label required (a leading `datasets`/`models` segment), with a one-time migration of existing dataset paths. Any concerns with requiring it, versus leaving datasets with no label as the default? 2. **Separate `model` tables** (mirroring the dataset tables) vs. reusing the dataset tables with a type column — any preference or downside we're missing? 3. **Loading models by mounting the files into the worker** — anyone have experience or concerns? Feedback welcome! GitHub link: https://github.com/apache/texera/discussions/6616 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
