GitHub user tanishqgandhi1908 created a discussion: Proposal - Supporting user-provided ML models in workflows
## Goal Make Texera a platform where a user can **bring their own trained model and run it inside a workflow** — upload the model, pick it in an operator, and get predictions on the data stream. ## Context - We already have a **Hugging Face operator** for hosted/third-party models — that's a separate, ongoing effort. - This proposal is about **models the user uploads themselves** (their own trained weights), stored and versioned in Texera like datasets. ## Scope - **Step 1: PyTorch** — first framework we support end-to-end. - **Next:** TensorFlow, then scikit-learn — same flow, more loaders. <img width="3568" height="1774" alt="image" src="https://github.com/user-attachments/assets/21d0d805-6d57-4f92-a0fc-b8c8b2f00396"; /> ## Storage Models are stored **exactly like datasets** — versioned in LakeFS with the bytes in MinIO — so we reuse the existing upload, versioning, and access-control stack. A model is simply an asset tagged with a type (`DATASET` or `MODEL`); To namespace by type, logical file paths now carry a resource-type prefix — `/datasets/<owner>/<name>/<version>/<file>` and, for models, `/models/<owner>/<name>/<version>/<file>` — resolved through the same path → LakeFS → MinIO chain. A dedicated **Models** section in the UI mirrors Datasets for browsing and uploading. <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/59b713e2-c7b7-489b-9faa-5421ef06b72b"; /> ## Design (from #4198)   ## Two user experiences Both reuse the existing dataset storage (LakeFS/MinIO); a model is just an uploaded asset the operator/UDF fetches at run time. 1. **No-code operator** — drag a "Model Inference" operator, select the uploaded model, choose the input columns and the output column. No code. Best for non-technical users. <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/13523ec7-7cef-4a5d-8348-00ebd989a6cb"; /> 2. **Python UDF** — select the model in the UDF's property panel; the framework fetches + loads it and exposes it to your code, so you only write the inference logic. For users who need custom pre/post-processing. <img width="1440" height="900" alt="image" src="https://github.com/user-attachments/assets/1bfe85ad-fc31-43a1-a9a6-9761285d8081"; /> ## Feedback wanted We'd love feedback on the overall idea and the proposed architecture: - **Library order:** does PyTorch → TensorFlow → scikit-learn match what users need most? - **Two UX:** any concerns with offering both the no-code operator and the UDF (vs. just one)? All thoughts, concerns, and alternative approaches welcome. GitHub link: https://github.com/apache/texera/discussions/6041 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
