mattcasters commented on issue #8332: URL: https://github.com/apache/hop/issues/8332#issuecomment-5666642543
Copy-paste this into https://github.com/apache/hop/issues/8332: --- ## Proposal: native Spark ML transforms (not another multi-engine PMI layer) **hop-mi / PMI** (Weka, sklearn, R MLR, Spark MLlib-via-Weka, DL4j, Keras zoo) is the wrong long-term shape. The engines are dated, the GPL/Weka license does not belong in the ASF assembly, and wrapping every algorithm behind one GUI fights how ML is done now. The pattern that has stuck in production ML is **Feature / Training / Inference**: - **Feature pipelines** — Hop already owns this (local + native Spark 4.1 + Delta/Iceberg). - **Training** — either `spark.ml` on a Dataset, or Python (sklearn / XGBoost / PyTorch) outside the row engine. - **Inference** — score a saved artifact (Spark `PipelineModel` or ONNX), with the same feature logic as training. Hop Python (`hop python` / Py4J) is for **metadata and orchestration**, not for feeding rows into MLlib. The Script transform “Python” engine is Jython. Neither is a Spark ML runtime. **Recommendation:** implement ML on the **native Spark engine as Dataset handlers**, same class of thing as Merge Join and Lake Table Output — not as generic `mapPartitions` mini-pipelines, and not as a local-engine Weka/MLlib hybrid. ### Transforms (v1) | Transform | Spark mapping | Notes | |---|---|---| | **Spark Vector Assembler** | `VectorAssembler` | numeric (+ later StringIndexer / OneHotEncoder) → `features` | | **Spark ML Train** | `estimator.fit(df)` + `model.write` | driver-side action; model path on the Spark FS | | **Spark ML Score** | `PipelineModel.load(...).transform(df)` | adds `prediction` / `probability` | | **Spark ML Evaluate** | evaluators | metrics rows or a small side file | **v1 algorithms only:** `LogisticRegression`, `LinearRegression`, `GBTClassifier` / `GBTRegressor`, `RandomForest*`, `ALS`, `KMeans`, plus `StringIndexer` + `OneHotEncoder`. That covers most lakehouse tabular jobs. `fit` is a cluster action. Do **not** call MLlib inside `mapPartitions` (that trains N local models). Treat Train like lake table writes: run the action while the graph is built, register an empty leaf Dataset so a later `count()` does not re-train. Local Hop engine stays out of `spark.ml`. Local scoring later is ONNX (or CPython), not MLlib jars in the Hop JVM. Same canvas, different run configuration, different handler — same idea as File vs Spark File. Leave DL / LLM training out of this ticket. Language Model Chat already covers that side. ### TODO - [ ] Write a short “ML on Hop” doc page: FTI split, hop-mi = legacy/community, this plugin = Spark path. - [ ] New Apache-2.0 plugin next to `plugins/engines/spark` (Spark 4.1 / Scala 2.13), **not** GPL hop-mi. - [ ] Native handler: **Spark Vector Assembler**. - [ ] Native handler: **Spark ML Train** (GBT + logistic first; model save path, overwrite flag, basic params). - [ ] Native handler: **Spark ML Score** (load `PipelineModel`, transform Dataset). - [ ] Native handler: **Spark ML Evaluate** (binary / multiclass / regression metrics). - [ ] Add RF, linear, ALS, KMeans, StringIndexer, OneHotEncoder. - [ ] Persist Train/Score as an action during graph materialisation (no double-fit on `Dataset.count()`). - [ ] Sample project: feature pipeline → Iceberg/Delta → Train → Score on native Spark. - [ ] Sibling sample (orchestration only): same features → Python sklearn → ONNX; Hop scores later. Documents the Python path without pretending Hop Python is MLlib. - [ ] Integration tests on `local[*]` + one `spark-submit` path. - [ ] Explicitly unsupported: Jython Script as ML, hop-mi Spark-via-Weka, DL4j/Keras/R in core. - [ ] Follow-ups (not v1): ONNX Score on local + mapPartitions; CPython/Arrow transform brought to Java 21 / Hop 2.20; optional MLflow / Feast metadata types. Happy to take Train + Score + Assembler for GBT/logistic against an Iceberg table as the first slice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
