bamaer opened a new issue, #8988: URL: https://github.com/apache/hop/issues/8988
### What needs to happen? The native Spark docs cover `local[*]` and `spark-submit` in Docker, but not Hop GUI, `hop-run` or Hop Server as the driver of a Spark standalone cluster. Setting that up runs into things the docs don't mention: - The Spark and Beam engines are marketplace plugins, not part of the client download. - The fat jar builder (`hop-conf --generate-fat-jar`) ships with the Beam plugin. - The *Spark standalone server* run configuration template leaves the fat jar empty, and runs then fail with `ClassCastException ... SerializedLambda ... MapPartitionsRDD.f`. - Every executor copies the ~900 MB fat jar into `/tmp` and into the worker's `work` folder. - Hop GUI and Hop Server keep their Spark application, with its cores, until they stop. - `spark-submit` started from the Hop installation folder fails with `... is not a native Spark pipeline engine configuration`. Add a page that sets up a standalone cluster on one machine and runs a pipeline on it from Hop GUI, `hop-run` and Hop Server, with a troubleshooting section. Correct the getting-started page where it says the engine ships with the client. ### Issue Priority Priority: 2 ### Issue Component Component: Documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
