bamaer opened a new issue, #9016: URL: https://github.com/apache/hop/issues/9016
### What would you like to happen? The native Spark engine cannot use YARN when Hop itself is the Spark driver (Hop GUI, `hop-run`, Hop Server), although the run configuration templates offer *YARN client*: - `plugins/engines/spark/lib` has no `spark-yarn` jar, so master `yarn` fails with `Could not parse Master URL: 'yarn'`. - With `spark-yarn` added, the YARN client still connects to `0.0.0.0:8032`. `HADOOP_CONF_DIR` / `YARN_CONF_DIR` only reach the classpath when `spark-submit` starts the JVM, so a Hop JVM never reads `yarn-site.xml`, `core-site.xml` or `hdfs-site.xml`. - A run configuration with an empty master silently runs on `local[*]` outside `spark-submit`. The *spark-submit (cluster)* template on Hop Server runs inside the Hop Server JVM, not on the cluster. **Proposed** - Ship `spark-yarn` (matching the engine's Spark version) with the Spark plugin. - Load the Hadoop configuration from `HADOOP_CONF_DIR` / `YARN_CONF_DIR`, with an optional *Hadoop configuration folder* option on the native Spark run configuration. - Log a warning (or fail) when the master is empty and the pipeline does not run under `spark-submit`. - Test with a single-node YARN cluster in Docker. - Update the docs: YARN client setup (fat jar, `spark.driver.host`, Spark version on the cluster) and troubleshooting. Kerberos-secured clusters are out of scope. ### Issue Priority Priority: 2 ### Issue Component Component: Hop Gui, Component: Hop Server -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
