arvindshmicrosoft commented on issue #322: Fix issues with setting up Spark URL: https://github.com/apache/fluo-muchos/pull/322#issuecomment-584482033 For context, this fixes the error below: ``` TASK [hadoop : copy spark yarn shuffle jar to hadoop lib] *********************************************************************************************************************************** Tuesday 11 February 2020 03:50:12 +0000 (0:00:00.188) 0:05:24.372 ****** fatal: [worker-3]: FAILED! => {"changed": true, "cmd": ["cp", "/home/centos/install/spark-2.4.5-bin-without-hadoop/yarn/spark-2.4.5-yarn-shuffle.jar", "/home/centos/install/hadoop-3.2.1/share/hadoop/yarn/lib/"], "delta": "0:00:00.004140", "end": "2020-02-11 03:50:13.223904", "msg": "non-zero return code", "rc": 1, "start": "2020-02-11 03:50:13.219764", "stderr": "cp: cannot stat '/home/centos/install/spark-2.4.5-bin-without-hadoop/yarn/spark-2.4.5-yarn-shuffle.jar': No such file or directory", "stderr_lines": ["cp: cannot stat '/home/centos/install/spark-2.4.5-bin-without-hadoop/yarn/spark-2.4.5-yarn-shuffle.jar': No such file or directory"], "stdout": "", "stdout_lines": []} ``` The efficacy of the fix was further validated by running the SparkPi sample: ``` spark-submit --class org.apache.spark.examples.SparkPi --master yarn --executor-cores 1 --queue thequeue $SPARK_HOME/examples/jars/spark-examples*.jar 10 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
