Hi All, I'm trying to use the new feature of yarn cluster mode to run Spark 2.4.0 jobs on Zeppelin 0.8.1. I've set the SPARK_HOME, SPARK_SUBMIT_OPTIONS, and HADOOP_CONF_DIR env variables in zeppelin-env.sh so that the Spark interpreter can be started in the cluster. I used `--jars` in SPARK_SUBMIT_OPTIONS to add local jars. However, when I tried to import a class from the jars in a Spark paragraph, the interpreter complained that it cannot find the package and class ("<console>:23: error: object ... is not a member of package ..."). Looks like the jars are not properly imported.
I followed the instruction here <https://zeppelin.apache.org/docs/0.8.1/interpreter/spark.html#2-loading-spark-properties> to add the jars, but it seems that it's not working in the cluster mode. And this issue seems to be related to this bug: https://jira.apache.org/jira/browse/ZEPPELIN-3986. Is there any update on fixing it? What is the right way to add local jars in yarn cluster mode? Any help and update are much appreciated. Here's the SPARK_SUBMIT_OPTIONS I used (packages and jars paths omitted): export SPARK_SUBMIT_OPTIONS="--driver-memory 12G --packages ... --jars ... --repositories https://repository.cloudera.com/artifactory/public/,https://repository.cloudera.com/content/repositories/releases/,http://repo.spring.io/plugins-release/ " Thanks, - Ethan -- Best, - Ethan