Hi, So I was trying to get the Kafka Streaming working in a standalone python application.
I needed to add the dependencies for this to work. The suggested way for this is doing --jars using spark-submit, which is not practical considering I wanted to launch and debug an application. Also you still have to set -driver-class-path for this to work. So after searching around I found a JIRA, which said, that setting SPARK_SUBMIT_CLASSPATH is workaround. This works on Linux, but on Windows just setting the environmental var did not seem to work. So I looked into spark-submit.cmd and it seems SPARK_SUBMIT_CLASSPATH is being set to: set SPARK_SUBMIT_CLASSPATH= This is then overridden if --driver-class-path is supplied. But did not want to supply the argsument, I just wanted my enviromental variable to persist like on Linux. So instead i changed the line set SPARK_SUBMIT_CLASSPATH=%SPARK_SUBMIT_CLASSPATH% Now everything works as expected and I can just inject the var in my python script before importing pypsark os.environ["SPARK_SUBMIT_CLASSPATH"]="path1.jar;path2.jar" So I was wondering if there is any reason this is not the default behavior or I am just doing something wrong? This is my first time posting, please excuse me if this is the wrong place to post this. Cheers, Artyom -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPARK-SUBMIT-CLASSPATH-Windows-PYSPARK-tp11868.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
