(Sorry to "repost" I originally answered/replied to an older question but my part was not expanding)
Question is: Looking for all the ways to specify a set of jars using --jars on spark-submit? I know this is old but I am about to submit a proposed docs change on --jars, and I had an issue with --jars today. In an older question, when this user submitted the following command line <http://apache-spark-user-list.1001560.n3.nabble.com/how-to-submit-multiple-jar-files-when-using-spark-submit-script-in-shell-tp16662.html> , is that a proper way to reference a jar? hdfs://master:8000/srcdata/kmeans (is that a directory? or a jar that doesn't end with .jar? I have not gotten into the machine learning libs yet to recognize this) I know the docs say, "Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes." So this application-jar can point to a directory and will be expanded? Or needs to be a path to a single specific jar? I ask because when I was testing --jars today, we had to explicitly provide a path to each jar: /usr/local/spark/bin/spark-submit --class jpsgcs.thold.PipeLinkageData ---jars=local:/usr/local/spark/jars/groovy-all-2.3.3.jar,local:/usr/local/spark/jars/guava-14.0.1.jar,local:/usr/local/spark/jars/jopt-simple-4.6.jar,local:/usr/local/spark/jars/jpsgcs-core-1.0.8-2.jar,local:/usr/local/spark/jars/jpsgcs-pipe-1.0.6-7.jar /usr/local/spark/jars/thold-0.0.1-1.jar (The only way I figured out to use the commas was a StackOverflow answer that led me to look beyond the docs to the command line: spark-submit --help results in : --jars JARS Comma-separated list of local jars to include on the driver and executor classpaths. And it seems that we do not need to put the main jar in the --jars argument, I have not tested yet if other classes in the application-jar (/usr/local/spark/jars/thold-0.0.1-1.jar) are shipped to workers, or if I need to put the application-jar in the --jars path to get classes not named after --class to be seen? Thanks for any ideas -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Various-ways-to-use-jars-Some-undocumented-ways-tp25943.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
