I'm trying to launch application inside the cluster (standalone mode)
According to docs, jar-url can be either file:// or hdfs:// format. (
https://spark.incubator.apache.org/docs/latest/spark-standalone.html)
But, when I tried to run spark-class It seemed unable to parse hdfs://xx
format.
<command>
spark-class org.apache.spark.deploy.Client launch \
cds-test05:7077 \
hdfs:///namenode:8020/user/datalab/filename.jar \
my.package.Runner \
-i /user/myself/input -o /user/myself/output -m
spark://sparkmaster:7077
<output>
Jar url 'hdfs:///namenode:8020/user/datalab/filename.jar' is not a valid URL
.
Jar must be in URL format (e.g. hdfs://XX, file://XX)
I've found that *ClientArguments class is using java.net
<http://java.net/>.URL class to parse jar-url, and It doesn't support hdfs
protocol.*
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Unable-to-submit-an-application-to-standalone-cluster-which-on-hdfs-tp1730.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
