Alan Braithwaite created SPARK-10643:
----------------------------------------
Summary: Support HDFS urls in spark-submit
Key: SPARK-10643
URL: https://issues.apache.org/jira/browse/SPARK-10643
Project: Spark
Issue Type: New Feature
Reporter: Alan Braithwaite
Priority: Minor
When using mesos with docker and marathon, it would be nice to be able to make
spark-submit deployable on marathon and have that download a jar from HDFS
instead of having to package the jar with the docker.
{code}
$ docker run -it docker.example.com/spark:latest
/usr/local/spark/bin/spark-submit --class
com.example.spark.streaming.EventHandler hdfs://hdfs/tmp/application.jar
Warning: Skip remote jar hdfs://hdfs/tmp/application.jar.
java.lang.ClassNotFoundException: com.example.spark.streaming.EventHandler
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:639)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
Although I'm aware that we can run in cluster mode with mesos, we've already
built some nice tools surrounding marathon for logging and monitoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
