Hi,
I have a Spark (Spark 1.5.2) application that streams data from Kafka to
HDFS. My application contains two Typesafe config files to configure
certain things like Kafka topic etc.
Now I want to run my application with spark-submit (cluster mode) in a
cluster.
The jar file with all dependencies of my project is stored on HDFS.
As long as my config files are included in the jar file everything works
fine. But this is unpractical for testing purposes because I always have to
rebuild the jar ...
Therefore I excluded the config files of my project and I added them via
"driver-class-path". This worked on client mode but if I move the config
files now to HDFS and run my application in cluster mode it can't find the
settings. Below you can find my spark-submit command:
/usr/local/spark/bin/spark-submit \
--total-executor-cores 10 \
--executor-memory 15g \
--verbose \
--deploy-mode cluster\
--class com.hdp.speedlayer.SpeedLayerApp \
--driver-class-path hdfs://iot-master:8020/user/spark/config \
--master spark://spark-master:6066 \
hdfs://iot-master:8020/user/spark/speed-layer-CONFIG.jar
Below you can also see the error I get when I run the driver program in
cluster mode:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
at
org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.ExceptionInInitializerError
at com.speedlayer.SpeedLayerApp.main(SpeedLayerApp.scala)
... 6 more
Caused by: com.typesafe.config.ConfigException$Missing: No
configuration setting found for key 'application'
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:145)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
...
I already tried it with the --file parameter and I also changed the HDFS
path to "hdfs:///iot-master:8020//user//spark//config because I thought
there could be a issue with name resolving in Spark. But unfortunately that
also didn't work. Does anybody know how I can get my application run in
cluster-mode?
