Hi User,
I am trying to run spark streaming job in yarn-cluster mode.
It is failing with code
val conf = new SparkConf().setAppName("XXX"). conf.setMaster("yarn-cluster")
val ssc = new StreamingContext(conf,
Seconds(properties.getProperty("batchDurationInSeconds").toInt));
org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
running on a cluster. Deployment to YARN is not supported directly by
SparkContext. Please use spark-submit.
at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
at
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
at
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" org.apache.spark.SparkException: Detected
yarn-cluster mode, but isn't running on a cluster. Deployment to YARN is
not supported directly by SparkContext. Please use spark-submit.
at org.apache.spark.SparkContext.<init>(SparkContext.scala:395)
at
org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:847)
at
org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:81)
at com.rockwell.uptime.Harmonizer$.createStream(Harmonizer.scala:234)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer$$anonfun$1.apply(Harmonizer.scala:215)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:837)
at com.rockwell.uptime.Harmonizer$.main(Harmonizer.scala:215)
at com.rockwell.uptime.Harmonizer.main(Harmonizer.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Please help.
--
Thanks and Regards,
Saurav Sinha
Contact: 9742879062
