You cannot run Spark in cluster mode by instantiating a SparkContext like
that.
You have to launch it with the "spark-submit" command line script.
On Thu, Jul 9, 2015 at 2:23 PM, jegordon <jgordo...@gmail.com> wrote:
> Hi to all,
>
> Is there any way to run pyspark scripts with yarn-cluster mode without
> using
> the spark-submit script? I need it in this way because i will integrate
> this
> code into a django web app.
>
> When i try to run any script in yarn-cluster mode i got the following error
> :
>
> org.apache.spark.SparkException: Detected yarn-cluster mode, but isn't
> running on a cluster. Deployment to YARN is not supported directly by
> SparkContext. Please use spark-submit.
>
>
> I'm creating the sparkContext in the following way :
>
> conf = (SparkConf()
> .setMaster("yarn-cluster")
> .setAppName("DataFrameTest"))
>
> sc = SparkContext(conf = conf)
>
> #Dataframe code ....
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Pyspark-not-working-on-yarn-cluster-mode-tp23755.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
--
Marcelo
