Can we see your Spark Configuration parameters ? The mater URL refers to as per java new SparkConf()....setMaster("local[*]") according to where you want to run this
On Mon, 2 Jan 2023 at 14:38, Shrikant Prasad <shrikant....@gmail.com> wrote: > Hi, > > I am trying to migrate one spark application from Spark 2.3 to 3.0.1. > > The issue can be reproduced using below sample code: > > object TestMain { > > val session = > SparkSession.builder().appName("test").enableHiveSupport().getOrCreate() > > def main(args: Array[String]): Unit = { > > import session.implicits._ > val a = *session.*sparkContext.parallelize(*Array* > (("A",1),("B",2))).toDF("_c1","_c2").*rdd*.map(x=> > x(0).toString).collect() > *println*(a.mkString("|")) > > } > } > > It runs successfully in Spark 2.3 but fails with Spark 3.0.1 with below > exception: > > Caused by: org.apache.spark.SparkException: A master URL must be set in > your configuration > > at > org.apache.spark.SparkContext.<init>(SparkContext.scala:394) > > at > org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2690) > > at > org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949) > > at scala.Option.getOrElse(Option.scala:189) > > at > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943) > > at TestMain$.<init>(TestMain.scala:7) > > at TestMain$.<clinit>(TestMain.scala) > > > From the exception it appears that it tries to create spark session on > executor also in Spark 3 whereas its not created again on executor in Spark > 2.3. > > Can anyone help in identfying why there is this change in behavior? > > Thanks and Regards, > > Shrikant > > -- > Regards, > Shrikant Prasad >