Hi
I am trying to run spark programmatically from eclipse with these
configurations for hadoop cluster locally
SparkConf sparkConf = new
SparkConf().setAppName("simpleTest2").setMaster("yarn")
.set("spark.executor.memory", "1g")
.set("deploy.mode", "cluster")
.set("spark.yarn.stagingDir",
"hdfs://localhost:9000/user/hadoop/")
.set("spark.shuffle.service.enabled", "false")
.set("spark.dynamicAllocation.enabled", "false")
.set("spark.cores.max", "1")
.set("spark.executor.instances","2")
.set("spark.executor.memory","500m") //
.set("spark.executor.cores","1")//
.set("spark.yarn.nodemanager.resource.cpu-vcores","4")
.set("spark.yarn.submit.file.replication", "1")
.set("spark.yarn.jars",
"hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
When I check on the http://localhost:8088/cluster/apps/RUNNING I can see that
my job is submitted but y terminal loops saying
21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
I ve noticed that this occurs after the application of a map on my Dataset.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
