Thank you For the answer
I have set now these properties as you suggest
SparkConf sparkConf = new
SparkConf().setAppName("simpleTest2").setMaster("yarn")
.set("spark.executor.memory", "1g")
.set("deploy.mode", "cluster")
.set("spark.yarn.stagingDir",
"hdfs://localhost:9000/user/hadoop/")
.set("spark.hadoop.fs.defaultFS","hdfs://localhost:9000")
.set("spark.dynamicAllocation.enabled", "false")
.set("spark.worker.cores","1")
.set("maximizeResourceAllocation", "true")
.set("spark.executor.memory","1g") //
.set("spark.executor.cores","1")//
.set("spark.worker.instances","1")
.set("spark.driver.instances","1")
.set("spark.driver.cores","1")
.set("spark.driver.memory","1g")
I have Checked at localhost:8080 I have 1 worker active (I have activated with
spark/sbin/start-worker.sh spark://localhost:7077). I guess whether I need this
worker active. Because when i launch my job its status does not change
On 2021/09/28 12:29:24, Stelios Philippou <stevo...@gmail.com> wrote:
> You need to check your Spark Cluster.
> I can assume that this is local
>
> http://localhost:8080
>
> As the example below you will need to have 2 worker instances and 1 driver
> instance so most probably 3 workers.
> They need to have at least one Core and 1g ram to them. But you must not
> use all the available ram. Aim for using 70% of the available ram at the
> beginning.
>
> You are also adding the
> > > > > .set("spark.executor.memory", "1g")
> twice. Perhaps you need to driver instance. ?
>
> An example would bem but you can translate them to SparkConf
>
> --conf spark.executor.cores=1 \
>
> --conf spark.executor.memory=1g \
>
>
> --conf spark.driver.instances=1 \
>
> --conf spark.driver.cores=1 \
>
> --conf spark.driver.memory=1g \
>
>
> Also setting the
> > > > > .set("spark.executor.instances","2")
> might not work and the executor would pick up all available instances.
>
> [image: image.png]
>
> On Tue, 28 Sept 2021 at 15:09, davvy benny <davv...@gmail.com> wrote:
>
> > How can I check it?
> >
> > On 2021/09/28 03:29:45, Stelios Philippou <stevo...@gmail.com> wrote:
> > > It might be possible that you do not have the resources on the cluster.
> > So
> > > your job will remain to wait for them as they cannot be provided.
> > >
> > > On Tue, 28 Sep 2021, 04:26 davvy benny, <davv...@gmail.com> wrote:
> > >
> > > > How can I solve the problem?
> > > >
> > > > On 2021/09/27 23:05:41, Thejdeep G <tejde...@gmail.com> wrote:
> > > > > Hi,
> > > > >
> > > > > That would usually mean that the application has not been allocated
> > the
> > > > executor resources from the resource manager yet.
> > > > >
> > > > > On 2021/09/27 21:37:30, davvy benny <davv...@gmail.com> wrote:
> > > > > > Hi
> > > > > > I am trying to run spark programmatically from eclipse with these
> > > > configurations for hadoop cluster locally
> > > > > > SparkConf sparkConf = new
> > > > SparkConf().setAppName("simpleTest2").setMaster("yarn")
> > > > > > .set("spark.executor.memory", "1g")
> > > > > > .set("deploy.mode", "cluster")
> > > > > > .set("spark.yarn.stagingDir",
> > > > "hdfs://localhost:9000/user/hadoop/")
> > > > > > .set("spark.shuffle.service.enabled", "false")
> > > > > > .set("spark.dynamicAllocation.enabled",
> > "false")
> > > > > > .set("spark.cores.max", "1")
> > > > > > .set("spark.executor.instances","2")
> > > > > > .set("spark.executor.memory","500m") //
> > > > > > .set("spark.executor.cores","1")//
> > > > > >
> > > > .set("spark.yarn.nodemanager.resource.cpu-vcores","4")
> > > > > >
> > .set("spark.yarn.submit.file.replication",
> > > > "1")
> > > > > > .set("spark.yarn.jars",
> > > > "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
> > > > > >
> > > > > > When I check on the http://localhost:8088/cluster/apps/RUNNING I
> > can
> > > > see that my job is submitted but y terminal loops saying
> > > > > > 21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted
> > any
> > > > resources; check your cluster UI to ensure that workers are registered
> > and
> > > > have sufficient resources
> > > > > >
> > > > > > I ve noticed that this occurs after the application of a map on my
> > > > Dataset.
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
