Re: Kube estimate for Spark

I think he’s running Spark on Kubernetes not YARN as cluster manager Sent from my iPhone > On Jun 3, 2021, at 6:05 AM, Mich Talebzadeh wrote: > >  > Please provide the spark version, the environment you are running (on-prem, > cloud etc), state if you are running in YARN etc and your spark-s

Re: Kube estimate for Spark

Please provide the spark version, the environment you are running (on-prem, cloud etc), state if you are running in YARN etc and your spark-submit parameters. Have you checked spark UI default on port 4040 under stages and executor tabs HTH view my Linkedin profile

Kube estimate for Spark

Hi Team, I am trying to understand how to estimate Kube cpu with respect to Spark executor cores. For example, Job configuration: (given to start) cores/executor = 4 # of executors = 240 But the allocated resources when we ran job are as follows, cores/executor = 4 # of executors = 47 So the q