Hi , I have seen multiple videos on spark tuning which shows how to determine # cores, #executors and memory size of the job.
In all that I have seen, it seems each job has to be given the max resources allowed in the cluster. How do we factor in input size as well? I am processing a 1gb compressed file then I can live with say 10 executors and not 21 etc.. Also do we consider other jobs in the cluster that could be running? I will use only 20 GB out of available 300 gb etc.. Thanks, Pradeep --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
