For the last question, have you looked at: https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation
FYI On Sun, May 15, 2016 at 5:19 AM, Mail.com <pradeep.mi...@mail.com> wrote: > Hi , > > I have seen multiple videos on spark tuning which shows how to determine # > cores, #executors and memory size of the job. > > In all that I have seen, it seems each job has to be given the max > resources allowed in the cluster. > > How do we factor in input size as well? I am processing a 1gb compressed > file then I can live with say 10 executors and not 21 etc.. > > Also do we consider other jobs in the cluster that could be running? I > will use only 20 GB out of available 300 gb etc.. > > Thanks, > Pradeep > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
