That’s right. So, for now we have been running spot-ml in a non-dynamic fashion, I’m just trying to find out what would be the advantage of running dynamics allocation vs. num-executors.
On 4/4/17, 9:20 AM, "[email protected] on behalf of Mark Grover" <[email protected] on behalf of [email protected]> wrote: I am sorry I meant to say (missed the "not" in my previous email): Nope. When *not* using dynamic allocation, it's a static number of executors > throughout the job, no max or min. If you specify the max value config, > it'd be either ignored or result in an error (likely just ignored - haven't > tested). As I understand, min and max don't make sense in a non dynamic allocation scenario, let me know if I am missing something though. Yeah, shuffle service is a pre-requisite for using Dynamic allocation, and yeah, it needs to be enabled. On Mon, Apr 3, 2017 at 10:55 PM, Giacomo Bernardi <[email protected]> wrote: > > Nope. When using dynamic allocation, it's a static number of executors > > throughout the job, no max or min. If you specify the max value config, > > it'd be either ignored or result in an error (likely just ignored - > haven't > > tested). > > No, you can still specify the min and max number of executors even if > you enable dynamic allocation, by using: > - spark.dynamicAllocation.maxExecutors (default: infinity) > - spark.dynamicAllocation.minExecutors (default: 0) > > That's why I was suggesting to simply change: > --num-executors ${SPK_EXEC} \ > into: > --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \ > > Also, do we need spark.shuffle.service.enabled=true, as it seems to be > the case in the docs? > > Giacomo >
