Re: Number of executors and dynamic allocation

[Barona, Ricardo]

I created this bug in JIRA: https://issues.apache.org/jira/browse/SPOT-136
We can continue discussing the details in there.

Thanks. 

On 4/3/17, 10:12 AM, "Barona, Ricardo" <ricardo.bar...@intel.com> wrote:

    Completely not a silly question, it’s actually a really good observation. I 
think we didn’t update correctly the ml_ops.sh script when we added these 
parameters. 
    
    What we could start discussing is whether we want dynamic allocation or if 
we want fixed number of executors. I’m going to let the mic open to see what 
people think about this.
    
    Thanks Giacomo.
    
    On 4/3/17, 3:29 AM, "Giacomo Bernardi" <m...@minux.it> wrote:
    
        Hi,
        hope this is not a silly question. In ml_ops.sh there are:
          --num-executors ${SPK_EXEC} \
        and:
          --conf spark.dynamicAllocation.enabled=true \
        
        which trigger the warning:
          WARN spark.SparkContext: Dynamic Allocation and num executors both
        set, thus dynamic allocation disabled.
        
        Shouldn't we remove the "--num-executors" and add instead:
          --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
        ?
        
        Thanks.
        Giacomo