I agree,
There are a few things in ml_ops that need to be rethought.
For this instance we could add a line in spot.conf,
i.e. SPK_DYNAMIC_ALLOCATION=true
in ml_ops you could do something like:
if SPK_DYNAMIC_ALLOCATION == true then
"--conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} --conf
spark.dynamicAllocation.enabled=true”
else
“--num-executors ${SPK_EXEC}”
i’m not sure if placement for both can be in the same position when calling
spark-submit, that could be a concern.
Also can you set maxExecutors without enabling dynamicAllocation?
- Nathanael
> On Apr 3, 2017, at 8:19 AM, Barona, Ricardo <[email protected]> wrote:
>
> I created this bug in JIRA: https://issues.apache.org/jira/browse/SPOT-136
> We can continue discussing the details in there.
>
> Thanks.
>
> On 4/3/17, 10:12 AM, "Barona, Ricardo" <[email protected]> wrote:
>
> Completely not a silly question, it’s actually a really good observation.
> I think we didn’t update correctly the ml_ops.sh script when we added these
> parameters.
>
> What we could start discussing is whether we want dynamic allocation or if
> we want fixed number of executors. I’m going to let the mic open to see what
> people think about this.
>
> Thanks Giacomo.
>
> On 4/3/17, 3:29 AM, "Giacomo Bernardi" <[email protected]> wrote:
>
> Hi,
> hope this is not a silly question. In ml_ops.sh there are:
> --num-executors ${SPK_EXEC} \
> and:
> --conf spark.dynamicAllocation.enabled=true \
>
> which trigger the warning:
> WARN spark.SparkContext: Dynamic Allocation and num executors both
> set, thus dynamic allocation disabled.
>
> Shouldn't we remove the "--num-executors" and add instead:
> --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
> ?
>
> Thanks.
> Giacomo
>
>
>
>
