Hi,
hope this is not a silly question. In ml_ops.sh there are:
  --num-executors ${SPK_EXEC} \
and:
  --conf spark.dynamicAllocation.enabled=true \

which trigger the warning:
  WARN spark.SparkContext: Dynamic Allocation and num executors both
set, thus dynamic allocation disabled.

Shouldn't we remove the "--num-executors" and add instead:
  --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
?

Thanks.
Giacomo

Reply via email to