Hi,
hope this is not a silly question. In ml_ops.sh there are:
--num-executors ${SPK_EXEC} \
and:
--conf spark.dynamicAllocation.enabled=true \
which trigger the warning:
WARN spark.SparkContext: Dynamic Allocation and num executors both
set, thus dynamic allocation disabled.
Shouldn't we remove the "--num-executors" and add instead:
--conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
?
Thanks.
Giacomo
