Github user jcuquemelle commented on the issue:
https://github.com/apache/spark/pull/19881
more aggressive must be forbidden, because the setting of 1.0 gives enough
executors so that if the executor provisioning was perfect (e.g. all executors
were available at the same time) and the mapping of tasks to executors was
optimal, each executor core (or taskSlot as in the original naming) would
process exactly one task. If you ask for more executors, you're sure they will
be wasted.
Which is why it felt natural to have a divisor semantics, because it
implies the parameter can only be used to reduce parallelism.
How about fullParallelismThrottlingRate ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
