Github user jcuquemelle commented on a diff in the pull request:
https://github.com/apache/spark/pull/19881#discussion_r175852906
--- Diff: docs/configuration.md ---
@@ -1795,6 +1796,19 @@ Apart from these, the following properties are also
available, and may be useful
Lower bound for the number of executors if dynamic allocation is
enabled.
</td>
</tr>
+<tr>
+ <td><code>spark.dynamicAllocation.fullParallelismDivisor</code></td>
+ <td>1</td>
+ <td>
+ By default, the dynamic allocation will request enough executors to
maximize the
+ parallelism according to the number of tasks to process. While this
minimizes the
+ latency of the job, with small tasks this setting wastes a lot of
resources due to
+ executor allocation overhead, as some executor might not even do any
work.
+ This setting allows to set a divisor that will be used to reduce the
number of
+ executors w.r.t. full parallelism
--- End diff --
done
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
