Github user tgravescs commented on a diff in the pull request:
    --- Diff: docs/ ---
    @@ -1795,6 +1796,19 @@ Apart from these, the following properties are also 
available, and may be useful
         Lower bound for the number of executors if dynamic allocation is 
    +  <td><code>spark.dynamicAllocation.fullParallelismDivisor</code></td>
    +  <td>1</td>
    +  <td>
    +    By default, the dynamic allocation will request enough executors to 
maximize the 
    +    parallelism according to the number of tasks to process. While this 
minimizes the 
    +    latency of the job, with small tasks this setting wastes a lot of 
resources due to
    +    executor allocation overhead, as some executor might not even do any 
    +    This setting allows to set a divisor that will be used to reduce the 
number of
    +    executors w.r.t. full parallelism
    +    Defaults to 1.0
    --- End diff --
    I think we should define that maxExecutors trumps this setting.  
    If I have 10000 tasks, divisor 2, I would expect 5000 executors, but if max 
executors is 1000, that is all I get. 
    we should add a test for this interaction as well


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to