srowen commented on a change in pull request #23907: [SPARK-27003] [MINOR][DOC]Default value of spark.executor.instances is applicable… URL: https://github.com/apache/spark/pull/23907#discussion_r261350340
########## File path: docs/running-on-yarn.md ########## @@ -270,7 +270,7 @@ To use a custom metrics.properties for the application master and executors, upd <td><code>spark.executor.instances</code></td> <td><code>2</code></td> <td> - The number of executors for static allocation. With <code>spark.dynamicAllocation.enabled</code>, the initial set of executors will be at least this large. + The number of executors for static allocation. With <code>spark.dynamicAllocation.enabled</code>, the initial set of executors will be at least this large. Default value for static allocation is 2, whereas in dynamic allocation, is 0. Review comment: Looking at the code, it's more complicated. `spark.executor.instances` really has no default. It's taken to be 0 in many parts of the code if not specified. In practice it defaults to 2 from `SchedulerBackendUtils.getInitialTargetExecutorNumber`. I can see updating the docs to clarify that it has a default of 0, though that would be kind of surprising to find 2 executors then without specifying it. Maybe no default is the most correct answer, but again, 2 is what you end up with in practice. In any event I don't think it's accurate to say the default changes if you have dynamic allocation enabled. It doesn't apply, really. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
