rdblue commented on a change in pull request #21977: [SPARK-25004][CORE] Add
spark.executor.pyspark.memory limit.
URL: https://github.com/apache/spark/pull/21977#discussion_r251249517
##########
File path: docs/configuration.md
##########
@@ -179,6 +179,18 @@ of the most common options to set are:
(e.g. <code>2g</code>, <code>8g</code>).
</td>
</tr>
+<tr>
+ <td><code>spark.executor.pyspark.memory</code></td>
+ <td>Not set</td>
+ <td>
+ The amount of memory to be allocated to PySpark in each executor, in MiB
+ unless otherwise specified. If set, PySpark memory for an executor will be
+ limited to this amount. If not set, Spark will not limit Python's memory
use
Review comment:
Sounds fine to me. I tested in a linux environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
