This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0d77d57 [MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory'
is dependent on 'resource'
0d77d57 is described below
commit 0d77d575e14e535fbe29b42e5612f3ddc64d42f4
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Thu Jan 31 15:51:40 2019 +0800
[MINOR][DOCS] Add a note that 'spark.executor.pyspark.memory' is dependent
on 'resource'
## What changes were proposed in this pull request?
This PR adds a note that explicitly `spark.executor.pyspark.memory` is
dependent on resource module's behaviours at Python memory usage.
For instance, I at least see some difference at
https://github.com/apache/spark/pull/21977#discussion_r251220966
## How was this patch tested?
Manually built the doc.
Closes #23664 from HyukjinKwon/note-resource-dependent.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
docs/configuration.md | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 806e16a..5b5891e 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -190,8 +190,10 @@ of the most common options to set are:
and it is up to the application to avoid exceeding the overhead memory
space
shared with other non-JVM processes. When PySpark is run in YARN or
Kubernetes, this memory
is added to executor resource requests.
-
- NOTE: Python memory usage may not be limited on platforms that do not
support resource limiting, such as Windows.
+ <br/>
+ <em>Note:</em> This feature is dependent on Python's `resource` module;
therefore, the behaviors and
+ limitations are inherited. For instance, Windows does not support resource
limiting and actual
+ resource is not limited on MacOS.
</td>
</tr>
<tr>
@@ -223,7 +225,8 @@ of the most common options to set are:
stored on disk. This should be on a fast, local disk in your system. It
can also be a
comma-separated list of multiple directories on different disks.
- NOTE: In Spark 1.0 and later this will be overridden by SPARK_LOCAL_DIRS
(Standalone), MESOS_SANDBOX (Mesos) or
+ <br/>
+ <em>Note:</em> This will be overridden by SPARK_LOCAL_DIRS (Standalone),
MESOS_SANDBOX (Mesos) or
LOCAL_DIRS (YARN) environment variables set by the cluster manager.
</td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
