Repository: spark Updated Branches: refs/heads/master 8d9495a8f -> 339859c4e
[SPARK-25261][MINOR][DOC] update the description for spark.executor|driver.memory in configuration.md ## What changes were proposed in this pull request? As described in [SPARK-25261](https://issues.apache.org/jira/projects/SPARK/issues/SPARK-25261)ï¼the unit of spark.executor.memory and spark.driver.memory is parsed as bytes in some cases if no unit specified, while in https://spark.apache.org/docs/latest/configuration.html#application-properties, they are descibed as MiB, which may lead to some misunderstandings. ## How was this patch tested? N/A Closes #22252 from ivoson/branch-correct-configuration. Lead-authored-by: huangtengfei02 <[email protected]> Co-authored-by: Huang Tengfei <[email protected]> Signed-off-by: Sean Owen <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/339859c4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/339859c4 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/339859c4 Branch: refs/heads/master Commit: 339859c4e4b27726ba8ce9a64451a981ef74de0c Parents: 8d9495a Author: huangtengfei02 <[email protected]> Authored: Fri Aug 31 09:06:38 2018 -0500 Committer: Sean Owen <[email protected]> Committed: Fri Aug 31 09:06:38 2018 -0500 ---------------------------------------------------------------------- docs/configuration.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/339859c4/docs/configuration.md ---------------------------------------------------------------------- diff --git a/docs/configuration.md b/docs/configuration.md index b5ff426..f344bcd 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -152,8 +152,9 @@ of the most common options to set are: <td><code>spark.driver.memory</code></td> <td>1g</td> <td> - Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in MiB - unless otherwise specified (e.g. <code>1g</code>, <code>2g</code>). + Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the + same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") + (e.g. <code>512m</code>, <code>2g</code>). <br /> <em>Note:</em> In client mode, this config must not be set through the <code>SparkConf</code> directly in your application, because the driver JVM has already started at that point. @@ -175,8 +176,8 @@ of the most common options to set are: <td><code>spark.executor.memory</code></td> <td>1g</td> <td> - Amount of memory to use per executor process, in MiB unless otherwise specified. - (e.g. <code>2g</code>, <code>8g</code>). + Amount of memory to use per executor process, in the same format as JVM memory strings with + a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>, <code>2g</code>). </td> </tr> <tr> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
