tgravescs commented on a change in pull request #29090:
URL: https://github.com/apache/spark/pull/29090#discussion_r462999630
##########
File path: docs/configuration.md
##########
@@ -249,7 +249,8 @@ of the most common options to set are:
<td>1g</td>
<td>
Amount of memory to use per executor process, in the same format as JVM
memory strings with
- a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>,
<code>2g</code>).
+ a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>,
<code>2g</code>) using
Review comment:
it seems we are a bit inconsistent across the documentation as wel
(pyspark.memory, memoryOverhead)l. other memory settings just say MiB unless
otherwise specified but don't mention the suffix options. I wonder if we make
them all consistent. Note one of the yarn configs says: Use lower-case
suffixes, e.g. k, m, g, t, and p, for kibi-, mebi-, gibi-, tebi-, and
pebibytes, respectively. but again doesn't say m is the default.
##########
File path:
launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java
##########
@@ -108,7 +108,7 @@
}
String mem = firstNonEmpty(memKey != null ? System.getenv(memKey) : null,
DEFAULT_MEM);
- cmd.add("-Xmx" + mem);
+ cmd.add("-Xmx" + addDefaultMSuffixIfNeeded(mem));
Review comment:
Note we should test those as well if you haven't already
##########
File path:
launcher/src/main/java/org/apache/spark/launcher/SparkClassCommandBuilder.java
##########
@@ -108,7 +108,7 @@
}
String mem = firstNonEmpty(memKey != null ? System.getenv(memKey) : null,
DEFAULT_MEM);
- cmd.add("-Xmx" + mem);
+ cmd.add("-Xmx" + addDefaultMSuffixIfNeeded(mem));
Review comment:
we should update the standalone docs as well for --memory
(SPARK_WORKER_MEMORY and SPARK_DAEMON_MEMORY) to say default to m and ideally
make consistent with above docs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
