Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/22252#discussion_r213774632
--- Diff: docs/configuration.md ---
@@ -152,7 +152,7 @@ of the most common options to set are:
<td><code>spark.driver.memory</code></td>
<td>1g</td>
<td>
- Amount of memory to use for the driver process, i.e. where
SparkContext is initialized, in MiB
+ Amount of memory to use for the driver process, i.e. where
SparkContext is initialized, in bytes
--- End diff --
Oh, hm.
```
$ ./bin/spark-shell --master local --driver-memory=1024
Error occurred during initialization of VM
Too small initial heap
$ ./bin/spark-shell --master local --conf spark.driver.memory=1024
Error occurred during initialization of VM
Too small initial heap
```
Although the option referenced above is used, it's used in YARN and K8S,
not in this code path.
Paging @vanzin for an opinion? I'd prefer to change docs rather than
behavior but it looks like we might have divergent behavior here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
