GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/10237
[SPARK-12251] Document and improve off-heap memory configurations
This patch adds documentation for Spark configurations that affect off-heap
memory and makes some naming and validation improvements for those configs.
- Deprecated `spark.unsafe.offHeap` in favor of a new
`spark.memory.useOffHeap` configuration. The motivation behind this change is
to gather all memory-related configurations under the same prefix.
- Add a check which prevents users from setting
`spark.memory.useOffHeap=true` when `spark.memory.offHeapSize == 0`. After
SPARK-11389 (#9344), which was committed in Spark 1.6, Spark enforces a hard
limit on the amount of off-heap memory that it will allocate to tasks. As a
result, enabling off-heap execution memory without setting `offHeapSize` will
lead to immediate OOMs. The new configuration validation makes this scenario
easier to diagnose, helping to avoid user confusion.
- Document these configurations on the configuration page.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-12251
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10237.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10237
----
commit 30744be4f9318b8534b605502fc6a15cfb76b84e
Author: Josh Rosen <[email protected]>
Date: 2015-12-10T01:59:41Z
Deprecate spark.unsafe.offHeap in favor of spark.memory.useOffHeap.
commit f69f4383c66ff1b7bb8664fe2bdffba46254db1c
Author: Josh Rosen <[email protected]>
Date: 2015-12-10T02:15:30Z
Add test to check that offHeap memory size is configured when off heap is
used
commit 2dd2d9e289e5259393fa5ad9d662aa20aaf5a967
Author: Josh Rosen <[email protected]>
Date: 2015-12-10T02:20:49Z
Fix mis-ordered parameters
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
