[
https://issues.apache.org/jira/browse/SPARK-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144988#comment-15144988
]
Josh Rosen commented on SPARK-12251:
------------------------------------
It's off by default; the 1.6.0 documentation was wrong and this has been
corrected for 1.61.
For "certain operations", this means places in Spark SQL which allocate
MemoryBlock / MemoryPage. Grep through the code and look for call sites of
TaskMemoryManager.allocatePage(), etc. if you want to get a comprehensive
picture of where this is used.
> Document Spark 1.6's off-heap memory configurations and add config validation
> -----------------------------------------------------------------------------
>
> Key: SPARK-12251
> URL: https://issues.apache.org/jira/browse/SPARK-12251
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Fix For: 1.6.0
>
>
> We need to document the new off-heap memory limit configurations which were
> added in Spark 1.6, add simple configuration validation (for instance, you
> shouldn't be able to enable off-heap execution when the off-heap memory limit
> is zero), and alias the old and confusing `spark.unsafe.offHeap`
> configuration to something that lives in the `spark.memory` namespace.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
