tgravescs commented on a change in pull request #28972:
URL: https://github.com/apache/spark/pull/28972#discussion_r450318645
##########
File path: core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
##########
@@ -245,6 +245,7 @@ object ResourceProfile extends Logging {
// Executor resources
val CORES = "cores"
val MEMORY = "memory"
+ val OFFHEAP_MEM = "offHeap"
val OVERHEAD_MEM = "memoryOverhead"
val PYSPARK_MEM = "pyspark.memory"
Review comment:
allSupportedExecutorResources needs to be updated to include this and
make sure we have a test for it.
##########
File path:
core/src/main/scala/org/apache/spark/resource/ExecutorResourceRequests.scala
##########
@@ -54,6 +54,19 @@ class ExecutorResourceRequests() extends Serializable {
this
}
+ /**
+ * Specify off heap memory. The value specified will be converted to MiB.
Review comment:
should document how this interacts with. spark.memory.offHeap.enabled
and I think we want to enforce that here. Or at least warn them. Right now you
have this working on yarn even if its disabled, where as before it would return
0 for this size if it was disabled.
##########
File path:
core/src/test/scala/org/apache/spark/resource/ResourceProfileSuite.scala
##########
@@ -55,6 +55,8 @@ class ResourceProfileSuite extends SparkFunSuite {
"pyspark memory empty if not specified")
assert(rprof.executorResources.get(ResourceProfile.OVERHEAD_MEM) == None,
"overhead memory empty if not specified")
+ assert(rprof.executorResources.get(ResourceProfile.OFFHEAP_MEM) == None,
+ "pyspark memory empty if not specified")
Review comment:
change pyspark memory to off heap
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
