dongjoon-hyun commented on a change in pull request #29477:
URL: https://github.com/apache/spark/pull/29477#discussion_r475289179
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
##########
@@ -323,4 +324,25 @@ private[spark] object KubernetesUtils extends Logging {
.build()
}
}
+
+ /**
+ * Convert MEMORY_OFFHEAP_SIZE to MB Unit, return 0 if
MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def executorOffHeapMemorySizeAsMb(kubernetesConf: KubernetesExecutorConf):
Int = {
+ val sizeInMB =
Utils.memoryStringToMb(kubernetesConf.get(MEMORY_OFFHEAP_SIZE).toString)
+ checkOffHeapEnabled(kubernetesConf, sizeInMB).toInt
+ }
+
+ /**
+ * return 0 if MEMORY_OFFHEAP_ENABLED is false.
+ */
+ def checkOffHeapEnabled(kubernetesConf: KubernetesExecutorConf, offHeapSize:
Long): Long = {
+ if (kubernetesConf.get(MEMORY_OFFHEAP_ENABLED)) {
+ require(offHeapSize > 0,
+ s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
Review comment:
This looks like inconsistent and redundant. We already centralize this
kind of check at `ConfigBuilder` and Apache Spark already check non-negativity
like the following. Also, I believe 0 is a valid value when
`MEMORY_OFFHEAP_ENABLED=true`.
```scala
private[spark] val MEMORY_OFFHEAP_SIZE =
ConfigBuilder("spark.memory.offHeap.size")
.doc("The absolute amount of memory which can be used for off-heap
allocation, " +
" in bytes unless otherwise specified. " +
"This setting has no impact on heap memory usage, so if your
executors' total memory " +
"consumption must fit within some hard limit then be sure to shrink
your JVM heap size " +
"accordingly. This must be set to a positive value when
spark.memory.offHeap.enabled=true.")
.version("1.6.0")
.bytesConf(ByteUnit.BYTE)
.checkValue(_ >= 0, "The off-heap memory size must not be negative")
.createWithDefault(0)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
