tgravescs commented on a change in pull request #30204:
URL: https://github.com/apache/spark/pull/30204#discussion_r523093550
##########
File path: core/src/main/scala/org/apache/spark/resource/ResourceProfile.scala
##########
@@ -342,6 +367,110 @@ object ResourceProfile extends Logging {
rp.getTaskCpus.getOrElse(conf.get(CPUS_PER_TASK))
}
+ /**
+ * Get offHeap memory size from [[ExecutorResourceRequest]]
+ * return 0 if MEMORY_OFFHEAP_ENABLED is false.
+ */
+ private[spark] def executorOffHeapMemorySizeAsMb(sparkConf: SparkConf,
+ execRequest: ExecutorResourceRequest): Long = {
+ Utils.checkOffHeapEnabled(sparkConf, execRequest.amount)
+ }
+
+ private[spark] case class ExecutorResourcesOrDefaults(
+ cores: Int,
+ executorMemoryMiB: Long,
+ memoryOffHeapMiB: Long,
+ pysparkMemoryMiB: Long,
+ memoryOverheadMiB: Long,
+ totalMemMiB: Long,
+ customResources: Map[String, ExecutorResourceRequest])
+
+ private[spark] case class DefaultProfileExecutorResources(
Review comment:
So they were intentionally different. the default version doesn't have
total and couple of the values are optional. This is used just to store what
was in the default configs, not really do much logic on it for things like am I
in a pyspark app or does the cluster manager calculate overhead differently.
Once we get ExecutorResourcesOrDefaults everything there is required to be
filled in as its either what is specific in the profile or falls back to the
default value, or logic based on cluster manager inputs. Since its all fields
are always filled out and the cluster manager code doesn't need special code to
handle Optional fields.
But having said all that, really the main difference is in the optional of
memoryOverheadMiB. I could use ExecutorResourcesOrDefaults for everything as
long as I had a way for that to indicate not set. I could use -1 as special
value if we really want to but thought having 2 was clearer. But again I'm
fine to combine as its not a big deal, what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
