[
https://issues.apache.org/jira/browse/SPARK-28021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-28021.
----------------------------------
Resolution: Invalid
Let's ask a question to mailing list before filing it as an issue. See
https://spark.apache.org/community.html
> A unappropriate exception in StaticMemoryManager.getMaxExecutionMemory
> ----------------------------------------------------------------------
>
> Key: SPARK-28021
> URL: https://issues.apache.org/jira/browse/SPARK-28021
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 2.4.3
> Reporter: child2d
> Priority: Minor
>
> When i review StaticMemoryManager.scala, there comes a question to me.
> {code:java}
> private def getMaxExecutionMemory(conf: SparkConf): Long = {
> val systemMaxMemory = conf.getLong("spark.testing.memory",
> Runtime.getRuntime.maxMemory)
> if (systemMaxMemory < MIN_MEMORY_BYTES) {
> throw new IllegalArgumentException(s"System memory $systemMaxMemory must
> " +
> s"be at least $MIN_MEMORY_BYTES. Please increase heap size using the
> --driver-memory " +
> s"option or spark.driver.memory in Spark configuration.")
> }
> if (conf.contains("spark.executor.memory")) {
> val executorMemory = conf.getSizeAsBytes("spark.executor.memory")
> if (executorMemory < MIN_MEMORY_BYTES) {
> throw new IllegalArgumentException(s"Executor memory $executorMemory
> must be at least " +
> s"$MIN_MEMORY_BYTES. Please increase executor memory using the " +
> s"--executor-memory option or spark.executor.memory in Spark
> configuration.")
> }
> }
> val memoryFraction = conf.getDouble("spark.shuffle.memoryFraction", 0.2)
> val safetyFraction = conf.getDouble("spark.shuffle.safetyFraction", 0.8)
> (systemMaxMemory * memoryFraction * safetyFraction).toLong
> }
> {code}
> When a executor tries to getMaxExecutionMemory, it should set systemMaxMemory
> by using Runtime.getRuntime.maxMemory first, then compares the value between
> systemMaxMemory and MIN_MEMORY_BYTES.
> If the compared value is true, program thows an exception to remind user to
> increase heap size by using --driver-memory.
> I wonder if it is wrong because the heap size of executors are setted by
> --executor-memory?
> Although there is another exception about adjusting executor's memory below,
> i just think that the first exception may be not appropriate.
> Thanks for answering my question!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
