Github user bjornjon commented on a diff in the pull request:
https://github.com/apache/spark/pull/12395#discussion_r60559808
--- Diff:
core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala ---
@@ -187,7 +187,6 @@ object UnifiedMemoryManager {
// This serves a function similar to `spark.memory.fraction`, but
guarantees that we reserve
// sufficient memory for the system even for small heaps. E.g. if we
have a 1GB JVM, then
// the memory used for execution and storage will be (1024 - 300) * 0.75
= 543MB by default.
- private val RESERVED_SYSTEM_MEMORY_BYTES = 300 * 1024 * 1024
--- End diff --
The problem is that when users intentionally or accidentally (the default
value for driver and executor memory is bytes) specify a very low memory value
for the driver or executors, the application fails without providing any
descriptive error message when using legacy mode (StaticMemoryManager). When
legacy mode is set to false (UnifiedMemoryManager), this very descriptive
IllegalArgumentException is thrown, indicating whether its driver or executor
memory that you need to increase. The intention here is to unify the behavior
between the memory managers to prevent users from needing assistance when
running in legacy mode.
The suggestion is to use the same min as the UnifiedMemoryManager uses, for
consistency, but a smaller min for the StaticMemoryManager could also be used.
There is no minimum enforced for the StaticMemoryManager today.
The decision that needs to be made revolves around if a minimum should be
enforced, like it is for the UnifiedMemoryManager and then what that minimum
should be.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
