Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/8175#discussion_r37030412
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala ---
@@ -175,7 +175,9 @@ private[spark] object ShuffleMemoryManager {
val minPageSize = 1L * 1024 * 1024 // 1MB
val maxPageSize = 64L * minPageSize // 64MB
val cores = if (numCores > 0) numCores else
Runtime.getRuntime.availableProcessors()
- val safetyFactor = 8
+ // Because of rounding to next power of 2, we may have safetyFactor as
8 in worst case
--- End diff --
This is a new comment. For safetyFactor = 16, then size could be doubled by
rounding, so it seems like having a safetyFactor as 8.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
