Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/894#discussion_r13220194
--- Diff:
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala
---
@@ -92,21 +92,22 @@ class ExecutorLauncher(args:
ApplicationMasterArguments, conf: Configuration, sp
appAttemptId = getApplicationAttemptId()
resourceManager = registerWithResourceManager()
- val appMasterResponse: RegisterApplicationMasterResponse =
registerApplicationMaster()
- // Compute number of threads for akka
- val minimumMemory =
appMasterResponse.getMinimumResourceCapability().getMemory()
-
- if (minimumMemory > 0) {
- val mem = args.executorMemory + YarnAllocationHandler.MEMORY_OVERHEAD
- val numCore = (mem / minimumMemory) + (if (0 != (mem %
minimumMemory)) 1 else 0)
-
- if (numCore > 0) {
- // do not override - hits
https://issues.apache.org/jira/browse/HADOOP-8406
- // TODO: Uncomment when hadoop is on a version which has this
fixed.
- // args.workerCores = numCore
- }
- }
+ // // do not override - hits
https://issues.apache.org/jira/browse/HADOOP-8406
+ // // TODO: Uncomment when hadoop is on a version which has this
fixed.
+ // val appMasterResponse: RegisterApplicationMasterResponse =
registerApplicationMaster()
+ // // Compute number of threads for akka
+ // val minimumMemory =
appMasterResponse.getMinimumResourceCapability().getMemory()
+ // // Additional memory overhead - in mb.
+ // val memoryOverhead =
sparkConf.getInt("spark.yarn.container.memoryOverhead", 384)
+ //
+ // if (minimumMemory > 0) {
+ // val mem = args.executorMemory + memoryOverhead
+ // val numCore = (mem / minimumMemory) + (if (0 != (mem %
minimumMemory)) 1 else 0)
+ // if (numCore > 0) {
+ // args.workerCores = numCore
+ // }
+ // }
--- End diff --
You have commented out registering application master, etc ? Are you sure
this PR allows yarn mode to still work ? Was this tested ?
Btw, please do not reorder the comments - the HADOOP-8406 was specific
reference to numCore only; not rest of the code.
(Different issue that it is probably not relevant anymore. Any thoughts
@tgravescs ?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
