tgravescs commented on a change in pull request #26682: [SPARK-29306][CORE]
Stage Level Sched: Executors need to track what ResourceProfile they are
created with
URL: https://github.com/apache/spark/pull/26682#discussion_r366900078
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
##########
@@ -272,11 +272,16 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
removeWorker(workerId, host, message)
context.reply(true)
- case RetrieveSparkAppConfig =>
+ case RetrieveSparkAppConfig(resourceProfileId) =>
+ // note this will be updated in later prs to get the ResourceProfile
from a
+ // ResourceProfileManager that matches the resource profile id
+ // for now just use default profile
+ val rp = ResourceProfile.getOrCreateDefaultProfile(conf)
val reply = SparkAppConfig(
sparkProperties,
SparkEnv.get.securityManager.getIOEncryptionKey(),
- Option(delegationTokens.get()))
+ Option(delegationTokens.get()),
+ rp)
Review comment:
They can but its really just extra code and an extra code path to have to
maintain. I specifically removed the old code that was generating the default
profile on the executor side from the configs because the driver already
calculated it and there isn't a reason to have 2 paths to calculate it. This
way it all works the same. Was there something specific here you are concerned
with?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
