vinodkc commented on code in PR #41746:
URL: https://github.com/apache/spark/pull/41746#discussion_r1259173902
##########
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:
##########
@@ -143,9 +143,10 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
protected val addressToExecutorId = new HashMap[RpcAddress, String]
- // Spark configuration sent to executors. This is a lazy val so that
subclasses of the
- // scheduler can modify the SparkConf object before this view is created.
- private lazy val sparkProperties = scheduler.sc.conf.getAll
+ // Spark configuration sent to executors. This is a method so that
subclasses of the
+ // scheduler can modify the SparkConf object and latest spark
configurations
+ // shall be sent to executors.
+ private def sparkProperties = scheduler.sc.conf.getAll
Review Comment:
Changed `lazy val` to `def` to get the latest spark configurations. With
`lazy val`, new executor gets a snapshot of original spark conf and that will
cause propagation of the initial log level to new executors.
To avoid that scenario, sparkProperties should not be a lazy val.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
