yifeih commented on a change in pull request #25823:
[SPARK-28211][Core][Shuffle] Propose Shuffle Driver Components API
URL: https://github.com/apache/spark/pull/25823#discussion_r329834219
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -524,6 +528,19 @@ class SparkContext(config: SparkConf) extends Logging {
executorEnvs ++= _conf.getExecutorEnv
executorEnvs("SPARK_USER") = sparkUser
+ val configuredPluginClasses = conf.get(SHUFFLE_IO_PLUGIN_CLASS)
+ val maybeIO = Utils.loadExtensions(
+ classOf[ShuffleDataIO], Seq(configuredPluginClasses), conf)
+ require(maybeIO.nonEmpty, s"At least one valid shuffle plugin must be
specified by config " +
+ s"${SHUFFLE_IO_PLUGIN_CLASS.key}, but $configuredPluginClasses resulted
in zero valid " +
+ s"plugins.")
+ require(maybeIO.size == 1,
+ s"Specified shuffle plugin(s) $configuredPluginClasses resulted in
more than one valid " +
+ s"plugin, but only one valid plugin should be specified")
+ _shuffleDriverComponents = maybeIO.head.driver()
+ _shuffleDriverComponents.initializeApplication().asScala.foreach {
+ case (k, v) => _conf.set(ShuffleDataIO.SHUFFLE_SPARK_CONF_PREFIX + k, v)
}
Review comment:
That's one use case I didn't think about. Intuitively, I guess it would make
sense for the user to be able to override the configs set by the plugin, since
the alternative might be confusing to a user. In that case, I think it would
make sense to log the configs, and also log when the value returned by the
driver won't take effect due to user overrides for that particular application.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
