xkrogen commented on a change in pull request #32746:
URL: https://github.com/apache/spark/pull/32746#discussion_r645145321
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -2148,4 +2148,37 @@ package object config {
// batch of block will be loaded in memory with memory mapping, which
has higher overhead
// with small MB sized chunk of data.
.createWithDefaultString("3m")
+
+ private[spark] val JAR_IVY_REPO_PATH =
+ ConfigBuilder("spark.jars.ivy")
+ .version("1.3.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val JAR_IVY_SETTING_PATH =
+ ConfigBuilder("spark.jars.ivySettings")
+ .version("2.2.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val JAR_PACKAGES =
+ ConfigBuilder("spark.jars.packages")
+ .version("1.5.0")
+ .stringConf
+ .toSequence
+ .createWithDefault(Nil)
+
+ private[spark] val JAR_PACKAGES_EXCLUSIONS =
+ ConfigBuilder("spark.jars.excludes")
+ .version("1.5.0")
+ .stringConf
+ .toSequence
+ .createWithDefault(Nil)
+
+ private[spark] val JAR_REPOSITORIES =
+ ConfigBuilder("spark.jars.repositories")
Review comment:
Can we add at least a simple `doc()` for these?
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -2148,4 +2148,37 @@ package object config {
// batch of block will be loaded in memory with memory mapping, which
has higher overhead
// with small MB sized chunk of data.
.createWithDefaultString("3m")
+
+ private[spark] val JAR_IVY_REPO_PATH =
+ ConfigBuilder("spark.jars.ivy")
+ .version("1.3.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val JAR_IVY_SETTING_PATH =
+ ConfigBuilder("spark.jars.ivySettings")
+ .version("2.2.0")
+ .stringConf
+ .createOptional
+
+ private[spark] val JAR_PACKAGES =
+ ConfigBuilder("spark.jars.packages")
+ .version("1.5.0")
+ .stringConf
+ .toSequence
+ .createWithDefault(Nil)
+
+ private[spark] val JAR_PACKAGES_EXCLUSIONS =
+ ConfigBuilder("spark.jars.excludes")
+ .version("1.5.0")
+ .stringConf
+ .toSequence
+ .createWithDefault(Nil)
+
+ private[spark] val JAR_REPOSITORIES =
+ ConfigBuilder("spark.jars.repositories")
+ .version("2.3.0")
+ .stringConf
+ .toSequence
+ .createWithDefault(Nil)
Review comment:
To be more consistent with the way they're used (e.g. in
`SparkSubmitArguments`), I wonder if these should all be `Optional[String]`
configs instead of `Seq[String]`?
##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -605,13 +607,13 @@ private[spark] class SparkSubmit extends Logging {
// Propagate attributes for dependency resolution at the driver side
OptionAssigner(args.packages, STANDALONE | MESOS | KUBERNETES,
- CLUSTER, confKey = "spark.jars.packages"),
+ CLUSTER, confKey = config.JAR_PACKAGES.key),
Review comment:
for all of the references in this file, there is already a wildcard
import for `config`, so you should be able to reference the names directly,
right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
