Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18555#discussion_r126293363
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -35,10 +35,21 @@ package object config {
ConfigBuilder(SparkLauncher.DRIVER_EXTRA_LIBRARY_PATH).stringConf.createOptional
private[spark] val DRIVER_USER_CLASS_PATH_FIRST =
-
ConfigBuilder("spark.driver.userClassPathFirst").booleanConf.createWithDefault(false)
+ ConfigBuilder("spark.driver.userClassPathFirst")
+ .doc("Whether to give user-added jars precedence over Spark's own
jars " +
+ "when loading classes in the driver. This feature can be used to
mitigate " +
+ "conflicts between Spark's dependencies and user dependencies. " +
+ "It is currently an experimental feature. This is used in cluster
mode only. ")
+ .booleanConf
+ .createWithDefault(false)
private[spark] val DRIVER_MEMORY = ConfigBuilder("spark.driver.memory")
+ .doc("Amount of memory to use for the driver process, i.e. " +
+ "where SparkContext is initialized. (e.g. 512, 1g, 2g). ")
.bytesConf(ByteUnit.MiB)
+ .checkValue(v => v > 0 && v <= Int.MaxValue / (1024 * 1024),
+ "The driver memory must be greater than 0MB " +
--- End diff --
Is it more accurate to use MiB instead of MB here? also please add WS like
`0 MiB`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
