Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12571#discussion_r61133560
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -66,12 +66,20 @@ private[spark] class SparkDeploySchedulerBackend(
"--cores", "{{CORES}}",
"--app-id", "{{APP_ID}}",
"--worker-url", "{{WORKER_URL}}")
- val extraJavaOpts =
sc.conf.getOption("spark.executor.extraJavaOptions")
+ var extraJavaOpts =
sc.conf.getOption("spark.executor.extraJavaOptions")
.map(Utils.splitCommandString).getOrElse(Seq.empty)
val classPathEntries =
sc.conf.getOption("spark.executor.extraClassPath")
.map(_.split(java.io.File.pathSeparator).toSeq).getOrElse(Nil)
val libraryPathEntries =
sc.conf.getOption("spark.executor.extraLibraryPath")
.map(_.split(java.io.File.pathSeparator).toSeq).getOrElse(Nil)
+ // Add GC Limit options if they are not present
+ val extraJavaOptsAsStr = extraJavaOpts.mkString(" ")
+ if (!extraJavaOptsAsStr.contains("-XX:GCTimeLimit")) {
+ extraJavaOpts :+= Utils.getGCTimeLimitOption
--- End diff --
The launcher code is not really used for creating the command line for
executors. It contains code that might be useful there, though.
For this particular case, I'd take a look at
`CommandBuilderUtils.addPermGenSizeOpt`. That's very similar to what is needed
here. At the very least, I'd create a helper method instead of repeating this
code in 3 different places.
You also need to double check that the option won't cause issues with
non-Oracle JVMs.
Finally, instead of creating a string, you could instead use
`extraJavaOpts.exists(_.startsWith("-XX:GCTimeLimit=")`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
