panbingkun opened a new pull request, #36273:
URL: https://github.com/apache/spark/pull/36273
### What changes were proposed in this pull request?
Added an exception to be thrown in SparkConf.validateSettings if set initial
memory(set by "spark.executor.extraJavaOptions=-Xms{XXX}G" ) larger than
maximum memory(set by "spark.executor.memory")
### How was this patch tested?
This patch was tested manually by passing in config options to Spark submit.
I also added a test in SparkConfSuite
**Test with large initial executor memory:**
./bin/spark-submit --master yarn --deploy-mode client --conf
spark.executor.memory=1G --conf spark.executor.extraJavaOptions=-Xms2G --class
org.apache.spark.examples.SparkPi ./examples/jars/spark-examples_*.jar 10
Exception in thread "main" java.lang.Exception: The initial memory value
2048Mb (set by spark.executor.extraJavaOptions=-Xms2G) must be smaller than the
value 1024Mb (set by spark.executor.memory=1G).
at
org.apache.spark.SparkConf.$anonfun$validateSettings$4(SparkConf.scala:541)
at
org.apache.spark.SparkConf.$anonfun$validateSettings$4$adapted(SparkConf.scala:522)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:522)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:390)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2706)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:949)
at scala.Option.getOrElse(Option.scala:189)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:943)
at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30)
at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
