panbingkun created SPARK-38960:
----------------------------------
Summary: Spark should fail fast if initial memory too large(set by
"spark.executor.extraJavaOptions") for executor to start
Key: SPARK-38960
URL: https://issues.apache.org/jira/browse/SPARK-38960
Project: Spark
Issue Type: Improvement
Components: Spark Core, Spark Submit, YARN
Affects Versions: 3.4.0
Reporter: panbingkun
Fix For: 3.4.0
if you set initial memory(set by "spark.executor.extraJavaOptions=-Xms\{XXX}G"
) larger than maximum memory(set by "spark.executor.memory")
Eg.
spark.executor.memory=1G
spark.executor.extraJavaOptions=-Xms2G
from the driver process you just see executor failures with no warning, since
the more meaningful errors are buried in the executor logs.
Eg., on Yarn, you see:
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size
Instead we should just fail fast with a clear error message in the driver logs.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
