[
https://issues.apache.org/jira/browse/SPARK-50186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-50186:
-----------------------------------
Labels: pull-request-available (was: )
> Remove Hardcoded OnOutOfMemoryError Setting in Executor Startup Script
> ----------------------------------------------------------------------
>
> Key: SPARK-50186
> URL: https://issues.apache.org/jira/browse/SPARK-50186
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.3.1, 4.0.0
> Reporter: Ruochen Zou
> Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-10-31-14-14-06-723.png,
> image-2024-10-31-14-17-51-349.png
>
>
> Currently, the Executor startup script hardcodes the addition of
> {{{}-XX:OnOutOfMemoryError='kill %p'{}}}, which causes the process to be
> killed when the Executor encounters an OOM (Out Of Memory) error.
> {code:java}
> // code in YarnSparkHadoopUtil
> private[yarn] def addOutOfMemoryErrorArgument(javaOpts: ListBuffer[String]):
> Unit = {
> if (!javaOpts.exists(_.contains("-XX:OnOutOfMemoryError"))) {
> if (Utils.isWindows) {
> javaOpts += escapeForShell("-XX:OnOutOfMemoryError=taskkill /F /PID
> %%%%p")
> } else {
> javaOpts += "-XX:OnOutOfMemoryError='kill %p'"
> }
> }
> }{code}
> As a result, the YarnAllocator receives an exit code of 143 and is unable to
> accurately determine the reason for the Executor's termination based on this
> exit code. Moreover, the CoarseGrainedExecutorBackend cannot guarantee that
> StatusUpdate messages are sent to the Driver before the process is killed.
> Could we remove this setting, since users can set it via the
> {{spark.executor.extraJavaOptions}} parameter if necessary?
> Executor log:
> !image-2024-10-31-14-14-06-723.png!
>
> Driver log:
> !image-2024-10-31-14-17-51-349.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
