[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
angerszhu updated SPARK-38910: ------------------------------ Description: {code:java} ShutdownHookManager.addShutdownHook(priority) { () => try { val maxAppAttempts = client.getMaxRegAttempts(sparkConf, yarnConf) val isLastAttempt = appAttemptId.getAttemptId() >= maxAppAttempts if (!finished) { // The default state of ApplicationMaster is failed if it is invoked by shut down hook. // This behavior is different compared to 1.x version. // If user application is exited ahead of time by calling System.exit(N), here mark // this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call // System.exit(0) to terminate the application. finish(finalStatus, ApplicationMaster.EXIT_EARLY, "Shutdown hook called before final status was reported.") } if (!unregistered) { // we only want to unregister if we don't want the RM to retry if (finalStatus == FinalApplicationStatus.SUCCEEDED || isLastAttempt) { unregister(finalStatus, finalMsg) cleanupStagingDir(new Path(System.getenv("SPARK_YARN_STAGING_DIR"))) } } } catch { case e: Throwable => logWarning("Ignoring Exception while stopping ApplicationMaster from shutdown hook", e) } }{code} unregister may throw exception, clean staging dir should before unregister. was: {code:java} {code} Not clean the staging dir when match case {code:jave} !launcherBackend.isConnected() && fireAndForget {code} > Clean sparkStaging dir should before unregister() > ------------------------------------------------- > > Key: SPARK-38910 > URL: https://issues.apache.org/jira/browse/SPARK-38910 > Project: Spark > Issue Type: Task > Components: YARN > Affects Versions: 3.2.1, 3.3.0 > Reporter: angerszhu > Priority: Major > > {code:java} > ShutdownHookManager.addShutdownHook(priority) { () => > try { > val maxAppAttempts = client.getMaxRegAttempts(sparkConf, yarnConf) > val isLastAttempt = appAttemptId.getAttemptId() >= maxAppAttempts > if (!finished) { > // The default state of ApplicationMaster is failed if it is > invoked by shut down hook. > // This behavior is different compared to 1.x version. > // If user application is exited ahead of time by calling > System.exit(N), here mark > // this application as failed with EXIT_EARLY. For a good > shutdown, user shouldn't call > // System.exit(0) to terminate the application. > finish(finalStatus, > ApplicationMaster.EXIT_EARLY, > "Shutdown hook called before final status was reported.") > } > if (!unregistered) { > // we only want to unregister if we don't want the RM to retry > if (finalStatus == FinalApplicationStatus.SUCCEEDED || > isLastAttempt) { > unregister(finalStatus, finalMsg) > cleanupStagingDir(new > Path(System.getenv("SPARK_YARN_STAGING_DIR"))) > } > } > } catch { > case e: Throwable => > logWarning("Ignoring Exception while stopping ApplicationMaster > from shutdown hook", e) > } > }{code} > unregister may throw exception, clean staging dir should before unregister. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org