Github user Sephiroth-Lin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6409#discussion_r31500453
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -91,51 +91,54 @@ private[spark] class Client(
* available in the alpha API.
*/
def submitApplication(): ApplicationId = {
- var appId: ApplicationId = null
+ // Before we submit current application, we cleanup staging director
as some old appStagingDir
+ // can not be deleted when those old jobs are failed or killed and so
on, please see SPARK-7705
+ // and SPARK-7503 for details.
+ cleanupStagingDir()
+
+ // Setup the credentials before doing anything else, so we have don't
have issues at any point.
+ setupCredentials()
+ yarnClient.init(yarnConf)
+ yarnClient.start()
+
+ logInfo("Requesting a new application from cluster with %d
NodeManagers"
+ .format(yarnClient.getYarnClusterMetrics.getNumNodeManagers))
+
+ // Get a new application from our RM
+ val newApp = yarnClient.createApplication()
+ val newAppResponse = newApp.getNewApplicationResponse()
+ val appId = newAppResponse.getApplicationId()
+
+ // Verify whether the cluster has enough resources for our AM
+ verifyClusterResources(newAppResponse)
+
+ // Set up the appropriate contexts to launch our AM
+ val containerContext = createContainerLaunchContext(newAppResponse)
+ val appContext = createApplicationSubmissionContext(newApp,
containerContext)
+
+ // Finally, submit and monitor the application
+ logInfo(s"Submitting application ${appId.getId} to ResourceManager")
+ yarnClient.submitApplication(appContext)
+ appId
+ }
+
+ /**
+ * Cleanup all subdirectory of SPARK_STAGING directory.
+ */
+ private def cleanupStagingDir(): Unit = {
+ val stagingDirPath = new Path(SPARK_STAGING)
--- End diff --
I'm so sorry, thank you for point out my mistake.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
