[
https://issues.apache.org/jira/browse/YARN-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627433#comment-16627433
]
Antal Bálint Steinbach commented on YARN-8806:
----------------------------------------------
Hi [~yuan_zac] ,
Thank you for the patch. I would like to add some minor comments:
1. ClientContext.dirDeleteOnExit can be private
2.
{code:java}
public File getLocalStagingArea(String jobName, boolean create) throws
IOException {
return getLocalStagingArea(jobName, create, true);
}
{code}
Is never used and can be removed.
3. There is a typo here in SubmarineConfiguration.java:
{code:java}
public static final String LOCAL_STAGING_DIR = PREFIX +
"local.staing.dir";{code}
4.
{code:java}
public File getLocalStagingArea(String jobName, boolean create, boolean
deleteOnExit) throws IOException{code}
parameter create is always true, it can be removed.
5. getLocalStagingArea does not only return the local staging area but sets up
some delete on shutdown hook. The name of the function is confusing. I would
remove the delete logic from here and call that directly where it is required.
That way you can do the configuration check inside delete.
6. In getLocalStagingArea you have the parameter deleteOnExit and in the
configuration you have isKeepLocalStagingDir. I would harmonize these 2 and use
them directly without using the logical not operator.
> Enable local staging directory and clean it up when submarine job is submitted
> ------------------------------------------------------------------------------
>
> Key: YARN-8806
> URL: https://issues.apache.org/jira/browse/YARN-8806
> Project: Hadoop YARN
> Issue Type: Sub-task
> Environment: In the /tmp dir, there are launch scripts which are not
> cleaned up as follows:
> -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46
> PRIMARY_WORKER-launch-script8635233314077649086.sh
> -rw-r--r-- 1 hadoop netease 1100 Sep 18 10:46
> WORKER-launch-script129488020578466938.sh
> -rw-r--r-- 1 hadoop netease 1028 Sep 18 10:46
> PS-launch-script471092031021738136.sh
> Reporter: Zac Zhou
> Assignee: Zac Zhou
> Priority: Major
> Attachments: YARN-8806.001.patch, YARN-8806.002.patch,
> YARN-8806.003.patch, YARN-8806.004.patch, YARN-8806.005.patch
>
>
> YarnServiceJobSubmitter.generateCommandLaunchScript creates container launch
> scripts in the local filesystem. Container launch scripts would be uploaded
> to hdfs staging dir, but would not be not deleted after the job is submitted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
