[
https://issues.apache.org/jira/browse/YARN-8725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zac Zhou updated YARN-8725:
---------------------------
Description:
Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and
PRIMARY_WORKER-launch-script****.sh to staging dir.
The core-site.xml, hdfs-site.xml and job.info would be overwritten if a job is
submitted multiple times.
But PRIMARY_WORKER-launch-script****.sh would not be overwritten, as it has
random numbers in its name.
The files in the staging dir are as follows：
{code:java}
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:11
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:02
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:06
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh
-rw-r----- 2 hadoop hdfs 15225 2018-08-17 18:46
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-16 20:48
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 14:53
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:16
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh
-rw-r----- 2 hadoop hdfs 8815 2018-08-27 15:54
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml
-rw-r----- 2 hadoop hdfs 11583 2018-08-27 15:54
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml
-rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
{code}
We should stop the staging dir from growing or have a way to clean it up
was:
Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and
PRIMARY_WORKER-launch-script****.sh to staging dir.
core-site.xml, hdfs-site.xml and job.info would be overwritten if a job is
submitted multiple times.
But PRIMARY_WORKER-launch-script****.sh would not be overwritten, as it has
random numbers in its name.
The files in the staging dir are as follows：
{code:java}
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:11
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:02
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:06
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh
-rw-r----- 2 hadoop hdfs 15225 2018-08-17 18:46
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-16 20:48
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 14:53
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh
-rw-r----- 2 hadoop hdfs 580 2018-08-17 10:16
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh
-rw-r----- 2 hadoop hdfs 8815 2018-08-27 15:54
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml
-rw-r----- 2 hadoop hdfs 11583 2018-08-27 15:54
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml
-rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56
hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
{code}
We should stop the staging dir from growing or have a way to clean it up
> Submarine job staging directory has a lot of useless
> PRIMARY_WORKER-launch-script-***.sh scripts when submitting a job multiple
> times
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8725
> URL: https://issues.apache.org/jira/browse/YARN-8725
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Zac Zhou
> Assignee: Zac Zhou
> Priority: Major
>
> Submarine jobs upload core-site.xml, hdfs-site.xml, job.info and
> PRIMARY_WORKER-launch-script****.sh to staging dir.
> The core-site.xml, hdfs-site.xml and job.info would be overwritten if a job
> is submitted multiple times.
> But PRIMARY_WORKER-launch-script****.sh would not be overwritten, as it has
> random numbers in its name.
> The files in the staging dir are as follows：
> {code:java}
> -rw-r----- 2 hadoop hdfs 580 2018-08-17 10:11
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script6954941665090337726.sh
> -rw-r----- 2 hadoop hdfs 580 2018-08-17 10:02
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script7037369696166769734.sh
> -rw-r----- 2 hadoop hdfs 580 2018-08-17 10:06
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8047707294763488040.sh
> -rw-r----- 2 hadoop hdfs 15225 2018-08-17 18:46
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8122565781159446375.sh
> -rw-r----- 2 hadoop hdfs 580 2018-08-16 20:48
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script8598604480700049845.sh
> -rw-r----- 2 hadoop hdfs 580 2018-08-17 14:53
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script971703616848859353.sh
> -rw-r----- 2 hadoop hdfs 580 2018-08-17 10:16
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/PRIMARY_WORKER-launch-script990214235580089093.sh
> -rw-r----- 2 hadoop hdfs 8815 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/core-site.xml
> -rw-r----- 2 hadoop hdfs 11583 2018-08-27 15:54
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/hdfs-site.xml
> -rw-rw-rw- 2 hadoop hdfs 846 2018-08-22 10:56
> hdfs://submarine/user/hadoop/submarine/jobs/standlone-tf/staging/job.info
> {code}
>
> We should stop the staging dir from growing or have a way to clean it up
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
