[
https://issues.apache.org/jira/browse/BIGTOP-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13527000#comment-13527000
]
Mark Grover commented on BIGTOP-330:
------------------------------------
I tried this with YARN.
Here are the contents of /tmp directory after running at job at 3:26
{code}
vagrant@lucid64:/tmp$ ls -lrt
total 72
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 01:30 forrest-vagrant
drwxr-xr-x 4 mapred mapred 4096 2012-12-08 02:50
Jetty_0_0_0_0_50030_job____yn7qmk
drwxr-xr-x 2 root root 4096 2012-12-08 02:50 hsperfdata_root
drwxr-xr-x 3 hdfs hdfs 4096 2012-12-08 03:02 hadoop-hdfs
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:02 nm-local-dir
drwxr-xr-x 2 yarn yarn 4096 2012-12-08 03:02 logs
drwxr-xr-x 3 mapred mapred 4096 2012-12-08 03:02 hadoop-yarn
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:04
Jetty_0_0_0_0_50090_secondary____y6aanv
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:15
Jetty_0_0_0_0_50070_hdfs____w2cu08
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:16
Jetty_0_0_0_0_50075_datanode____hwtdwq
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:24
Jetty_0_0_0_0_8088_cluster____u0rgz3
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:25
Jetty_0_0_0_0_8042_node____19tj0x
drwxr-xr-x 2 mapred mapred 4096 2012-12-08 03:25 hsperfdata_mapred
drwxr-xr-x 5 mapred mapred 4096 2012-12-08 03:25
Jetty_0_0_0_0_19888_jobhistory____.djq1tw
drwxr-xr-x 2 hdfs hdfs 4096 2012-12-08 03:25 hsperfdata_hdfs
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 03:26 hadoop-vagrant
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 03:26 hsperfdata_vagrant
drwxr-xr-x 2 yarn yarn 4096 2012-12-08 03:26 hsperfdata_yarn
{code}
Then I ran another job at 3:29. Here are the contents of /tmp after that
{code}
vagrant@lucid64:/tmp$ ls -lrt
total 72
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 01:30 forrest-vagrant
drwxr-xr-x 4 mapred mapred 4096 2012-12-08 02:50
Jetty_0_0_0_0_50030_job____yn7qmk
drwxr-xr-x 2 root root 4096 2012-12-08 02:50 hsperfdata_root
drwxr-xr-x 3 hdfs hdfs 4096 2012-12-08 03:02 hadoop-hdfs
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:02 nm-local-dir
drwxr-xr-x 2 yarn yarn 4096 2012-12-08 03:02 logs
drwxr-xr-x 3 mapred mapred 4096 2012-12-08 03:02 hadoop-yarn
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:04
Jetty_0_0_0_0_50090_secondary____y6aanv
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:15
Jetty_0_0_0_0_50070_hdfs____w2cu08
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:16
Jetty_0_0_0_0_50075_datanode____hwtdwq
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:24
Jetty_0_0_0_0_8088_cluster____u0rgz3
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:25
Jetty_0_0_0_0_8042_node____19tj0x
drwxr-xr-x 2 mapred mapred 4096 2012-12-08 03:25 hsperfdata_mapred
drwxr-xr-x 5 mapred mapred 4096 2012-12-08 03:25
Jetty_0_0_0_0_19888_jobhistory____.djq1tw
drwxr-xr-x 2 hdfs hdfs 4096 2012-12-08 03:25 hsperfdata_hdfs
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 03:29 hsperfdata_vagrant
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 03:29 hadoop-vagrant
drwxr-xr-x 2 yarn yarn 4096 2012-12-08 03:29 hsperfdata_yarn
{code}
Clearly, the last 3 directories get updated every time a job gets run. They are:
{code}
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 03:29 hsperfdata_vagrant
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 03:29 hadoop-vagrant
drwxr-xr-x 2 yarn yarn 4096 2012-12-08 03:29 hsperfdata_yarn
{code}
The others (listed below) did get created and are related to YARN but don't get
updated on every run.
{code}
vagrant@lucid64:/tmp$ ls -lrt
total 72
drwxr-xr-x 2 vagrant vagrant 4096 2012-12-08 01:30 forrest-vagrant
drwxr-xr-x 4 mapred mapred 4096 2012-12-08 02:50
Jetty_0_0_0_0_50030_job____yn7qmk
drwxr-xr-x 2 root root 4096 2012-12-08 02:50 hsperfdata_root
drwxr-xr-x 3 hdfs hdfs 4096 2012-12-08 03:02 hadoop-hdfs
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:02 nm-local-dir
drwxr-xr-x 2 yarn yarn 4096 2012-12-08 03:02 logs
drwxr-xr-x 3 mapred mapred 4096 2012-12-08 03:02 hadoop-yarn
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:04
Jetty_0_0_0_0_50090_secondary____y6aanv
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:15
Jetty_0_0_0_0_50070_hdfs____w2cu08
drwxr-xr-x 4 hdfs hdfs 4096 2012-12-08 03:16
Jetty_0_0_0_0_50075_datanode____hwtdwq
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:24
Jetty_0_0_0_0_8088_cluster____u0rgz3
drwxr-xr-x 5 yarn yarn 4096 2012-12-08 03:25
Jetty_0_0_0_0_8042_node____19tj0x
drwxr-xr-x 2 mapred mapred 4096 2012-12-08 03:25 hsperfdata_mapred
drwxr-xr-x 5 mapred mapred 4096 2012-12-08 03:25
Jetty_0_0_0_0_19888_jobhistory____.djq1tw
drwxr-xr-x 2 hdfs hdfs 4096 2012-12-08 03:25 hsperfdata_hdfs
{code}
> hadoop 0.23 pseudo conf needs to set more properties to avoid using /tmp as
> its datadir
> ---------------------------------------------------------------------------------------
>
> Key: BIGTOP-330
> URL: https://issues.apache.org/jira/browse/BIGTOP-330
> Project: Bigtop
> Issue Type: Bug
> Components: General
> Affects Versions: 0.4.0
> Reporter: Bruno Mahé
> Assignee: Mark Grover
> Priority: Blocker
> Fix For: 0.5.0
>
>
> {quote}
> [root@localhost conf]# ls /tmp/
> hsperfdata_hdfs hsperfdata_yarn
> Jetty_0_0_0_0_50075_datanode____hwtdwq Jetty_0_0_0_0_9999_node____7nyhcr
> nm-local-dir
> hsperfdata_root Jetty_0_0_0_0_50070_hdfs____w2cu08
> Jetty_0_0_0_0_8088_cluster____u0rgz3 logs
> vmware-root
> {quote}
> Some of these directories contain rightfully some temp files, but some others
> seem to contain some data. They should be in /var/lib/hadoop/...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
