Hello Charles, Thanks a lot for your reply and help!
Yes, the NN data (image, edit files) is kept separate from the data files. "dfs.name.dir=/hadoop/hdfs/name" whereas "dfs.data.dir=/hadoop/hdfs/data". Their contents match with the description you specified Can I safely go ahead and delete all contents from the data dir? Thanks & regards Arko On Thu, Feb 14, 2013 at 8:16 PM, Charles Baker <cba...@sdl.com> wrote: > Hey Arko. Glad to hear it worked out for you. Just make sure that your NN's > image and edit files aren't also (perhaps inadvertently) stored in that > directory structure. The NameNode image/edit dir contains the following: > > drwxr-xr-x 2 hdfs hdfs 4096 Feb 14 17:48 current > drwxr-xr-x 2 hdfs hdfs 4096 Jul 17 2012 image > -rw-r--r-- 1 hdfs hdfs 0 Sep 27 12:28 in_use.lock > drwxr-xr-x 2 hdfs hdfs 4096 Sep 17 23:25 previous.checkpoint > > whereas, the DN data dir should contain: > > drwxr-xr-x 2 hdfs hdfs 57344 Feb 14 14:04 blocksBeingWritten > drwxr-xr-x 66 hdfs hdfs 8192 Jan 22 02:44 current > drwxr-xr-x 2 hdfs hdfs 6 Aug 10 2012 detach > -rw-r--r-- 1 hdfs hdfs 0 Sep 27 11:19 in_use.lock > -rw-r--r-- 1 hdfs hdfs 157 Aug 10 2012 storage > drwxr-xr-x 2 hdfs hdfs 6 Feb 4 17:34 tmp > > > It's a good practice to locate the JobTracker and SecondaryNameNode services > on a different machine than the NN. If you have an opportunity to do so, I > recommend getting a separate machine for those two services as well. > > -Chuck > > > -----Original Message----- > From: Arko Provo Mukherjee [mailto:arkoprovomukher...@gmail.com] > Sent: Thursday, February 14, 2013 4:21 PM > To: hdfs-user@hadoop.apache.org > Subject: Re: Managing space in Master Node > > Hello Everyone - I have decommissioned my NN so that only the JobTracker, > NameNode and SecondaryNameNodes are running on it now. > I would like to thank everyone who helped me to perform this task! > > I have just one more issue: The data files created in the Master Node seem to > be still there. in /hdfs/data/current Now that this server is not being used > as a DataNode, can I safely delete the files under this folder to clear up > space? > > Thanks & regards > Arko > > On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee > <arkoprovomukher...@gmail.com> wrote: >> Thanks! I have already requested for downtime. Will do the changes soon! >> Warm regards >> Arko >> >> On Thu, Feb 14, 2013 at 3:31 AM, <ramon....@accenture.com> wrote: >>> Hi Arko. >>> >>> Only thing you need to do is not running the TaskTracker and DataNode > demons on your master machine. Ensure you do not have this on your slaves > file con hadoop's config directory when you start the system. I'm supposing > you are on Open Source release. For other distributions look at its > documentation about how to remove those demons from your master node machine. >>> >>> >>> -----Original Message----- >>> From: Arko Provo Mukherjee [mailto:arkoprovomukher...@gmail.com] >>> Sent: miércoles, 13 de febrero de 2013 20:32 >>> To: hdfs-user@hadoop.apache.org >>> Subject: Managing space in Master Node >>> >>> Hello Gurus, >>> >>> I am managing a Hadoop Cluster to run some experiments. >>> >>> The issue I am continuously facing is that the Master Node runs out >>> of disk space due to logs and data files. >>> >>> I can monitor and delete log files. However, I cannot delete the HDFS > data. >>> >>> Thus, is there a way to force Hadoop not to save any HDFS data in the >>> Master Node? >>> >>> Then I can use my master to handle the metadata only and store the logs. >>> >>> Thanks & regards >>> Arko >>> >>> >>> This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise private information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. >>> >>> Where allowed by local law, electronic communications with Accenture and > its affiliates, including e-mail and instant messaging (including content), > may be scanned by our systems for the purposes of information security and > assessment of internal compliance with Accenture policy. >>> >>> _____________________________________________________________________ >>> _________________ >>> >>> www.accenture.com >>> > SDL Enterprise Technologies, Inc. - all rights reserved. The information > contained in this email may be confidential and/or legally privileged. It has > been sent for the sole use of the intended recipient(s). If you are not the > intended recipient of this mail, you are hereby notified that any > unauthorized review, use, disclosure, dissemination, distribution, or copying > of this communication, or any of its contents, is strictly prohibited. If you > have received this communication in error, please reply to the sender and > destroy all copies of the message. > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA >
