Hi, I see that the NameNode IP is present in conf/slaves file.
Will removing the IP of the NN from the conf/slaves file and restarting hadoop help? Thanks & regards Arko On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee <arkoprovomukher...@gmail.com> wrote: > Hi, > > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes > (running DN and Task Tracker). > > My issue is that the my NN is getting filled up pretty fast as the NN > acts as a DN. > > I want my NN not to function as the DN. > > Does including the IP for the NN in the "dfs.hosts.exclude" and > running "$ hadoop dfsadmin -refreshNodes" solve my problem? > > Thanks & regards > Arko > > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <donta...@gmail.com> wrote: >> Stopping DN on one machine means excluding it. What exactly is the >> requirement? How many nodes do you have right now? >> >> Warm Regards, >> Tariq >> https://mtariq.jux.com/ >> cloudfront.blogspot.com >> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee >> <arkoprovomukher...@gmail.com> wrote: >>> >>> Hi, >>> >>> I am not sure the provided link matches my situation. >>> >>> That is because, I am not trying to exclude any node but just trying >>> to stop the NN act as a DataNode. >>> >>> Thanks & regards >>> Arko >>> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee >>> <arkoprovomukher...@gmail.com> wrote: >>> > Thanks! Let me study and find out if it works for me. >>> > Thanks & regards >>> > Arko >>> > >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee >>> > <arkoprovomukher...@gmail.com> wrote: >>> >> Hi, >>> >> >>> >> But the new location will also be the same NN physical server. >>> >> So how does that help? >>> >> >>> >> I would like to clear up the space in the master by removing all the >>> >> HDFS data to the slave nodes. >>> >> >>> >> Thanks & regards >>> >> Arko >>> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <donta...@gmail.com> >>> >> wrote: >>> >>> No, copy this directory to the new location of your choice. >>> >>> >>> >>> Warm Regards, >>> >>> Tariq >>> >>> https://mtariq.jux.com/ >>> >>> cloudfront.blogspot.com >>> >>> >>> >>> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee >>> >>> <arkoprovomukher...@gmail.com> wrote: >>> >>>> >>> >>>> Thanks! >>> >>>> However, in that case, how do I handle the current data that is >>> >>>> already in >>> >>>> NN? >>> >>>> Do I have to reformat HDFS? >>> >>>> Thanks & regards >>> >>>> Arko >>> >>>> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <donta...@gmail.com> >>> >>>> wrote: >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine >>> >>>> > to >>> >>>> > NN. >>> >>>> > Remove the hostname of the NN machine from the 'slaves' file. >>> >>>> > >>> >>>> > Warm Regards, >>> >>>> > Tariq >>> >>>> > https://mtariq.jux.com/ >>> >>>> > cloudfront.blogspot.com >>> >>>> > >>> >>>> > >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee >>> >>>> > <arkoprovomukher...@gmail.com> wrote: >>> >>>> >> >>> >>>> >> Hi, >>> >>>> >> >>> >>>> >> Thanks for the help! >>> >>>> >> >>> >>>> >> However, I am still unsure about how to "turn off" the datanode >>> >>>> >> feature on the NN. >>> >>>> >> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed >>> >>>> >> to >>> >>>> >> a directory. >>> >>>> >> >>> >>>> >> Should I just comment out that property? What would happen to the >>> >>>> >> current data that there is in the Master? Will it get removed >>> >>>> >> automatically? >>> >>>> >> >>> >>>> >> Thanks & regards >>> >>>> >> Arko >>> >>>> >> >>> >>>> >> >>> >>>> >> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq >>> >>>> >> <donta...@gmail.com> >>> >>>> >> wrote: >>> >>>> >> > You can specify the logging level as specified by Charles. But >>> >>>> >> > turning >>> >>>> >> > logs >>> >>>> >> > off is never a good idea. Logs are really helpful in problem >>> >>>> >> > diagnosis, >>> >>>> >> > which are eventual. >>> >>>> >> > >>> >>>> >> > Warm Regards, >>> >>>> >> > Tariq >>> >>>> >> > https://mtariq.jux.com/ >>> >>>> >> > cloudfront.blogspot.com >>> >>>> >> > >>> >>>> >> > >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee >>> >>>> >> > <arkoprovomukher...@gmail.com> wrote: >>> >>>> >> >> >>> >>>> >> >> Hi, >>> >>>> >> >> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode. >>> >>>> >> >> >>> >>>> >> >> I would like to "turn off" this feature. >>> >>>> >> >> >>> >>>> >> >> Request help regarding the same. >>> >>>> >> >> >>> >>>> >> >> Thanks & regards >>> >>>> >> >> Arko >>> >>>> >> >> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cba...@sdl.com> >>> >>>> >> >> wrote: >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the >>> >>>> >> >> > NameNode >>> >>>> >> >> > which >>> >>>> >> >> > is >>> >>>> >> >> > not recommended practice. Normally, the only files the NN >>> >>>> >> >> > stores >>> >>>> >> >> > are >>> >>>> >> >> > the >>> >>>> >> >> > image and edits files. It does not store any actual HDFS >>> >>>> >> >> > data. If >>> >>>> >> >> > you >>> >>>> >> >> > must >>> >>>> >> >> > run a DN on the NN, try turning down the logging in >>> >>>> >> >> > /conf/log4j.properties: >>> >>>> >> >> > >>> >>>> >> >> > #hadoop.root.logger=INFO,console >>> >>>> >> >> > #hadoop.root.logger=WARN,console >>> >>>> >> >> > hadoop.root.logger=ERROR,console >>> >>>> >> >> > >>> >>>> >> >> > Depending on the logging information you require, of course. >>> >>>> >> >> > >>> >>>> >> >> > -Chuck >>> >>>> >> >> > >>> >>>> >> >> > >>> >>>> >> >> > -----Original Message----- >>> >>>> >> >> > From: Arko Provo Mukherjee >>> >>>> >> >> > [mailto:arkoprovomukher...@gmail.com] >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org >>> >>>> >> >> > Subject: Managing space in Master Node >>> >>>> >> >> > >>> >>>> >> >> > Hello Gurus, >>> >>>> >> >> > >>> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments. >>> >>>> >> >> > >>> >>>> >> >> > The issue I am continuously facing is that the Master Node >>> >>>> >> >> > runs >>> >>>> >> >> > out >>> >>>> >> >> > of >>> >>>> >> >> > disk >>> >>>> >> >> > space due to logs and data files. >>> >>>> >> >> > >>> >>>> >> >> > I can monitor and delete log files. However, I cannot delete >>> >>>> >> >> > the >>> >>>> >> >> > HDFS >>> >>>> >> >> > data. >>> >>>> >> >> > >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS >>> >>>> >> >> > data in >>> >>>> >> >> > the >>> >>>> >> >> > Master >>> >>>> >> >> > Node? >>> >>>> >> >> > >>> >>>> >> >> > Then I can use my master to handle the metadata only and >>> >>>> >> >> > store the >>> >>>> >> >> > logs. >>> >>>> >> >> > >>> >>>> >> >> > Thanks & regards >>> >>>> >> >> > Arko >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The >>> >>>> >> >> > information contained in this email may be confidential >>> >>>> >> >> > and/or >>> >>>> >> >> > legally >>> >>>> >> >> > privileged. It has been sent for the sole use of the intended >>> >>>> >> >> > recipient(s). >>> >>>> >> >> > If you are not the intended recipient of this mail, you are >>> >>>> >> >> > hereby >>> >>>> >> >> > notified >>> >>>> >> >> > that any unauthorized review, use, disclosure, dissemination, >>> >>>> >> >> > distribution, >>> >>>> >> >> > or copying of this communication, or any of its contents, is >>> >>>> >> >> > strictly >>> >>>> >> >> > prohibited. If you have received this communication in error, >>> >>>> >> >> > please >>> >>>> >> >> > reply >>> >>>> >> >> > to the sender and destroy all copies of the message. >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, >>> >>>> >> >> > Wakefield, MA >>> >>>> >> >> > 01880, >>> >>>> >> >> > USA >>> >>>> >> >> > >>> >>>> >> > >>> >>>> >> > >>> >>>> > >>> >>>> > >>> >>> >>> >>> >> >>
