Hi, I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes (running DN and Task Tracker).
My issue is that the my NN is getting filled up pretty fast as the NN acts as a DN. I want my NN not to function as the DN. Does including the IP for the NN in the "dfs.hosts.exclude" and running "$ hadoop dfsadmin -refreshNodes" solve my problem? Thanks & regards Arko On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <donta...@gmail.com> wrote: > Stopping DN on one machine means excluding it. What exactly is the > requirement? How many nodes do you have right now? > > Warm Regards, > Tariq > https://mtariq.jux.com/ > cloudfront.blogspot.com > > > On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee > <arkoprovomukher...@gmail.com> wrote: >> >> Hi, >> >> I am not sure the provided link matches my situation. >> >> That is because, I am not trying to exclude any node but just trying >> to stop the NN act as a DataNode. >> >> Thanks & regards >> Arko >> >> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee >> <arkoprovomukher...@gmail.com> wrote: >> > Thanks! Let me study and find out if it works for me. >> > Thanks & regards >> > Arko >> > >> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee >> > <arkoprovomukher...@gmail.com> wrote: >> >> Hi, >> >> >> >> But the new location will also be the same NN physical server. >> >> So how does that help? >> >> >> >> I would like to clear up the space in the master by removing all the >> >> HDFS data to the slave nodes. >> >> >> >> Thanks & regards >> >> Arko >> >> >> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <donta...@gmail.com> >> >> wrote: >> >>> No, copy this directory to the new location of your choice. >> >>> >> >>> Warm Regards, >> >>> Tariq >> >>> https://mtariq.jux.com/ >> >>> cloudfront.blogspot.com >> >>> >> >>> >> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee >> >>> <arkoprovomukher...@gmail.com> wrote: >> >>>> >> >>>> Thanks! >> >>>> However, in that case, how do I handle the current data that is >> >>>> already in >> >>>> NN? >> >>>> Do I have to reformat HDFS? >> >>>> Thanks & regards >> >>>> Arko >> >>>> >> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <donta...@gmail.com> >> >>>> wrote: >> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine >> >>>> > to >> >>>> > NN. >> >>>> > Remove the hostname of the NN machine from the 'slaves' file. >> >>>> > >> >>>> > Warm Regards, >> >>>> > Tariq >> >>>> > https://mtariq.jux.com/ >> >>>> > cloudfront.blogspot.com >> >>>> > >> >>>> > >> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee >> >>>> > <arkoprovomukher...@gmail.com> wrote: >> >>>> >> >> >>>> >> Hi, >> >>>> >> >> >>>> >> Thanks for the help! >> >>>> >> >> >>>> >> However, I am still unsure about how to "turn off" the datanode >> >>>> >> feature on the NN. >> >>>> >> >> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed >> >>>> >> to >> >>>> >> a directory. >> >>>> >> >> >>>> >> Should I just comment out that property? What would happen to the >> >>>> >> current data that there is in the Master? Will it get removed >> >>>> >> automatically? >> >>>> >> >> >>>> >> Thanks & regards >> >>>> >> Arko >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq >> >>>> >> <donta...@gmail.com> >> >>>> >> wrote: >> >>>> >> > You can specify the logging level as specified by Charles. But >> >>>> >> > turning >> >>>> >> > logs >> >>>> >> > off is never a good idea. Logs are really helpful in problem >> >>>> >> > diagnosis, >> >>>> >> > which are eventual. >> >>>> >> > >> >>>> >> > Warm Regards, >> >>>> >> > Tariq >> >>>> >> > https://mtariq.jux.com/ >> >>>> >> > cloudfront.blogspot.com >> >>>> >> > >> >>>> >> > >> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee >> >>>> >> > <arkoprovomukher...@gmail.com> wrote: >> >>>> >> >> >> >>>> >> >> Hi, >> >>>> >> >> >> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode. >> >>>> >> >> >> >>>> >> >> I would like to "turn off" this feature. >> >>>> >> >> >> >>>> >> >> Request help regarding the same. >> >>>> >> >> >> >>>> >> >> Thanks & regards >> >>>> >> >> Arko >> >>>> >> >> >> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cba...@sdl.com> >> >>>> >> >> wrote: >> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the >> >>>> >> >> > NameNode >> >>>> >> >> > which >> >>>> >> >> > is >> >>>> >> >> > not recommended practice. Normally, the only files the NN >> >>>> >> >> > stores >> >>>> >> >> > are >> >>>> >> >> > the >> >>>> >> >> > image and edits files. It does not store any actual HDFS >> >>>> >> >> > data. If >> >>>> >> >> > you >> >>>> >> >> > must >> >>>> >> >> > run a DN on the NN, try turning down the logging in >> >>>> >> >> > /conf/log4j.properties: >> >>>> >> >> > >> >>>> >> >> > #hadoop.root.logger=INFO,console >> >>>> >> >> > #hadoop.root.logger=WARN,console >> >>>> >> >> > hadoop.root.logger=ERROR,console >> >>>> >> >> > >> >>>> >> >> > Depending on the logging information you require, of course. >> >>>> >> >> > >> >>>> >> >> > -Chuck >> >>>> >> >> > >> >>>> >> >> > >> >>>> >> >> > -----Original Message----- >> >>>> >> >> > From: Arko Provo Mukherjee >> >>>> >> >> > [mailto:arkoprovomukher...@gmail.com] >> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM >> >>>> >> >> > To: hdfs-user@hadoop.apache.org >> >>>> >> >> > Subject: Managing space in Master Node >> >>>> >> >> > >> >>>> >> >> > Hello Gurus, >> >>>> >> >> > >> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments. >> >>>> >> >> > >> >>>> >> >> > The issue I am continuously facing is that the Master Node >> >>>> >> >> > runs >> >>>> >> >> > out >> >>>> >> >> > of >> >>>> >> >> > disk >> >>>> >> >> > space due to logs and data files. >> >>>> >> >> > >> >>>> >> >> > I can monitor and delete log files. However, I cannot delete >> >>>> >> >> > the >> >>>> >> >> > HDFS >> >>>> >> >> > data. >> >>>> >> >> > >> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS >> >>>> >> >> > data in >> >>>> >> >> > the >> >>>> >> >> > Master >> >>>> >> >> > Node? >> >>>> >> >> > >> >>>> >> >> > Then I can use my master to handle the metadata only and >> >>>> >> >> > store the >> >>>> >> >> > logs. >> >>>> >> >> > >> >>>> >> >> > Thanks & regards >> >>>> >> >> > Arko >> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The >> >>>> >> >> > information contained in this email may be confidential >> >>>> >> >> > and/or >> >>>> >> >> > legally >> >>>> >> >> > privileged. It has been sent for the sole use of the intended >> >>>> >> >> > recipient(s). >> >>>> >> >> > If you are not the intended recipient of this mail, you are >> >>>> >> >> > hereby >> >>>> >> >> > notified >> >>>> >> >> > that any unauthorized review, use, disclosure, dissemination, >> >>>> >> >> > distribution, >> >>>> >> >> > or copying of this communication, or any of its contents, is >> >>>> >> >> > strictly >> >>>> >> >> > prohibited. If you have received this communication in error, >> >>>> >> >> > please >> >>>> >> >> > reply >> >>>> >> >> > to the sender and destroy all copies of the message. >> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, >> >>>> >> >> > Wakefield, MA >> >>>> >> >> > 01880, >> >>>> >> >> > USA >> >>>> >> >> > >> >>>> >> > >> >>>> >> > >> >>>> > >> >>>> > >> >>> >> >>> > >
