Yes. Warm Regards, Tariq https://mtariq.jux.com/ cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 5:24 AM, Arko Provo Mukherjee < arkoprovomukher...@gmail.com> wrote: > Sure, thanks! > One last question before I go ahead and make the changes. > My hdfs-site.xml doesn't have an entry for dfs.replication. > So can I just assume that the system is using the default of 3? > I ask this because if it is 1, then I will first have to increase it > so as to not loose data when I stop the NN being a DN. > Thanks & regards > Arko > > On Wed, Feb 13, 2013 at 5:51 PM, Mohammad Tariq <donta...@gmail.com> > wrote: > > Decommissioning is for removing DNs and not the NN. > > > > Warm Regards, > > Tariq > > https://mtariq.jux.com/ > > cloudfront.blogspot.com > > > > > > On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee > > <arkoprovomukher...@gmail.com> wrote: > >> > >> Hi, > >> > >> I was just wondering if it is okay to exclude the NN IP. > >> > >> I hope it won't create any "side effects". > >> > >> This is because I don't want to actually decommission the NN node, but > >> just make it stop working as a DataNode. > >> > >> However, if you are sure there are no problems, I will go ahead and > >> make the changes. > >> > >> Thanks & regards > >> Arko > >> > >> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <donta...@gmail.com> > >> wrote: > >> > It should. > >> > > >> > BTW what's the problem with decommissioning?It's the correct way of > >> > removing > >> > the DN, as said by Charles earlier. Create the exclude file and add > the > >> > property in the hdfs-site.xml file. > >> > > >> > Warm Regards, > >> > Tariq > >> > https://mtariq.jux.com/ > >> > cloudfront.blogspot.com > >> > > >> > > >> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee > >> > <arkoprovomukher...@gmail.com> wrote: > >> >> > >> >> Hi, > >> >> > >> >> I see that the NameNode IP is present in conf/slaves file. > >> >> > >> >> Will removing the IP of the NN from the conf/slaves file and > >> >> restarting hadoop help? > >> >> > >> >> Thanks & regards > >> >> Arko > >> >> > >> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee > >> >> <arkoprovomukher...@gmail.com> wrote: > >> >> > Hi, > >> >> > > >> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave > >> >> > Nodes > >> >> > (running DN and Task Tracker). > >> >> > > >> >> > My issue is that the my NN is getting filled up pretty fast as the > NN > >> >> > acts as a DN. > >> >> > > >> >> > I want my NN not to function as the DN. > >> >> > > >> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and > >> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem? > >> >> > > >> >> > Thanks & regards > >> >> > Arko > >> >> > > >> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq < > donta...@gmail.com> > >> >> > wrote: > >> >> >> Stopping DN on one machine means excluding it. What exactly is the > >> >> >> requirement? How many nodes do you have right now? > >> >> >> > >> >> >> Warm Regards, > >> >> >> Tariq > >> >> >> https://mtariq.jux.com/ > >> >> >> cloudfront.blogspot.com > >> >> >> > >> >> >> > >> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee > >> >> >> <arkoprovomukher...@gmail.com> wrote: > >> >> >>> > >> >> >>> Hi, > >> >> >>> > >> >> >>> I am not sure the provided link matches my situation. > >> >> >>> > >> >> >>> That is because, I am not trying to exclude any node but just > >> >> >>> trying > >> >> >>> to stop the NN act as a DataNode. > >> >> >>> > >> >> >>> Thanks & regards > >> >> >>> Arko > >> >> >>> > >> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee > >> >> >>> <arkoprovomukher...@gmail.com> wrote: > >> >> >>> > Thanks! Let me study and find out if it works for me. > >> >> >>> > Thanks & regards > >> >> >>> > Arko > >> >> >>> > > >> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee > >> >> >>> > <arkoprovomukher...@gmail.com> wrote: > >> >> >>> >> Hi, > >> >> >>> >> > >> >> >>> >> But the new location will also be the same NN physical server. > >> >> >>> >> So how does that help? > >> >> >>> >> > >> >> >>> >> I would like to clear up the space in the master by removing > all > >> >> >>> >> the > >> >> >>> >> HDFS data to the slave nodes. > >> >> >>> >> > >> >> >>> >> Thanks & regards > >> >> >>> >> Arko > >> >> >>> >> > >> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq > >> >> >>> >> <donta...@gmail.com> > >> >> >>> >> wrote: > >> >> >>> >>> No, copy this directory to the new location of your choice. > >> >> >>> >>> > >> >> >>> >>> Warm Regards, > >> >> >>> >>> Tariq > >> >> >>> >>> https://mtariq.jux.com/ > >> >> >>> >>> cloudfront.blogspot.com > >> >> >>> >>> > >> >> >>> >>> > >> >> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee > >> >> >>> >>> <arkoprovomukher...@gmail.com> wrote: > >> >> >>> >>>> > >> >> >>> >>>> Thanks! > >> >> >>> >>>> However, in that case, how do I handle the current data that > >> >> >>> >>>> is > >> >> >>> >>>> already in > >> >> >>> >>>> NN? > >> >> >>> >>>> Do I have to reformat HDFS? > >> >> >>> >>>> Thanks & regards > >> >> >>> >>>> Arko > >> >> >>> >>>> > >> >> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq > >> >> >>> >>>> <donta...@gmail.com> > >> >> >>> >>>> wrote: > >> >> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate > that > >> >> >>> >>>> > machine > >> >> >>> >>>> > to > >> >> >>> >>>> > NN. > >> >> >>> >>>> > Remove the hostname of the NN machine from the 'slaves' > >> >> >>> >>>> > file. > >> >> >>> >>>> > > >> >> >>> >>>> > Warm Regards, > >> >> >>> >>>> > Tariq > >> >> >>> >>>> > https://mtariq.jux.com/ > >> >> >>> >>>> > cloudfront.blogspot.com > >> >> >>> >>>> > > >> >> >>> >>>> > > >> >> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee > >> >> >>> >>>> > <arkoprovomukher...@gmail.com> wrote: > >> >> >>> >>>> >> > >> >> >>> >>>> >> Hi, > >> >> >>> >>>> >> > >> >> >>> >>>> >> Thanks for the help! > >> >> >>> >>>> >> > >> >> >>> >>>> >> However, I am still unsure about how to "turn off" the > >> >> >>> >>>> >> datanode > >> >> >>> >>>> >> feature on the NN. > >> >> >>> >>>> >> > >> >> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir > is > >> >> >>> >>>> >> pointed > >> >> >>> >>>> >> to > >> >> >>> >>>> >> a directory. > >> >> >>> >>>> >> > >> >> >>> >>>> >> Should I just comment out that property? What would > happen > >> >> >>> >>>> >> to > >> >> >>> >>>> >> the > >> >> >>> >>>> >> current data that there is in the Master? Will it get > >> >> >>> >>>> >> removed > >> >> >>> >>>> >> automatically? > >> >> >>> >>>> >> > >> >> >>> >>>> >> Thanks & regards > >> >> >>> >>>> >> Arko > >> >> >>> >>>> >> > >> >> >>> >>>> >> > >> >> >>> >>>> >> > >> >> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq > >> >> >>> >>>> >> <donta...@gmail.com> > >> >> >>> >>>> >> wrote: > >> >> >>> >>>> >> > You can specify the logging level as specified by > >> >> >>> >>>> >> > Charles. > >> >> >>> >>>> >> > But > >> >> >>> >>>> >> > turning > >> >> >>> >>>> >> > logs > >> >> >>> >>>> >> > off is never a good idea. Logs are really helpful in > >> >> >>> >>>> >> > problem > >> >> >>> >>>> >> > diagnosis, > >> >> >>> >>>> >> > which are eventual. > >> >> >>> >>>> >> > > >> >> >>> >>>> >> > Warm Regards, > >> >> >>> >>>> >> > Tariq > >> >> >>> >>>> >> > https://mtariq.jux.com/ > >> >> >>> >>>> >> > cloudfront.blogspot.com > >> >> >>> >>>> >> > > >> >> >>> >>>> >> > > >> >> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee > >> >> >>> >>>> >> > <arkoprovomukher...@gmail.com> wrote: > >> >> >>> >>>> >> >> > >> >> >>> >>>> >> >> Hi, > >> >> >>> >>>> >> >> > >> >> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode. > >> >> >>> >>>> >> >> > >> >> >>> >>>> >> >> I would like to "turn off" this feature. > >> >> >>> >>>> >> >> > >> >> >>> >>>> >> >> Request help regarding the same. > >> >> >>> >>>> >> >> > >> >> >>> >>>> >> >> Thanks & regards > >> >> >>> >>>> >> >> Arko > >> >> >>> >>>> >> >> > >> >> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker > >> >> >>> >>>> >> >> <cba...@sdl.com> > >> >> >>> >>>> >> >> wrote: > >> >> >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode > on > >> >> >>> >>>> >> >> > the > >> >> >>> >>>> >> >> > NameNode > >> >> >>> >>>> >> >> > which > >> >> >>> >>>> >> >> > is > >> >> >>> >>>> >> >> > not recommended practice. Normally, the only files > the > >> >> >>> >>>> >> >> > NN > >> >> >>> >>>> >> >> > stores > >> >> >>> >>>> >> >> > are > >> >> >>> >>>> >> >> > the > >> >> >>> >>>> >> >> > image and edits files. It does not store any actual > >> >> >>> >>>> >> >> > HDFS > >> >> >>> >>>> >> >> > data. If > >> >> >>> >>>> >> >> > you > >> >> >>> >>>> >> >> > must > >> >> >>> >>>> >> >> > run a DN on the NN, try turning down the logging in > >> >> >>> >>>> >> >> > /conf/log4j.properties: > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > #hadoop.root.logger=INFO,console > >> >> >>> >>>> >> >> > #hadoop.root.logger=WARN,console > >> >> >>> >>>> >> >> > hadoop.root.logger=ERROR,console > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > Depending on the logging information you require, of > >> >> >>> >>>> >> >> > course. > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > -Chuck > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > -----Original Message----- > >> >> >>> >>>> >> >> > From: Arko Provo Mukherjee > >> >> >>> >>>> >> >> > [mailto:arkoprovomukher...@gmail.com] > >> >> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM > >> >> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org > >> >> >>> >>>> >> >> > Subject: Managing space in Master Node > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > Hello Gurus, > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > I am managing a Hadoop Cluster to run some > >> >> >>> >>>> >> >> > experiments. > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > The issue I am continuously facing is that the > Master > >> >> >>> >>>> >> >> > Node > >> >> >>> >>>> >> >> > runs > >> >> >>> >>>> >> >> > out > >> >> >>> >>>> >> >> > of > >> >> >>> >>>> >> >> > disk > >> >> >>> >>>> >> >> > space due to logs and data files. > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > I can monitor and delete log files. However, I > cannot > >> >> >>> >>>> >> >> > delete > >> >> >>> >>>> >> >> > the > >> >> >>> >>>> >> >> > HDFS > >> >> >>> >>>> >> >> > data. > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any > >> >> >>> >>>> >> >> > HDFS > >> >> >>> >>>> >> >> > data in > >> >> >>> >>>> >> >> > the > >> >> >>> >>>> >> >> > Master > >> >> >>> >>>> >> >> > Node? > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > Then I can use my master to handle the metadata only > >> >> >>> >>>> >> >> > and > >> >> >>> >>>> >> >> > store the > >> >> >>> >>>> >> >> > logs. > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> >> > Thanks & regards > >> >> >>> >>>> >> >> > Arko > >> >> >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights > >> >> >>> >>>> >> >> > reserved. > >> >> >>> >>>> >> >> > The > >> >> >>> >>>> >> >> > information contained in this email may be > >> >> >>> >>>> >> >> > confidential > >> >> >>> >>>> >> >> > and/or > >> >> >>> >>>> >> >> > legally > >> >> >>> >>>> >> >> > privileged. It has been sent for the sole use of the > >> >> >>> >>>> >> >> > intended > >> >> >>> >>>> >> >> > recipient(s). > >> >> >>> >>>> >> >> > If you are not the intended recipient of this mail, > >> >> >>> >>>> >> >> > you > >> >> >>> >>>> >> >> > are > >> >> >>> >>>> >> >> > hereby > >> >> >>> >>>> >> >> > notified > >> >> >>> >>>> >> >> > that any unauthorized review, use, disclosure, > >> >> >>> >>>> >> >> > dissemination, > >> >> >>> >>>> >> >> > distribution, > >> >> >>> >>>> >> >> > or copying of this communication, or any of its > >> >> >>> >>>> >> >> > contents, > >> >> >>> >>>> >> >> > is > >> >> >>> >>>> >> >> > strictly > >> >> >>> >>>> >> >> > prohibited. If you have received this communication > in > >> >> >>> >>>> >> >> > error, > >> >> >>> >>>> >> >> > please > >> >> >>> >>>> >> >> > reply > >> >> >>> >>>> >> >> > to the sender and destroy all copies of the message. > >> >> >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, > >> >> >>> >>>> >> >> > Wakefield, MA > >> >> >>> >>>> >> >> > 01880, > >> >> >>> >>>> >> >> > USA > >> >> >>> >>>> >> >> > > >> >> >>> >>>> >> > > >> >> >>> >>>> >> > > >> >> >>> >>>> > > >> >> >>> >>>> > > >> >> >>> >>> > >> >> >>> >>> > >> >> >> > >> >> >> > >> > > >> > > > > > >
