Re: Decommissioning a node takes forever
Hi All, Thanks for the excellent suggestion , once we find out the blocks which have replication factor more than actual datanodes and we run setrep = datanodes . Should the decomissiong proceed smoothly . From your past experiences how much time does a decommissioning take to complete . During decommissioning can i execute my MR jobs and will it affect performance of cluster ? On Thu, Mar 27, 2014 at 7:58 AM, Mingjiang Shi m...@gopivotal.com wrote: Hi Brahma, It might be some files have replication factor more than the actual number of datanodes, so namenode will not be able to decommission the machine because it cannot get the replica count settled. Run the following command to check the replication factor of the files on the hdfs, see if any file has replication factor more than the actual number of datanodes. sudo -u hdfs hdfs fsck / -files -blocks | grep -B 1 repl= On Wed, Mar 26, 2014 at 3:49 PM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: can you please elaborate more..? Like how many nodes are there in cluster and what's the replication factor for files..? Normally decommission will be success once all the replica's from the excluded node is replicated another node in cluster(another node should be availble,,).. Thanks Regards Brahma Reddy Battula -- *From:* Bharath Kumar [bharath...@gmail.com] *Sent:* Wednesday, March 26, 2014 12:47 PM *To:* user@hadoop.apache.org *Subject:* Decommissioning a node takes forever Hi All, I am a novice hadoop user . I tried removing a node from my cluster of 2 nodes by adding the ip in excludes file and running dfsadmin -refreshNodes command . But decommissioning takes a very long time I left it over the weekend and still it was not complete. Your inputs will help -- Warm Regards, *Bharath* -- Cheers -MJ -- Warm Regards, *Bharath Kumar *
Decommissioning a node takes forever
Hi All, I am a novice hadoop user . I tried removing a node from my cluster of 2 nodes by adding the ip in excludes file and running dfsadmin -refreshNodes command . But decommissioning takes a very long time I left it over the weekend and still it was not complete. Your inputs will help -- Warm Regards, *Bharath*
Re: Decommissioning a node takes forever
Hi, which version HDFS you used? On Wed, Mar 26, 2014 at 3:17 PM, Bharath Kumar bharath...@gmail.com wrote: Hi All, I am a novice hadoop user . I tried removing a node from my cluster of 2 nodes by adding the ip in excludes file and running dfsadmin -refreshNodes command . But decommissioning takes a very long time I left it over the weekend and still it was not complete. Your inputs will help -- Warm Regards, *Bharath*
Re: Decommissioning a node takes forever
Hi Brahma, It might be some files have replication factor more than the actual number of datanodes, so namenode will not be able to decommission the machine because it cannot get the replica count settled. Run the following command to check the replication factor of the files on the hdfs, see if any file has replication factor more than the actual number of datanodes. sudo -u hdfs hdfs fsck / -files -blocks | grep -B 1 repl= On Wed, Mar 26, 2014 at 3:49 PM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: can you please elaborate more..? Like how many nodes are there in cluster and what's the replication factor for files..? Normally decommission will be success once all the replica's from the excluded node is replicated another node in cluster(another node should be availble,,).. Thanks Regards Brahma Reddy Battula -- *From:* Bharath Kumar [bharath...@gmail.com] *Sent:* Wednesday, March 26, 2014 12:47 PM *To:* user@hadoop.apache.org *Subject:* Decommissioning a node takes forever Hi All, I am a novice hadoop user . I tried removing a node from my cluster of 2 nodes by adding the ip in excludes file and running dfsadmin -refreshNodes command . But decommissioning takes a very long time I left it over the weekend and still it was not complete. Your inputs will help -- Warm Regards, *Bharath* -- Cheers -MJ
decommissioning a node
Our cluster has a node that reboot randomly. So I've gone to Ambari, decommissioned its HDFS service, stopped all services, and deleted the node from the cluster. I expected and fsck to immediately show under-replicated blocks, but everything comes up fine. How do I tell the cluster that this node is really gone, and it should start replicating the missing blocks? Thanks John
RE: decommissioning a node
OK, restarting all services now fsck shows under-replication. Was it the NameNode restart? John From: John Lilley [mailto:john.lil...@redpoint.net] Sent: Tuesday, March 04, 2014 5:47 AM To: user@hadoop.apache.org Subject: decommissioning a node Our cluster has a node that reboot randomly. So I've gone to Ambari, decommissioned its HDFS service, stopped all services, and deleted the node from the cluster. I expected and fsck to immediately show under-replicated blocks, but everything comes up fine. How do I tell the cluster that this node is really gone, and it should start replicating the missing blocks? Thanks John
Problem in decommissioning data node
The version of hadoop is 0.20,while I try to decommission a datanode from the cluster: (1) Add the network addresses(IP adrress) of the nodes to be decommissioned to the exclude file. (2)Add property mapred.hosts.exclude to mapred-sites.xml and dfs.hosts.exclude to dfs-sites.xml like follow: *property* * namemapred.hosts.exclude/name* * value/data/stat/hadoop/conf/excludes/value* * finaltrue/final* */property* * * (3)And then I run hadoop dfsadmin -refreshNodes, in report i see:Decommission Status : Decommission in progress *Name: 10.134.135.30:50010* *Decommission Status : Decommission in progress* *Configured Capacity: 1546959884288 (1.41 TB)* *DFS Used: 2647441408 (2.47 GB)* *Non DFS Used: 651818385408 (607.05 GB)* *DFS Remaining: 892494057472(831.2 GB)* *DFS Used%: 0.17%* *DFS Remaining%: 57.69%* *Last contact: Thu Sep 16 11:32:44 CST 2010* * * but more than one hour has passed,the status of the decommissioning data node hasn't been changed to decommissioned, anyone can tell me how to deal with this? thanks. Shuai Chen Intelligent Transportation Information Systems Laboratory Department of Electronic Engineering, Tsinghua University Beijing 100084, P. R. China Mobile:15210950974
Re: Question about decommissioning a node
s the source of transfers always the node being decommissioned? No. the source of the transfer(s) could be any other node in the cluster. dhruba On Wed, Sep 16, 2009 at 10:06 AM, Harold Lim rold...@yahoo.com wrote: Thanks. Another Question: During decommissioning process, does HDFS strictly copy blocks/files from the decommissioned node to the other live nodes? Or do blocks get copied from other live nodes too? i.e., is the source of transfers always the node being decommissioned? -Harold --- On Tue, 9/15/09, Dhruba Borthakur dhr...@gmail.com wrote: From: Dhruba Borthakur dhr...@gmail.com Subject: Re: Question about decommissioning a node To: hdfs-user@hadoop.apache.org Date: Tuesday, September 15, 2009, 5:03 PM This might help: http://wiki.apache.org/hadoop/FAQ#17 https://issues.apache.org/jira/browse/HADOOP-681 Bandwidth can be throttled on a datanode via dfs.balance.bandwidthPerSec thanks, dhruba On Tue, Sep 15, 2009 at 1:55 PM, Harold Lim rold...@yahoo.com wrote: Hi All, Is there a document or can anyone explain to me what actually happens during the decommissioning process? What does the name node do? Also, how much bandwidth does the decommissioning process take? Is there a cap to the maximum bandwidth consumed? Thanks, Harold
Re: Question about decommissioning a node
This might help: http://wiki.apache.org/hadoop/FAQ#17 https://issues.apache.org/jira/browse/HADOOP-681 Bandwidth can be throttled on a datanode via dfs.balance.bandwidthPerSec thanks, dhruba On Tue, Sep 15, 2009 at 1:55 PM, Harold Lim rold...@yahoo.com wrote: Hi All, Is there a document or can anyone explain to me what actually happens during the decommissioning process? What does the name node do? Also, how much bandwidth does the decommissioning process take? Is there a cap to the maximum bandwidth consumed? Thanks, Harold
