replication factor in hdfs-site.xml
Hi Users, I want behaviour for hadoop cluster for writing/reading in following replication cases: 1. replication=2 and in cluster (3 datatnodes+namenode) one datanode gets down. 2. replication=2 and in cluster (3 datatnodes+namenode) 2 datanode gets down. BR, Satyam
Don't want to read during namenode is in safemode
Hi Users, Usually read operation is allowed when namenode is in safemode, due to this my application is showing faulty behaviour. I don't want to do even read operation during safemode state of namenode, but for this i have to check status of namenode before reading. I couldn't find any api which show namenode current status. Please help how can i fix this problem. Regards, Satyam
Making All datanode down
Hi Users, In my cluster setup i am doing one test case of making only all datanodes down and keep namenode running. In this case my application gets error with remoteException: could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation Then i make all datanodes up, but i still faces above exception looks like datanode came up but not sync with namenode. I have to then restart my application for writing files to hdfs. After restart it starts processing fine but i don't want to kill my application in this case. Prompt help is highly appreciated!! Regards, Satyam
Re: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.
You have not given namenode uri in /etc/hosts file , thus it can't resolve it to ipaddress and your namenode would also be not started. Preferable practice is to start your cluster through start-dfs.sh command, it implicitly starts first namenode and then all its datanodes. Also make sure you have given ipaddress in salve file, if not then also make entry for hostnames in /etc/hosts file BR, Satyam On 08/05/2014 12:21 AM, S.L wrote: The contents are 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 On Sun, Aug 3, 2014 at 11:21 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com mailto:riteshoneinamill...@gmail.com wrote: check the contents of '/etc/hosts' file On Mon, Aug 4, 2014 at 3:27 AM, S.L simpleliving...@gmail.com mailto:simpleliving...@gmail.com wrote: Hi All, I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a master and three slave nodes , the slave nodes are listed in the $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from the slaves to the Master Name node on port 9000, however when I start the datanode on any of the slaves I get the following exception . 2014-08-03 08:04:27,952 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1086620743-170.75.152.162-1407064313305 (Datanode Uuid null) service to server1.dealyaft.com/170.75.152.162:9000 http://server1.dealyaft.com/170.75.152.162:9000 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode because hostname cannot be resolved . The following are the contents of my core-site.xml. configuration property namefs.default.name http://fs.default.name/name valuehdfs://server1.mydomain.com:9000 http://server1.mydomain.com:9000/value /property /configuration Also in my hdfs-site.xml I am not setting any value for dfs.hosts or dfs.hosts.exclude properties. Thanks.
When all datanodes get down and namenode is still up
Hello Users, When i made all datnodes down and namenode is still up writing failures occures which is fine. But after few mins i have made all datanode up one by one. But still i observe writing failures and it looks like datanodes are not available. This state resolves only when i restart my application and hadoop cluster restart with format.And this is what i don't want to do. If anyone face this type of issue then please suggest what to do. Regards, Satyam
Which replication value will be used
Hello,
I have given dfs.replication=2 in hdfs-site.xml as:
property
namedfs.replication/name
value2/value
/property
Also, In my application i have used api for wrting in hdfs:
public FSDataOutputStream create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress
) throws IOException {
return this.create(f, FsPermission.getFileDefault().applyUMask(
FsPermission.getUMask(getConf())), overwrite, bufferSize,
replication, blockSize, progress);
}
here i also given replication as fourth parameter.
So my question is which replication value will be used that we have
given in api or in hdfs-site.xml
Warm Regards,
Satyam
One datanode is down then write/read starts failing
Hello, I have hadoop cluster setup of one namenode and two datanodes. And i continuously write/read/delete through hdfs on namenode through hadoop client. Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests. I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases. Can anybody face this issue or i am doing something wrong in my setup. Thanx in advance. Warm Regards, Satyam
Re: One datanode is down then write/read starts failing
Yes, there is lot of space available at that instant. I am not sure but i have read somewhere that we must have datanodes live = replication factor given at namenode at any point of time. If live datanodes get less than replication factor then this write/read failure occurs. In my case i have given replication factor 2 and initially i have 2 live datanodes. Then i killed one of the datanode and number of live DN becomes 1 and still replication is 2. Please comment. On 07/28/2014 09:31 PM, Wellington Chevreuil wrote: Can you make sure you still have enough HDFS space once you kill this DN? If not, HDFS will automatically enter safemode if it detects there's no hdfs space available. The error message on the logs should have some hints on this. Cheers. On 28 Jul 2014, at 16:56, Satyam Singh satyam.si...@ericsson.com wrote: Hello, I have hadoop cluster setup of one namenode and two datanodes. And i continuously write/read/delete through hdfs on namenode through hadoop client. Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests. I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases. Can anybody face this issue or i am doing something wrong in my setup. Thanx in advance. Warm Regards, Satyam
Re: One datanode is down then write/read starts failing
@vikas i have initially set 2 but after that i have make one DN down. So you are saying from initial i have to make replication factor as 1 even i have DN 2 active initially. If so then what is the reason? On 07/28/2014 10:02 PM, Vikas Srivastava wrote: What replication have you set for cluster. Its should be 1 in your case. On Jul 28, 2014 9:26 PM, Satyam Singh satyam.si...@ericsson.com wrote: Hello, I have hadoop cluster setup of one namenode and two datanodes. And i continuously write/read/delete through hdfs on namenode through hadoop client. Then i kill one of the datanode, still one is working but writing on datanode is getting failed for all write requests. I want to overcome this scenario because at live traffic scenario any of datanode might get down then how do we handle those cases. Can anybody face this issue or i am doing something wrong in my setup. Thanx in advance. Warm Regards, Satyam
Reading files from hdfs directory
Hello, I want to read files from hdfs remotely through camel-hdfs client. I have made changes in camel-hdfs component for supporting hadoop2.2.0 . I checked file that I want to read, exists on hdfs: [hduser@bl460cx2425 ~]$ hadoop fs -ls /user/hduser/collector/test.txt 14/03/13 09:13:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items -rw-r--r-- 3 root supergroup 1886 2014-03-13 09:13 /user/hduser/collector/test.txt But, I get following exception when I am trying to read it through my client from remote machine : 2014-03-13 14:08:25 STDIO [ERROR] Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/hduser/collector/test.txt at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:51) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1499) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1448) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1428) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1402) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:468) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:269) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59566) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) 2014-03-13 14:08:25 STDIO [ERROR] at org.apache.hadoop.ipc.Client.call(Client.java:1347) 2014-03-13 14:08:25 STDIO [ERROR] at org.apache.hadoop.ipc.Client.call(Client.java:1300) 2014-03-13 14:08:25 STDIO [ERROR] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) 2014-03-13 14:08:25 STDIO [ERROR] at com.sun.proxy.$Proxy17.getBlockLocations(Unknown Source) 2014-03-13 14:08:25 STDIO [ERROR] at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) 2014-03-13 14:08:25 STDIO [ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2014-03-13 14:08:25 STDIO [ERROR] at java.lang.reflect.Method.invoke(Method.java:606) 2014-03-13 14:08:25 STDIO [ERROR] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) 2014-03-13 14:08:25 STDIO [ERROR] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) 2014-03-13 14:08:25 STDIO [ERROR] at com.sun.proxy.$Proxy17.getBlockLocations(Unknown Source) 2014-03-13 14:08:25 STDIO [ERROR] at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:188) 2014-03-13 14:08:25 STDIO [ERROR] at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1064) 2014-03-13 14:08:25 STDIO [ERROR] ... 24 more In my client I have ftp files from remote ftp server and put it in hdfs system in path: /user/hduser/collector. Then we send this file name to our hdfs file reading client and gives above exception. Prompt help is really appreciated :) BR, Satyam
