ok .thanks for that information . As i said i am running 2 datanodes on same machine . so my haddop home has 2 conf folders . conf and conf2 and in turn 2 hdfs-site.xml in both conf folders . I guess dfs.replication value in hdfs-site.xml of conf folder should be 3 . What should i have it in conf2 ? should it be 1 there ?
sorry if question sounds stupid . But i unfamiliar with these kind of settings ( 2 datanodes on same machine ..so having 2 conf ) If data is split across multiple datanodes , then processing capacity would be improved - ( thats what i guess ) since my file is only 240 KB , it occupies only one block . It cannot use second block and remain in another datanode . So now , does it make sense to reduce the block size so that blocks are split between 2 datanodes —if i want to take very much advantage of multiple datanodes . Best Regards, Sindhu On 25 May 2014, at 21:47, Peyman Mohajerian <mohaj...@gmail.com> wrote: > Block size are typically 64 M or 12 M, so in your case only a single block is > involved which means if you have a single replica then only a single data > node will be used. The default replication is three and since you only have > two data nodes, you will most likely have two copies of the data in two > separate data nodes. > > > On Sun, May 25, 2014 at 12:40 PM, Sindhu Hosamane <sindh...@gmail.com> wrote: > >>> Hello Friends, >>> >>> I am running multiple datanodes on a single machine . >>> >>> The output of jps command shows >>> Namenode Datanode Datanode Jobtracker tasktracker >>> Secondary Namenode >>> >>> Which assures that 2 datanodes are up and running .I execute cascalog >>> queries on this 2 datanode hadoop cluster , And i get the results of query >>> too. >>> I am not sure if it is really using both datanodes . ( bcoz anyways i get >>> results with one datanode ) >>> >>> (read somewhere about HDFS storing data in datanodes like below ) >>> 1) A HDFS scheme might automatically move data from one DataNode to >>> another if the free space on a DataNode falls below a certain threshold. >>> 2) Internally, a file is split into one or more blocks and these blocks >>> are stored in a set of DataNodes. >>> >>> My doubts are : >>> * Do i have to make any configuration changes in hadoop to tell it to share >>> datablocks between 2 datanodes or does it do automatically . >>> * Also My test data is not too big . its only 240 KB . According to point >>> 1) i don't know if such small test data can initiate automatic movement of >>> data from one datanode to another . >>> * Also what should dfs.replication value be when i am running 2 datanodes >>> ? (i guess its 2 ) >>> >>> >>> Any advice or help would be very much appreciated . >>> >>> Best Regards, >>> Sindhu >> >> > >
