Hello Hari, can you give me some insight as to how I can put the ip addresses or rack numbers of datanodes in the choose target method.
Warm regards, Shuubham On Fri, Apr 19, 2019 at 10:05 PM Hariharan <hariharan...@gmail.com> wrote: > Favoured nodes is a per-file property that you can set at the time of > creation. Note that future rebalancing may not respect this. You can read > more about it here - > https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java#L1171 > > If there is a fixed relation b/w your files and the datanode they need to > be on, then as Hilmi suggested you are better off writing a custom policy > that extends the default one. You would need to extend the chooseTarget > method to apply your logic there. > > ~ Hari > > On Fri, Apr 19, 2019 at 12:44 PM Shuubham Ojha <shuubhamo...@gmail.com> > wrote: > >> Hi Hilmi, thanks for your response. I have already done the following: >> 1. Defined the rack topology in a script and added that to core-site.xml. >> I think this ensures that the hadoop on namenode knows the ip addresses of >> datanodes and corresponding rack numbers. >> 2. I have gone through the default block placement policy code multiple >> times and I can see that a parameter called favored nodes has been used. I >> believe this parameter is of interest to me. >> >> The trouble is that I have a set of nodes with known ip addresses but I >> cant see any specific ip addresses being used in the placement policy code. >> I specifically want to know how to tell my hadoop code to place a >> particular block of data on a particular datanode given it's ip address. I >> think this problem can be resolved by setting the given datanode as favored >> node for the time when the data block of interest is being dequeued to be >> sent to datanodes but I can't how that should be done. >> >> Warm regards, >> Shuubham Ojha >> >> On Fri, Apr 19, 2019 at 4:12 AM Hilmi Egemen Ciritoğlu < >> hilmi.egemen.cirito...@gmail.com> wrote: >> >>> Hi Shuubham, >>> >>> You can simply create your own block placement class by extending >>> BlockPlacementPolicy. >>> As a starting point, you may want to have a look at Block Placement >>> Default Policy Source Code >>> <https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java> >>> . >>> >>> Regards, >>> H. Egemen Ciritoglu >>> >>> On Thu, 18 Apr 2019 at 22:34, Shuubham Ojha <shuubhamo...@gmail.com> >>> wrote: >>> >>>> Can anyone give me some idea as to how I can write my own block >>>> placement strategy in hadoop 3. >>>> >>>> >>>> Shuubham Ojha >>>> >>>
