You bring up a good point. The creating and processing of block reports do take a lot of resources. It affects DFS Scalability and performance to some extent. Here are some more details: http://issues.apache.org/jira/browse/HADOOP-1079
There is one thread in the Datanode that sends block confirmations to the Namenode. The same thread computes the block report and sends it to Namenode. This sequential nature is critical in ensuring that there is no erroneous race condition in the Namenode. See http://issues.apache.org/jira/browse/HADOOP-1135 The reason block reports are there is to detect inconsistencies if blocks get deleted underneath the Datanode process. For example, an administrator can erroneously remove blk_xxx files. Also, a disk corruption can make a section of blk_xxx files unreadable. This is the reason block reports exists. My current thinking is that "block report processing" should compare the blkxxx files on disk with the data structure in the Datanode memory. If and only if there is some discrepancy between these two, then a block report be sent to the Namenode. If we do this, then we will practically get rid of 99% of block reports. Thanks, dhruba ________________________________ From: Cagdas Gerede [mailto:[EMAIL PROTECTED] Sent: Tuesday, April 29, 2008 11:32 PM To: [email protected] Cc: dhruba Borthakur Subject: Block reports: memory vs. file system, and Dividing offerService into 2 threads Currently, Block reports are computed by scanning all the files and folders in the local disk. This happens not only in startup, but also in periodic block reports. What is the intention behind doing this way instead of creating the block report from the data structure already located in the memory? One issue with the current approach is that it takes quite some time. For instance, the system I am working with has 1 million blocks in each data node and every time block report is computed, it takes about 4 minutes. You might expect this may not any effect. But there is. The problem is the computation and sending of block reports and sending the list of the new received blocks are part of the same thread. As a result, when the block report computation takes a long time, it delays the reporting of new received blocks. As you know when you are writing a multi block file and you request a new block from the namenode, the namenode checks whether the very previous block is replicated enough number of times. If at this point the data node is computing the block report, it is very likely that it didn't inform the namenode about the very previous block yet. As a result, the namenode rejects this. Then client tries to repeat this 5 more times while doubling the wait time in between up to about 6 seconds (starts with 200ms and doubles it 5 times). Then, client raises an exception to the application. Given that the block report computation takes minutes, this situation is very likely to occur. Do you have any suggestions on how to handle this situation? Are there any plans to take block reporting code segment and reporting of new received blocks to separate threads (ref: offerService method of DataNode.java file)? Thanks for your response, -- ------------ Best Regards, Cagdas Evren Gerede Home Page: http://cagdasgerede.info <http://cagdasgerede.info/>
