On Mar 21, 2011, at 6:08 PM, Sanjay Radia wrote:

> 
> On Mar 14, 2011, at 10:57 AM, Sanjay Radia wrote:
> 
>> 
>> On Mar 12, 2011, at 8:43 AM, Allen Wittenauer wrote:
>> 
>>> 
>>>     To me, this series of changes looks like it is going to make
>>> running a grid much much harder for very little benefit.  In
>>> particular, I don't see the difference between running multiple NN/
>>> DN combinations verses running federation, especially with client
>>> side mount tables in play.
>> 
>> 
>> 
>> Main difference between independent HDFS clusters and HDFS federation
>> is that in federation one can shares the storage of the DNs and the DNs.
>> There is a very detailed document that describes this on the Jira.
>> 
>> If you are running a single NN and you don't need the scaling then
>> running and managing hadoop is for all practical purposes unchanged.
>> 
>> 
>> sanjay
>>> 
>> 
> 
> 
> Allen, not sure if I explained the difference above.
> Base on the discussion we had at the Hug, I want to clarify a few things
> 
> In federation the NNs and the DNs are part of  a cluster. It is not as if a 
> data node is willing to store blocks for any NN anywhere in the data center.
> We still expect a data center to have multiple hadoop clusters each with a 
> set of data nodes and each cluster with 1 or more NNs.
> A DN stores block for only ONE cluster.

A few questions:
- Do we have a clear definition for a cluster?
- With the above definition, is it an error if not all DNs belong to the same 
set of NNs?
- With the working definition of a cluster, what namespace guarantees are given 
to clients?

The reason I ask is not because I oppose the idea of federations, but rather am 
curious of about the terminology and how it's 'advertised' to the user.  I 
rather like the design; it has similar ideas to a NSF project I've seen 
(http://www.reddnet.org/).

> 
> You had asked about how one debugs a corrupt file or corrupt block.
> In the old world a file's inode contains the block ids of its blocks. There 
> is also a mapping from block id to block location (ie which DN).
> In the federated hdfs, each block is identified by a longer block id, called 
> the extended block id= blockPool Id + block id.
> A block pool is owned by only ONE NN.
> Hence if you are trying to locate a block then you map the extended block id 
> to the block location (ie DN) - this is the same as before, except that the 
> identifier
> of the block is merely longer.
> 
> If you are trying to debug from the point of view of the DN:
> In federated HDFS, the blocks stored in the DN are segregated in directories 
> by the blockPool Id.
> The block pool id can be mapped to a NN since each Block pool has only  ONE 
> owner.
> Hence to map from a block to a particular NN is easy - the first part of the 
> Block's longer identifier  will tell you which NN owns that block.
> 

This sounds good.

Brian

Attachment: smime.p7s
Description: S/MIME cryptographic signature

Reply via email to