Dhruba, It would be very valuable for the community to share your experience if you performed any independent testing of the federation branch.
Thanks, --Konstantin On Tue, Apr 26, 2011 at 9:27 PM, Dhruba Borthakur <dhr...@gmail.com> wrote: > I feel that making the datanode talk to multiple namenodes is very > valuable, > especially when there is plenty of storage available on a single datanode > machine (think 24 TB to 36 TB) and a single namenode does not have enough > memory to hold all file metadata for such a large cluster in memory. > > This is a feature that we are in dire need of, and could put it to good use > starting "yesterday"! > > thanks, > dhruba > > On Tue, Apr 26, 2011 at 5:59 PM, Konstantin Boudnik <c...@apache.org> > wrote: > > > Sanjay, > > > > I assume the outlined changes won't an earlier version of HDFS from > > upgrads to the federation version, right? > > > > Cos > > > > On Tue, Apr 26, 2011 at 17:26, Sanjay Radia <sra...@yahoo-inc.com> > wrote: > > > > > > Changes to the code base > > > - The fundamental code change is to extend the notion of block id to > now > > > include a block pool id. > > > - The NN had little change, the protocols did change to include the > > block > > > pool id. > > > - The DN code did change. Each data structure is now indexed by the > block > > > pool id -- while this is a code change, it is architecturally very > simple > > > and low risk. > > > - We also did a fair amount of cleanup of threads used to send block > > reports > > > - while it was not strictly necessary to do the cleanup we took the > extra > > > effort to pay the technical debt. As Dhruba recently noted, adding > > support > > > to send block reports to primary and secondary NN for HA will be now > much > > > easier to do. > > > > > > -- > Connect to me at http://www.facebook.com/dhruba >
