On Mon, Mar 14, 2011 at 11:19 PM, suresh srinivas <srini30...@gmail.com>wrote:
> Thanks for starting off the discussion. > > > This is a huge new feature with 86 jiras already filed, which > substantially increases the complexity of the code base. > These are 86 jiras file in a feature branch. We decided to make these > changes, in smaller increments, instead of a jumbo patch. This was done in > good faith, as community did not want a jumbo patch (as seen in several > discussions), to make reviewing of the patch easy and to record the changes > for reference. > Thanks for doing it that way. > > Having an in-depth motivation and benchmarking will be needed before the > community decides on adopting it for support. > This comes as a surprise, especially from Konstantin :-). The first part of > the proposal and design both cover motivation. > That is a different motivation. The document talks about why you should use federation. I am asking about motivation of supporting the code base while not using it. At least this is how understand Allen's question and some of my colleagues'. So far our tests show no difference with federation. > This is exactly what is needed. If you could put some numbers in the jira for the reference. Also it is interesting to know whether there is a benefit in splitting the namespace. Can I e.g. do more getBlockLocations per second? This is one of the aspects of scaling, right? > As we developed this feature, some significant improvements have been made > to the system - fast snapshots (snapshot time down from 1hr 45 mins to 1 > min!), fast startup, cleanup of storage, fixing multi threading issues in > several places, decommissioning improvements etc. > This is motivation. I am glad I asked. > > The purpose of my reply was to get this discussion going, as I found > Allens question unanswered for 2 weeks. > My email was sent on March 3rd. Allen's email was sent on March 12th. > Sorry, my bad. > > The concern he has seems legitimate to me. If ops think federation will > "make running a grid much much harder" I want to know why and how much > harder. > I would like to understand the concerns here. Allen please add details. > > > The way I see it now, Federation introduces > > - lots of code complexity to the system > > - harder manageability, according to Allen > > - potential performance degradation (tbd) > I have addressed these already. > > > And the main question for those 95% of users, who don't run large > clusters > or don't want to place all their compute resources in one data center, is > what is the advantage in supporting it? > This is a valid concern. Hence the single namenode configuration that most > installations run today, will run as is. We put a lot of development and > testing effort to ensure this. > I don't know what you mean by "as is". My experience with this word in real estate tells me it can be anything.
