With the availability of HADOOP-5332 I remove my objection.
> -----Original Message----- > From: Dhruba Borthakur [mailto:[email protected]] > Sent: Wednesday, February 25, 2009 9:32 PM > To: [email protected] > Subject: Re: Hadoop 0.20.0 > > I posted a patch for HADOOP-5332. I am suggesting that this patch be > applied > into the 0.19, 0.20 and trunk. This patch switches off "append" by > default, > but it can be switched on by setting the config parameter > dfs.support.append. This does not mean that "append" is bug free in the > code, it just allows developers to continue testing with append > functionality till the bugs are fixed. > > thanks, > dhruba > > On Wed, Feb 25, 2009 at 9:05 PM, Hemanth Yamijala <yhema...@yahoo- > inc.com>wrote: > > > +1 for HADOOP-5332. I am in the same position as Brian, as an outside > > observer. This will help us to move on Hadoop 0.20 which has a lot of > other > > features as well that users are asking for. > > > > Thanks > > hemanth > > > > > > > >> On Feb 25, 2009, at 10:20 PM, Nigel Daley wrote: > >> > >> > >>> On Feb 25, 2009, at 7:52 PM, Dhruba Borthakur wrote: > >>> > >>> "Whipping out a patch" says nothing about its reliability. > >>>> > >>>> i would like some focus from the developer's community to properly > fix > >>>> this > >>>> issue. I am willing to spend as much as time it takes ot get it > fixed > >>>> the > >>>> right way, I but I would like even more constructive engagement > from > >>>> more > >>>> people to get this one right. May I request you to see if you can > >>>> volunteer > >>>> to spend some time testing some of this code at scale ?(I have > access to > >>>> 10 > >>>> machines only for testing). > >>>> > >>> > >>> > >> Dhruba, can you define "testing some of this code at scale"? Do you > simply > >> need access or folks who can run challenging jobs? Scaring up access > to the > >> cluster can be easy, but admin / user time isn't really available. > >> > >> Sorry, I can't commit any time/resources to this right now. Perhaps > some > >>> hbase folks can. In the meantime, can we make append configurable in > 0.19.2 > >>> and 0.20.0? I filed > >>> https://issues.apache.org/jira/browse/HADOOP-5332 > >>> > >> > >> As an outside, irrelevant observer, I think this is a really good > >> compromise. Helps out HBase but also would help prevent rushing. > >> > >> Brian > >> > >> > >>> > >>> Cheers, > >>> Nige > >>> > >>> > >>>> > >>>> thanks > >>>> dhruba > >>>> > >>>> On Wed, Feb 25, 2009 at 7:34 PM, Nigel Daley <[email protected]> > >>>> wrote: > >>>> > >>>> > >>>>> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote: > >>>>> > >>>>> Hi Jim, > >>>>> > >>>>>> > >>>>>> I can understand your problem. I can probably whip out a fix for > >>>>>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be > nice > >>>>>> if > >>>>>> somebody else (Hairong, Sanjay, Konstantin?) can volunteer to > discuss > >>>>>> and > >>>>>> review the patches/fixes. > >>>>>> > >>>>>> > >>>>> "Whipping out a patch" doesn't give me any confidence that this > feature > >>>>> will be fixed properly. We're building a file system. Data > reliability > >>>>> and > >>>>> accuracy are absolutely key. We know that this feature has been > very > >>>>> lightly tested. > >>>>> > >>>>> Nigel: wht is the proposed deadline for 0.20? > >>>>> > >>>>>> > >>>>>> > >>>>> March 6. > >>>>> > >>>>> Nige > >>>>> > >>>>> > >>>>> thanks, > >>>>> > >>>>>> dhruba > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) < > >>>>>> [email protected]> wrote: > >>>>>> > >>>>>> --1 > >>>>>> > >>>>>>> > >>>>>>> HBase really needs 4379. My testing to date indicates that it > does > >>>>>>> work > >>>>>>> (although I have a bit more testing to do). > >>>>>>> > >>>>>>> I was ok with not putting it into 0.19.1 provided it was in > 0.19.2 > >>>>>>> and > >>>>>>> 0.20.0. > >>>>>>> > >>>>>>> It's a big problem for us now and is hurting our ability to keep > our > >>>>>>> community alive. (They will go to Cassandra or something else to > >>>>>>> ensure > >>>>>>> reliability). > >>>>>>> > >>>>>>> > >>>>>>> -----Original Message----- > >>>>>>> > >>>>>>>> From: Nigel Daley [mailto:[email protected]] > >>>>>>>> Sent: Tuesday, February 24, 2009 4:02 PM > >>>>>>>> To: [email protected] > >>>>>>>> Subject: Hadoop 0.20.0 > >>>>>>>> > >>>>>>>> Folks, > >>>>>>>> > >>>>>>>> Hadoop 0.19.1 is now available with the file append feature > >>>>>>>> disabled. > >>>>>>>> It's time to talk about a Hadoop 0.20.0 release. > >>>>>>>> > >>>>>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago. The > last > >>>>>>>> few blockers are now almost fixed (should be next week) except > for > >>>>>>>> HADOOP-4379. HADOOP-4379 is work that is needed to properly > >>>>>>>> implement > >>>>>>>> file append. > >>>>>>>> > >>>>>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and > apply > >>>>>>>> the > >>>>>>>> same disabling of file append in Hadoop 0.20.0 that we put in > place > >>>>>>>> to > >>>>>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225). > >>>>>>>> > >>>>>>>> I will call a vote for 0.20.0 when blockers are fixed. > >>>>>>>> > >>>>>>>> Cheers, > >>>>>>>> Nigel > >>>>>>>> > >>>>>>>> > >>>>>>>> Folks, > >>>>>>>> > >>>>>>>>> > >>>>>>>>> Some Hadoop deployments have upgraded to 0.19.0. Clearly, the > 0.19 > >>>>>>>>> branch has issues and a 0.19.1 release is needed. > >>>>>>>>> > >>>>>>>>> Quality issues in the changes made for the file append feature > have > >>>>>>>>> prevented some from deploying Hadoop 0.19. One of these > changes > >>>>>>>>> (sync) has now been "fixed" by reducing its semantics in > Hadoop > >>>>>>>>> 0.18.3 (HADOOP-4997). This was necessary to stabilize the 0.18 > >>>>>>>>> branch. > >>>>>>>>> > >>>>>>>>> I would like to propose that we apply this same "fix" to sync > in > >>>>>>>>> 0.19.1 and 0.20.0. Since append requires the full semantics of > >>>>>>>>> sync, I propose we also disable append (perhaps throw > >>>>>>>>> UnsupportedOperationException from API?). Yes, this would > >>>>>>>>> unfortunately be an incompatible change between 0.19.0 and > 0.19.1. > >>>>>>>>> We can then take the time needed to fix append properly in > 0.21.0. > >>>>>>>>> > >>>>>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are > fixed. > >>>>>>>>> > >>>>>>>>> Nigel > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >> > >
