Great work. So what are the next steps? It seems from the discussion that all outstanding requests have been addressed. Does the branch ready for merge or, you have to rebase the branch as well?
Time to start a merge vote? Enis On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <vladrodio...@gmail.com> wrote: > >> mapreduce dependency has been moved to client side - no mapreduce job > > 1. We have no code in the client module anymore, due to dependency on > internal server API (HFile and WAL access). > 2. Backup/ restore are client - driven operations, but all the code resides > in the server module > 3. No MR in Master, no procedure - driven execution. > 4. Old good MR from command-line. > 5. Security was simplified and now only super-user is allowed to run > backup/restores. > 6. HBase Backup API was gone due to 1. Now only command-line access to > backup tools. > > These consequences of refactoring has been discussed in HBASE-16727. > > -Vlad > > > > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Reviving this thread. > > > > The following has taken place: > > > > mapreduce dependency has been moved to client side - no mapreduce job > > launched from master or region server. > > document patch (HBASE-16574) has been integrated. > > Updated mega patch has been attached to HBASE-14123: this covers the > > refactor in #1 above and the protobuf 3 merge. > > > > If community has more feedback on the merge proposal, I would love to > hear > > it. > > > > Thanks > > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com> > wrote: > > > > > I'd like to see the docs proposed on HBASE-16574 integrated into our > > > project's documentation prior to merge. > > > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > This feature can be marked experimental due to some limitations such > as > > > > security. > > > > > > > > Your previous round of comments have been addressed. > > > > Command line tool has gone through: > > > > > > > > HBASE-16620 Fix backup command-line tool usability issues > > > > HBASE-16655 hbase backup describe with incorrect backup id results in > > NPE > > > > > > > > The updated doc has been attached to HBASE-16574. > > > > > > > > Cheers > > > > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote: > > > > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > >> > > > >> > Are there more (review) comments ? > > > >> > > > > >> > > > > >> Are outstanding comments addressed? > > > >> > > > >> I don't see answer to my 'is this experimental/will it be marked > > > >> experimental' question. > > > >> > > > >> I ran into some issues trying to use the feature and suggested that > a > > > >> feature likes this needs polish else it'll just rot, unused. Has > > polish > > > >> been applied? All ready for another 'user' test? Suggest that you > > update > > > >> here going forward for the benefit of those trying to follow along > and > > > who > > > >> are not watching JIRA change fly-by. > > > >> > > > >> It looks like doc got a revision -- I have to check -- to take on > > > >> suggestion made above but again, suggest, that this thread gets > > updated. > > > >> > > > >> Thanks, > > > >> St.Ack > > > >> > > > >> > > > >> > > > >> > Thanks > > > >> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das < > d...@hortonworks.com > > > > > > >> > wrote: > > > >> > > > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others > > for > > > the > > > >> > > thorough reviews and testing. Thanks Ted and Vlad for taking > care > > of > > > >> the > > > >> > > feedback. Are we all good to do the merge now? Rather do sooner > > than > > > >> > later. > > > >> > > ________________________________________ > > > >> > > From: saint....@gmail.com <saint....@gmail.com> on behalf of > > Stack > > > < > > > >> > > st...@duboce.net> > > > >> > > Sent: Monday, September 12, 2016 1:18 PM > > > >> > > To: HBase Dev List > > > >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch > > HBASE-7912 > > > >> > > > > > >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> > > > wrote: > > > >> > > > > > >> > > > Mega patch (rev 18) is on HBASE-14123. > > > >> > > > > > > >> > > > Please comment on HBASE-14123 on how you want to review. > > > >> > > > > > > >> > > > > > >> > > > > > >> > > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest > > > updating > > > >> > it. > > > >> > > RB is pretty good for review. Patch is only 1.5M so should be > > fine. > > > >> > > > > > >> > > St.Ack > > > >> > > > > > >> > > > > > >> > > > > > > >> > > > Thanks > > > >> > > > > > > >> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> > > wrote: > > > >> > > > > > > >> > > > > On review of the 'patch', do I just compare the branch to > > > master or > > > >> > is > > > >> > > > > there a megapatch posted somewhere (I think I saw one but it > > > seemed > > > >> > > stale > > > >> > > > > and then I 'lost' the tab). Sorry for dumb question. > > > >> > > > > St.Ack > > > >> > > > > > > > >> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> > > > wrote: > > > >> > > > > > > > >> > > > > > Late to the game. A few comments after rereading this > thread > > > as a > > > >> > > > 'user'. > > > >> > > > > > > > > >> > > > > > + Before merge, a user-facing feature like this should > work > > > (If > > > >> > this > > > >> > > is > > > >> > > > > "higher-bar > > > >> > > > > > for new features", bring it on -- smile). > > > >> > > > > > + As a user, I tried the branch with tools after reviewing > > the > > > >> > > > > just-posted > > > >> > > > > > doc. I had an 'interesting' experience (left comments up > on > > > >> > issue). I > > > >> > > > > think > > > >> > > > > > the tooling/doc. important to get right. If it breaks > easily > > > or > > > >> is > > > >> > > > > > inconsistent (or lacks 'polish'), operators will judge the > > > whole > > > >> > > > > > backup/restore tooling chain as not trustworthy and > abandon > > > it. > > > >> > Lets > > > >> > > > not > > > >> > > > > > have this happen to this feature. > > > >> > > > > > + Matteo's suggestion (with a helpful starter list) that > > there > > > >> > needs > > > >> > > to > > > >> > > > > be > > > >> > > > > > explicit qualification on what is actually being delivered > > -- > > > >> > > > including a > > > >> > > > > > listing of limitations (some look serious such as data > bleed > > > from > > > >> > > other > > > >> > > > > > regions in WALs, but maybe I don't care for my use > case...) > > -- > > > >> > needs > > > >> > > to > > > >> > > > > > accompany the merge. Lets fold them into the user doc. in > > the > > > >> > > technical > > > >> > > > > > overview area as suggested so user expectations are > properly > > > >> > managed > > > >> > > > > > (otherwise, they expect the world and will just give up > when > > > we > > > >> > fall > > > >> > > > > > short). Vladimir did a list of what is in each of the > phases > > > >> above > > > >> > > > which > > > >> > > > > > would serve as a good start. > > > >> > > > > > + Is this feature 'experimental' (Matteo asks above). I'd > > > prefer > > > >> it > > > >> > > is > > > >> > > > > > not. If it is, it should be labelled all over that it is > > so. I > > > >> see > > > >> > > > > current > > > >> > > > > > state called out as a '... technical preview feature'. > Does > > > this > > > >> > mean > > > >> > > > > > not-for-users? > > > >> > > > > > > > > >> > > > > > St.Ack > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu < > > yuzhih...@gmail.com> > > > >> > wrote: > > > >> > > > > > > > > >> > > > > >> Sean: > > > >> > > > > >> Do you have more comments ? > > > >> > > > > >> > > > >> > > > > >> Cheers > > > >> > > > > >> > > > >> > > > > >> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov < > > > >> > > > > vladrodio...@gmail.com > > > >> > > > > >> > > > > >> > > > > >> wrote: > > > >> > > > > >> > > > >> > > > > >> > Sean, > > > >> > > > > >> > > > > >> > > > > >> > Backup/Restore can fail due to various reasons: network > > > outage > > > >> > > > > (cluster > > > >> > > > > >> > wide), various time-outs in HBase and HDFS layer, M/R > > > failure > > > >> > due > > > >> > > to > > > >> > > > > >> "HDFS > > > >> > > > > >> > exceeded quota", user error (manual deletion of data) > and > > > so > > > >> on > > > >> > so > > > >> > > > on. > > > >> > > > > >> That > > > >> > > > > >> > is impossible to enumerate all possible types of > failures > > > in a > > > >> > > > > >> distributed > > > >> > > > > >> > system - that is not our goal/task. > > > >> > > > > >> > > > > >> > > > > >> > We focus completely on backup system table consistency > > in a > > > >> > > presence > > > >> > > > > of > > > >> > > > > >> any > > > >> > > > > >> > type of failure. That is what I call "tolerance to > > > failures". > > > >> > > > > >> > > > > >> > > > > >> > On a failure: > > > >> > > > > >> > > > > >> > > > > >> > BACKUP. All backup system information (prior to backup) > > > will > > > >> be > > > >> > > > > restored > > > >> > > > > >> > and all temporary data, related to a failed session, in > > > HDFS > > > >> > will > > > >> > > be > > > >> > > > > >> > deleted > > > >> > > > > >> > RESTORE. We do not care about system data, because > > restore > > > >> does > > > >> > > not > > > >> > > > > >> change > > > >> > > > > >> > it. Temporary data in HDFS will be cleaned up and table > > > will > > > >> be > > > >> > > in a > > > >> > > > > >> state > > > >> > > > > >> > back to where it was before operation started. > > > >> > > > > >> > > > > >> > > > > >> > This is what user should expect in case of a failure. > > > >> > > > > >> > > > > >> > > > > >> > -Vlad > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > -Vlad > > > >> > > > > >> > > > > >> > > > > >> > On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey < > > > >> bus...@apache.org > > > >> > > > > > >> > > > > wrote: > > > >> > > > > >> > > > > >> > > > > >> > > Failing in a consistent way, with docs that explain > the > > > >> > various > > > >> > > > > >> > > expected failures would be sufficient. > > > >> > > > > >> > > > > > >> > > > > >> > > On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov > > > >> > > > > >> > > <vladrodio...@gmail.com> wrote: > > > >> > > > > >> > > > Do not worry Sean, doc is coming today as a preview > > and > > > >> our > > > >> > > > writer > > > >> > > > > >> > Frank > > > >> > > > > >> > > > will be working on a putting it into Apache repo. > > > >> Timeline > > > >> > > > > depends > > > >> > > > > >> on > > > >> > > > > >> > > > Franks schedule but I hope we will get it rather > > sooner > > > >> than > > > >> > > > > later. > > > >> > > > > >> > > > > > > >> > > > > >> > > > As for failure testing, we are focusing only on a > > > >> consistent > > > >> > > > state > > > >> > > > > >> of > > > >> > > > > >> > > > backup system data in a presence of any type of > > > failures, > > > >> We > > > >> > > are > > > >> > > > > not > > > >> > > > > >> > > going > > > >> > > > > >> > > > to implement anything more "fancy", than that. We > > > allow > > > >> > both: > > > >> > > > > >> backup > > > >> > > > > >> > and > > > >> > > > > >> > > > restore to fail. What we do not allow is to have > > system > > > >> data > > > >> > > > > >> corrupted. > > > >> > > > > >> > > > Will it suffice for you? Do you have any other > > > concerns, > > > >> you > > > >> > > > want > > > >> > > > > >> us to > > > >> > > > > >> > > > address? > > > >> > > > > >> > > > > > > >> > > > > >> > > > -Vlad > > > >> > > > > >> > > > > > > >> > > > > >> > > > > > > >> > > > > >> > > > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey < > > > >> > > bus...@apache.org > > > >> > > > > > > > >> > > > > >> > wrote: > > > >> > > > > >> > > > > > > >> > > > > >> > > >> "docs will come to Apache soon" does not address > my > > > >> concern > > > >> > > > > around > > > >> > > > > >> > docs > > > >> > > > > >> > > at > > > >> > > > > >> > > >> all, unless said docs have already made it into > the > > > >> project > > > >> > > > > repo. I > > > >> > > > > >> > > don't > > > >> > > > > >> > > >> want third party resources for using a major and > > > >> important > > > >> > > > > feature > > > >> > > > > >> of > > > >> > > > > >> > > the > > > >> > > > > >> > > >> project, I want us to provide end users with what > > they > > > >> need > > > >> > > to > > > >> > > > > get > > > >> > > > > >> the > > > >> > > > > >> > > job > > > >> > > > > >> > > >> done. > > > >> > > > > >> > > >> > > > >> > > > > >> > > >> I see some calls for patience on the failure > > testing, > > > but > > > >> > the > > > >> > > > > >> appeal > > > >> > > > > >> > to > > > >> > > > > >> > > us > > > >> > > > > >> > > >> having done a bad job of requiring proper tests of > > > >> previous > > > >> > > > > >> features > > > >> > > > > >> > > just > > > >> > > > > >> > > >> makes me more concerned about not getting them > > here. I > > > >> > don't > > > >> > > > want > > > >> > > > > >> to > > > >> > > > > >> > set > > > >> > > > > >> > > >> yet another bad example that will then be pointed > to > > > in > > > >> the > > > >> > > > > future. > > > >> > > > > >> > > >> > > > >> > > > > >> > > >> On Sep 8, 2016 10:50, "Ted Yu" < > yuzhih...@gmail.com > > > > > > >> > wrote: > > > >> > > > > >> > > >> > > > >> > > > > >> > > >> > Is there any concern which is not addressed ? > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > Do we need another Vote thread ? > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > Thanks > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell < > > > >> > > > > >> apurt...@apache.org > > > >> > > > > >> > > > > > >> > > > > >> > > >> > wrote: > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > > Vlad, > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > I apologize for using the term 'half-baked' > in a > > > way > > > >> > that > > > >> > > > > could > > > >> > > > > >> > > seem a > > > >> > > > > >> > > >> > > description of HBASE-7912. I meant that as a > > > general > > > >> > > > > >> hypothetical. > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir > > Rodionov > > > < > > > >> > > > > >> > > >> > vladrodio...@gmail.com> > > > >> > > > > >> > > >> > > wrote: > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > > >> I'm not sure that "There is already lots > of > > > >> > > half-baked > > > >> > > > > >> code > > > >> > > > > >> > in > > > >> > > > > >> > > the > > > >> > > > > >> > > >> > > > branch, > > > >> > > > > >> > > >> > > > so what's the harm in adding more?" > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > >> > > > I meant - not production - ready yet. This > is > > > 2.0 > > > >> > > > > development > > > >> > > > > >> > > branch > > > >> > > > > >> > > >> > and, > > > >> > > > > >> > > >> > > > hence many features are in works, > > > >> > > > > >> > > >> > > > not being tested well etc. I do not consider > > > backup > > > >> > as > > > >> > > > half > > > >> > > > > >> > baked > > > >> > > > > >> > > >> > > feature - > > > >> > > > > >> > > >> > > > it has passed our internal QA and has very > > good > > > >> doc, > > > >> > > > which > > > >> > > > > we > > > >> > > > > >> > will > > > >> > > > > >> > > >> > > provide > > > >> > > > > >> > > >> > > > to Apache shortly. > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > >> > > > -Vlad > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > >> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew > > Purtell < > > > >> > > > > >> > > apurt...@apache.org> > > > >> > > > > >> > > >> > > > wrote: > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > >> > > > > We shouldn't admit half baked changes that > > > won't > > > >> be > > > >> > > > > >> finished. > > > >> > > > > >> > > >> However > > > >> > > > > >> > > >> > > in > > > >> > > > > >> > > >> > > > > this case the crew working on this feature > > are > > > >> long > > > >> > > > > timers > > > >> > > > > >> and > > > >> > > > > >> > > less > > > >> > > > > >> > > >> > > > likely > > > >> > > > > >> > > >> > > > > than just about anyone to leave something > > in a > > > >> half > > > >> > > > baked > > > >> > > > > >> > > state. Of > > > >> > > > > >> > > >> > > > course > > > >> > > > > >> > > >> > > > > there is no guarantee how anything will > turn > > > out, > > > >> > > but I > > > >> > > > > am > > > >> > > > > >> > > willing > > > >> > > > > >> > > >> to > > > >> > > > > >> > > >> > > > take > > > >> > > > > >> > > >> > > > > a little on faith if they feel their best > > path > > > >> > > forward > > > >> > > > > now > > > >> > > > > >> is > > > >> > > > > >> > to > > > >> > > > > >> > > >> > merge > > > >> > > > > >> > > >> > > to > > > >> > > > > >> > > >> > > > > trunk. I only wish I had bandwidth to have > > > done > > > >> > some > > > >> > > > real > > > >> > > > > >> > > kicking > > > >> > > > > >> > > >> of > > > >> > > > > >> > > >> > > the > > > >> > > > > >> > > >> > > > > tires by now. Maybe this week. > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > (Yes, I'm using some of that time for this > > > email > > > >> > :-) > > > >> > > > but > > > >> > > > > I > > > >> > > > > >> > type > > > >> > > > > >> > > >> > fast.) > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > That said, I would like to agitate for > > making > > > 2.0 > > > >> > > more > > > >> > > > > real > > > >> > > > > >> > and > > > >> > > > > >> > > >> spend > > > >> > > > > >> > > >> > > > some > > > >> > > > > >> > > >> > > > > time on it now that I'm winding down with > > > 0.98. I > > > >> > > think > > > >> > > > > >> that > > > >> > > > > >> > > means > > > >> > > > > >> > > >> > > > > branching for 2.0 real soon now and even > > > evicting > > > >> > > > things > > > >> > > > > >> from > > > >> > > > > >> > > 2.0 > > > >> > > > > >> > > >> > > branch > > > >> > > > > >> > > >> > > > > that aren't finished or stable, leaving > them > > > only > > > >> > > once > > > >> > > > > >> again > > > >> > > > > >> > in > > > >> > > > > >> > > the > > > >> > > > > >> > > >> > > > master > > > >> > > > > >> > > >> > > > > branch. Or, maybe just evicting them. > Let's > > > take > > > >> it > > > >> > > > case > > > >> > > > > by > > > >> > > > > >> > > case. > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > I think this feature can come in > relatively > > > >> safely. > > > >> > > As > > > >> > > > > >> added > > > >> > > > > >> > > >> > insurance, > > > >> > > > > >> > > >> > > > > let's admit the possibility it could be > > > reverted > > > >> on > > > >> > > the > > > >> > > > > 2.0 > > > >> > > > > >> > > branch > > > >> > > > > >> > > >> if > > > >> > > > > >> > > >> > > > folks > > > >> > > > > >> > > >> > > > > working on stabilizing 2.0 decide to evict > > it > > > >> > because > > > >> > > > it > > > >> > > > > is > > > >> > > > > >> > > >> > unfinished > > > >> > > > > >> > > >> > > or > > > >> > > > > >> > > >> > > > > unstable, because that certainly can > > happen. I > > > >> > would > > > >> > > > > >> expect if > > > >> > > > > >> > > talk > > > >> > > > > >> > > >> > > like > > > >> > > > > >> > > >> > > > > that starts, we'd get help finishing or > > > >> stabilizing > > > >> > > > > what's > > > >> > > > > >> > under > > > >> > > > > >> > > >> > > > discussion > > > >> > > > > >> > > >> > > > > for revert. Or, we'd have a revert. Either > > way > > > >> the > > > >> > > > > outcome > > > >> > > > > >> is > > > >> > > > > >> > > >> > > acceptable. > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > On Wed, Sep 7, 2016 at 8:56 AM, Dima > Spivak > > < > > > >> > > > > >> > > dimaspi...@apache.org > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > > > wrote: > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > > I'm not sure that "There is already lots > > of > > > >> > > > half-baked > > > >> > > > > >> code > > > >> > > > > >> > in > > > >> > > > > >> > > >> the > > > >> > > > > >> > > >> > > > > branch, > > > >> > > > > >> > > >> > > > > > so what's the harm in adding more?" is a > > > good > > > >> > code > > > >> > > > > commit > > > >> > > > > >> > > >> > philosophy > > > >> > > > > >> > > >> > > > for > > > >> > > > > >> > > >> > > > > a > > > >> > > > > >> > > >> > > > > > fault-tolerant distributed data store. > ;) > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > >> > > > > > More seriously, a lack of test coverage > > for > > > >> > > existing > > > >> > > > > >> > features > > > >> > > > > >> > > >> > > shouldn't > > > >> > > > > >> > > >> > > > > be > > > >> > > > > >> > > >> > > > > > used as justification for introducing > new > > > >> > features > > > >> > > > with > > > >> > > > > >> the > > > >> > > > > >> > > same > > > >> > > > > >> > > >> > > > > > shortcomings. Ultimately, it's the end > > user > > > who > > > >> > > will > > > >> > > > > feel > > > >> > > > > >> > the > > > >> > > > > >> > > >> pain, > > > >> > > > > >> > > >> > > so > > > >> > > > > >> > > >> > > > > > shouldn't we do everything we can to > > > mitigate > > > >> > that? > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > >> > > > > > -Dima > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > >> > > > > > On Wed, Sep 7, 2016 at 8:46 AM, Vladimir > > > >> > Rodionov < > > > >> > > > > >> > > >> > > > > vladrodio...@gmail.com> > > > >> > > > > >> > > >> > > > > > wrote: > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > >> > > > > > > Sean, > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > * have docs > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > Agree. We have a doc and backup is the > > > most > > > >> > > > > documented > > > >> > > > > >> > > feature > > > >> > > > > >> > > >> > :), > > > >> > > > > >> > > >> > > we > > > >> > > > > >> > > >> > > > > > will > > > >> > > > > >> > > >> > > > > > > release it shortly to Apache. > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > * have sunny-day correctness tests > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > Feature has close to 60 test cases, > > which > > > >> run > > > >> > > for > > > >> > > > > >> approx > > > >> > > > > >> > 30 > > > >> > > > > >> > > >> min. > > > >> > > > > >> > > >> > > We > > > >> > > > > >> > > >> > > > > can > > > >> > > > > >> > > >> > > > > > > add more, if community do not mind :) > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > * have correctness-in-face-of-failure > > > tests > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > Any examples of these tests in > existing > > > >> > features? > > > >> > > > In > > > >> > > > > >> > works, > > > >> > > > > >> > > we > > > >> > > > > >> > > >> > > have a > > > >> > > > > >> > > >> > > > > > clear > > > >> > > > > >> > > >> > > > > > > understanding of what should be done > by > > > the > > > >> > time > > > >> > > of > > > >> > > > > 2.0 > > > >> > > > > >> > > >> release. > > > >> > > > > >> > > >> > > > > > > That is very close goal for us, to > > verify > > > IT > > > >> > > monkey > > > >> > > > > for > > > >> > > > > >> > > >> existing > > > >> > > > > >> > > >> > > > code. > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > * don't rely on things outside of > HBase > > > for > > > >> > > normal > > > >> > > > > >> > operation > > > >> > > > > >> > > >> > (okay > > > >> > > > > >> > > >> > > > for > > > >> > > > > >> > > >> > > > > > > advanced operation) > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > We do not. > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > Enormous time has been spent already > on > > > the > > > >> > > > > development > > > >> > > > > >> > and > > > >> > > > > >> > > >> > testing > > > >> > > > > >> > > >> > > > the > > > >> > > > > >> > > >> > > > > > > feature, it has passed our internal > > tests > > > and > > > >> > > many > > > >> > > > > >> rounds > > > >> > > > > >> > of > > > >> > > > > >> > > >> code > > > >> > > > > >> > > >> > > > > reviews > > > >> > > > > >> > > >> > > > > > > by HBase committers. We do not mind if > > > >> someone > > > >> > > from > > > >> > > > > >> HBase > > > >> > > > > >> > > >> > community > > > >> > > > > >> > > >> > > > > > > (outside of HW) will review the code, > > but > > > it > > > >> > will > > > >> > > > > >> probably > > > >> > > > > >> > > >> takes > > > >> > > > > >> > > >> > > > > forever > > > >> > > > > >> > > >> > > > > > to > > > >> > > > > >> > > >> > > > > > > wait for volunteer?, the feature is > > quite > > > >> large > > > >> > > > (1MB+ > > > >> > > > > >> > > >> cumulative > > > >> > > > > >> > > >> > > > patch) > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > 2.0 branch is full of half baked > > features, > > > >> most > > > >> > > of > > > >> > > > > them > > > >> > > > > >> > are > > > >> > > > > >> > > in > > > >> > > > > >> > > >> > > active > > > >> > > > > >> > > >> > > > > > > development, therefore I am not > > following > > > you > > > >> > > here, > > > >> > > > > >> Sean? > > > >> > > > > >> > > Why > > > >> > > > > >> > > >> > > > > HBASE-7912 > > > >> > > > > >> > > >> > > > > > is > > > >> > > > > >> > > >> > > > > > > not good enough yet to be integrated > > into > > > 2.0 > > > >> > > > branch? > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > -Vlad > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > On Wed, Sep 7, 2016 at 8:23 AM, Sean > > > Busbey < > > > >> > > > > >> > > bus...@apache.org > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > > > wrote: > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > > On Tue, Sep 6, 2016 at 10:36 PM, > Josh > > > >> Elser < > > > >> > > > > >> > > >> > > josh.el...@gmail.com> > > > >> > > > > >> > > >> > > > > > > wrote: > > > >> > > > > >> > > >> > > > > > > > > So, the answer to Sean's original > > > >> question > > > >> > is > > > >> > > > "as > > > >> > > > > >> > > robust as > > > >> > > > > >> > > >> > > > > snapshots > > > >> > > > > >> > > >> > > > > > > > > presently are"? (independence of > > > >> > > backup/restore > > > >> > > > > >> > failure > > > >> > > > > >> > > >> > > tolerance > > > >> > > > > >> > > >> > > > > > from > > > >> > > > > >> > > >> > > > > > > > > snapshot failure tolerance) > > > >> > > > > >> > > >> > > > > > > > > > > > >> > > > > >> > > >> > > > > > > > > Is this just a question WRT > context > > of > > > >> the > > > >> > > > > change, > > > >> > > > > >> or > > > >> > > > > >> > > is it > > > >> > > > > >> > > >> > > means > > > >> > > > > >> > > >> > > > > > for a > > > >> > > > > >> > > >> > > > > > > > veto > > > >> > > > > >> > > >> > > > > > > > > from you, Sean? Just trying to > make > > > sure > > > >> > I'm > > > >> > > > > >> following > > > >> > > > > >> > > >> along > > > >> > > > > >> > > >> > > > > > > adequately. > > > >> > > > > >> > > >> > > > > > > > > > > > >> > > > > >> > > >> > > > > > > > > > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > I'd say ATM I'm -0, bordering on -1 > > but > > > not > > > >> > for > > > >> > > > > >> reasons > > > >> > > > > >> > I > > > >> > > > > >> > > can > > > >> > > > > >> > > >> > > > > > articulate > > > >> > > > > >> > > >> > > > > > > > well. > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > Here's an attempt. > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > We've been trying to move, as a > > > community, > > > >> > > > towards > > > >> > > > > >> > > minimizing > > > >> > > > > >> > > >> > > risk > > > >> > > > > >> > > >> > > > to > > > >> > > > > >> > > >> > > > > > > > downstream folks by getting > "complete > > > >> enough > > > >> > > for > > > >> > > > > use" > > > >> > > > > >> > > gates > > > >> > > > > >> > > >> in > > > >> > > > > >> > > >> > > > place > > > >> > > > > >> > > >> > > > > > > > before we introduce new features. > This > > > was > > > >> > > > spurred > > > >> > > > > >> by a > > > >> > > > > >> > > some > > > >> > > > > >> > > >> > > > features > > > >> > > > > >> > > >> > > > > > > > getting in half-baked and never > making > > > it > > > >> to > > > >> > > "can > > > >> > > > > >> really > > > >> > > > > >> > > use" > > > >> > > > > >> > > >> > > > status > > > >> > > > > >> > > >> > > > > > > > (I'm thinking of distributed log > > replay > > > and > > > >> > the > > > >> > > > > >> zk-less > > > >> > > > > >> > > >> > > assignment > > > >> > > > > >> > > >> > > > > > > > stuff, I don't recall if there was > > > more). > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > The gates, generally, included > things > > > like: > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > * have docs > > > >> > > > > >> > > >> > > > > > > > * have sunny-day correctness tests > > > >> > > > > >> > > >> > > > > > > > * have > correctness-in-face-of-failure > > > tests > > > >> > > > > >> > > >> > > > > > > > * don't rely on things outside of > > HBase > > > for > > > >> > > > normal > > > >> > > > > >> > > operation > > > >> > > > > >> > > >> > > (okay > > > >> > > > > >> > > >> > > > > for > > > >> > > > > >> > > >> > > > > > > > advanced operation) > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > As an example, we kept the MOB work > > off > > > in > > > >> a > > > >> > > > branch > > > >> > > > > >> and > > > >> > > > > >> > > out > > > >> > > > > >> > > >> of > > > >> > > > > >> > > >> > > > master > > > >> > > > > >> > > >> > > > > > > > until it could pass these criteria. > > The > > > big > > > >> > > > > exemption > > > >> > > > > >> > > we've > > > >> > > > > >> > > >> had > > > >> > > > > >> > > >> > > to > > > >> > > > > >> > > >> > > > > > > > this was the hbase-spark > integration, > > > where > > > >> > we > > > >> > > > all > > > >> > > > > >> > agreed > > > >> > > > > >> > > it > > > >> > > > > >> > > >> > > could > > > >> > > > > >> > > >> > > > > > > > land in master because it was very > > well > > > >> > > isolated > > > >> > > > > (the > > > >> > > > > >> > > slide > > > >> > > > > >> > > >> > away > > > >> > > > > >> > > >> > > > from > > > >> > > > > >> > > >> > > > > > > > including docs as a first-class part > > of > > > >> > > building > > > >> > > > up > > > >> > > > > >> that > > > >> > > > > >> > > >> > > > integration > > > >> > > > > >> > > >> > > > > > > > has led me to doubt the wisdom of > this > > > >> > > decision). > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > We've also been treating inclusion > in > > a > > > >> > > "probably > > > >> > > > > >> will > > > >> > > > > >> > be > > > >> > > > > >> > > >> > > released > > > >> > > > > >> > > >> > > > to > > > >> > > > > >> > > >> > > > > > > > downstream" branches as a higher > bar, > > > >> > requiring > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > * don't moderately impact > performance > > > when > > > >> > the > > > >> > > > > >> feature > > > >> > > > > >> > > isn't > > > >> > > > > >> > > >> in > > > >> > > > > >> > > >> > > use > > > >> > > > > >> > > >> > > > > > > > * don't severely impact performance > > when > > > >> the > > > >> > > > > feature > > > >> > > > > >> is > > > >> > > > > >> > in > > > >> > > > > >> > > >> use > > > >> > > > > >> > > >> > > > > > > > * either default-to-on or show > enough > > > >> demand > > > >> > to > > > >> > > > > >> believe > > > >> > > > > >> > a > > > >> > > > > >> > > >> > > > non-trivial > > > >> > > > > >> > > >> > > > > > > > number of folks will turn the > feature > > on > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > The above has kept MOB and > hbase-spark > > > >> > > > integration > > > >> > > > > >> out > > > >> > > > > >> > of > > > >> > > > > >> > > >> > > branch-1, > > > >> > > > > >> > > >> > > > > > > > presumably while they've "gotten > more > > > >> stable" > > > >> > > in > > > >> > > > > >> master > > > >> > > > > >> > > from > > > >> > > > > >> > > >> > the > > > >> > > > > >> > > >> > > > odd > > > >> > > > > >> > > >> > > > > > > > vendor inclusion. > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > Are we going to have a 2.0 release > > > before > > > >> the > > > >> > > end > > > >> > > > > of > > > >> > > > > >> the > > > >> > > > > >> > > >> year? > > > >> > > > > >> > > >> > > > We're > > > >> > > > > >> > > >> > > > > > > > coming up on 1.5 years since the > > > release of > > > >> > > > version > > > >> > > > > >> 1.0; > > > >> > > > > >> > > >> seems > > > >> > > > > >> > > >> > > like > > > >> > > > > >> > > >> > > > > > > > it's about time, though I haven't > seen > > > any > > > >> > > > concrete > > > >> > > > > >> > plans > > > >> > > > > >> > > >> this > > > >> > > > > >> > > >> > > > year. > > > >> > > > > >> > > >> > > > > > > > Presuming we are going to have one > by > > > the > > > >> end > > > >> > > of > > > >> > > > > the > > > >> > > > > >> > > year, it > > > >> > > > > >> > > >> > > > seems a > > > >> > > > > >> > > >> > > > > > > > bit close to still be adding in > > > "features > > > >> > that > > > >> > > > need > > > >> > > > > >> > > maturing" > > > >> > > > > >> > > >> > on > > > >> > > > > >> > > >> > > > the > > > >> > > > > >> > > >> > > > > > > > branch. > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > The lack of a concrete plan for 2.0 > > > keeps > > > >> me > > > >> > > from > > > >> > > > > >> > > considering > > > >> > > > > >> > > >> > > these > > > >> > > > > >> > > >> > > > > > > > things blocker at the moment. But I > > know > > > >> > first > > > >> > > > hand > > > >> > > > > >> how > > > >> > > > > >> > > much > > > >> > > > > >> > > >> > > > trouble > > > >> > > > > >> > > >> > > > > > > > folks have had with other features > > that > > > >> have > > > >> > > gone > > > >> > > > > >> into > > > >> > > > > >> > > >> > downstream > > > >> > > > > >> > > >> > > > > > > > facing releases without robustness > > > checks > > > >> > (i.e. > > > >> > > > > >> > > replication), > > > >> > > > > >> > > >> > and > > > >> > > > > >> > > >> > > > I'm > > > >> > > > > >> > > >> > > > > > > > concerned about what we're setting > up > > if > > > >> 2.0 > > > >> > > goes > > > >> > > > > out > > > >> > > > > >> > with > > > >> > > > > >> > > >> this > > > >> > > > > >> > > >> > > > > > > > feature in its current state. > > > >> > > > > >> > > >> > > > > > > > > > > >> > > > > >> > > >> > > > > > > > > > >> > > > > >> > > >> > > > > > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > -- > > > >> > > > > >> > > >> > > > > Best regards, > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > - Andy > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > Problems worthy of attack prove their > worth > > by > > > >> > > hitting > > > >> > > > > >> back. - > > > >> > > > > >> > > Piet > > > >> > > > > >> > > >> > > Hein > > > >> > > > > >> > > >> > > > > (via Tom White) > > > >> > > > > >> > > >> > > > > > > > >> > > > > >> > > >> > > > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > -- > > > >> > > > > >> > > >> > > Best regards, > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > - Andy > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > Problems worthy of attack prove their worth by > > > >> hitting > > > >> > > > back. > > > >> > > > > - > > > >> > > > > >> > Piet > > > >> > > > > >> > > >> Hein > > > >> > > > > >> > > >> > > (via Tom White) > > > >> > > > > >> > > >> > > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > -- > > > busbey > > > > > >
