In spite of repeated requests for eng summary of state of this feature -- summary of what is in 2.0, what is not, what the capabilities are, how well it has been tested and at what scale -- all I get, when the requests are not ignored, are pointers to lists of ill-describing jiras and some pending user facing doc update.
For other features, mob or region server groups, I know that they have been running at scale in production for as much as a year and more. I have some confidence these items basically work. For backup/restore I have no such sense even after spending time in review and trying to use the feature. As release manager, I have say over what makes it into a release. Unless the work is done to convince me that backup/restore is more than a lump of code and a few unit tests that can pass on some fellows laptop, I am going to kick it out of branch-2. Let the feature harden more in master branch before it ships in a release. S On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <[email protected]> wrote: > >> Have I grasped the state of things correctly, Vlad? > > Josh, the only thing which is still pending is doc update. All other > features are good to have but not a blockers for 2.0 release. > > -Vlad > > On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <[email protected] > > > wrote: > > > >> What testing and at what > > >> scale has testing been done? > > > > Do we have have that for other features? > > > > > > On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov < > [email protected] > > > wrote: > > > >> >> It asks: "How do I figure what of backup/restore feature is going to > >> be in > >> >>hbase-2.0.0? > >> > >> Hmm, wait for doc update. > >> > >> > >> On Fri, Sep 8, 2017 at 2:39 PM, Stack <[email protected]> wrote: > >> > >>> HBASE-14414 is a JIRA with a list of random seeming issues w/ > >>> non-descript > >>> summaries: "Add nonce support to TableBackupProcedure, BackupID must > >>> include backup set name, ...". The last comment in that issue is from > >>> July. > >>> It asks: "How do I figure what of backup/restore feature is going to be > >>> in > >>> hbase-2.0.0? Thanks Vladimir Rodionov > >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov > >>> >." > >>> to which there is no answer. Doc update is TODO. > >>> > >>> Where is the summary of the capability in hbase-2? What testing and at > >>> what > >>> scale has testing been done? Is this 'stable or experimental'? If I > can't > >>> get basic info on this feature though I ask repeatedly, what hope does > >>> the > >>> poor old operator have? > >>> > >>> St.Ack > >>> > >>> > >>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov < > >>> [email protected]> > >>> wrote: > >>> > >>> > HBASE-14414 > >>> > > >>> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <[email protected]> wrote: > >>> > > >>> > > Where do I go to get the current status of this feature? Looking in > >>> JIRA > >>> > I > >>> > > see loads of issues open against backup including some against > >>> > hbase-2.0.0 > >>> > > and no progress being made that I can discern. > >>> > > > >>> > > Thanks, > >>> > > S > >>> > > > >>> > > > >>> > > > >>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <[email protected]> wrote: > >>> > > > >>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <[email protected]> wrote: > >>> > > > > >>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov < > >>> > > >> [email protected]> wrote: > >>> > > >> > >>> > > >>> >> and/or he answered most of the review feedback > >>> > > >>> > >>> > > >>> No, questions are still open, but I do not see any blockers and > >>> we > >>> > have > >>> > > >>> HBASE-16940 to address these questions. > >>> > > >>> > >>> > > >>> > >>> > > >> Agree. No blockers but stuff that should be dealt with (No one > >>> will > >>> > pay > >>> > > >> me any attention once merge goes in -- smile). > >>> > > >> > >>> > > >> > >>> > > > Let me clarify the above. I want review addressed before merge > >>> happens. > >>> > > > Sorry if any confusion. > >>> > > > St.Ack > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > >> St.Ack > >>> > > >> > >>> > > >> > >>> > > >> > >>> > > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das < > >>> [email protected]> > >>> > > >>> wrote: > >>> > > >>> > >>> > > >>> > Hi Stack, hats off to you for spending so much time on this! > >>> > Thanks! > >>> > > >>> From > >>> > > >>> > my understanding, Vlad has raised follow-up jiras for the > >>> issues > >>> > you > >>> > > >>> > raised, and/or he answered most of the review feedback. So, > do > >>> you > >>> > > >>> think we > >>> > > >>> > could do a merge vote now? > >>> > > >>> > Devaraj. > >>> > > >>> > ________________________________________ > >>> > > >>> > From: Vladimir Rodionov <[email protected]> > >>> > > >>> > Sent: Monday, November 21, 2016 8:34 PM > >>> > > >>> > To: [email protected] > >>> > > >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch > >>> > HBASE-7912 > >>> > > >>> > > >>> > > >>> > >> I have spent a good bit of time reviewing and testing this > >>> > > feature. > >>> > > >>> I > >>> > > >>> > would > >>> > > >>> > >> like my review and concerns addressed and I'd like it to > be > >>> > clear > >>> > > >>> how; > >>> > > >>> > >> either explicit follow-on issues, pointers to where in the > >>> patch > >>> > > or > >>> > > >>> doc > >>> > > >>> > my > >>> > > >>> > >> remarks have been catered to, etc. Until then, I am > against > >>> > > commit. > >>> > > >>> > > >>> > > >>> > Stack, mega patch review comments will be addressed in the > >>> > dedicated > >>> > > >>> JIRA: > >>> > > >>> > HBASE-16940 > >>> > > >>> > I have open several other JIRAs to address your other > comments > >>> (not > >>> > > on > >>> > > >>> > review board). > >>> > > >>> > > >>> > > >>> > Details are here (end of the thread): > >>> > > >>> > https://issues.apache.org/jira/browse/HBASE-14123 > >>> > > >>> > > >>> > > >>> > Let me know what else should we do to move merge forward. > >>> > > >>> > > >>> > > >>> > -Vlad > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > On Fri, Nov 18, 2016 at 4:54 PM, Stack <[email protected]> > >>> wrote: > >>> > > >>> > > >>> > > >>> > > On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu < > [email protected] > >>> > > >>> > > wrote: > >>> > > >>> > > > >>> > > >>> > > > Thanks, Matteo. > >>> > > >>> > > > > >>> > > >>> > > > bq. restore is not clear if given an incremental id it > >>> will do > >>> > > the > >>> > > >>> full > >>> > > >>> > > > restore from full up to that point or if i need to apply > >>> > manually > >>> > > >>> > > > everything > >>> > > >>> > > > > >>> > > >>> > > > The restore takes into consideration of the dependent > >>> > backup(s). > >>> > > >>> > > > So there is no need to apply preceding backup(s) > manually. > >>> > > >>> > > > > >>> > > >>> > > > > >>> > > >>> > > I ask this question on the issue. It is not clear from the > >>> usage > >>> > or > >>> > > >>> doc > >>> > > >>> > how > >>> > > >>> > > to run a restore from incremental. Can you fix in doc and > >>> usage > >>> > how > >>> > > >>> so I > >>> > > >>> > > can be clear and try it. Currently I am stuck verifying a > >>> round > >>> > > trip > >>> > > >>> > backup > >>> > > >>> > > restore made of incrementals. > >>> > > >>> > > > >>> > > >>> > > Thanks, > >>> > > >>> > > S > >>> > > >>> > > > >>> > > >>> > > > >>> > > >>> > > > >>> > > >>> > > > On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi < > >>> > > >>> > > [email protected]> > >>> > > >>> > > > wrote: > >>> > > >>> > > > > >>> > > >>> > > > > I did one last pass to the mega patch. I don't see > >>> anything > >>> > > major > >>> > > >>> > that > >>> > > >>> > > > > should block the merge. > >>> > > >>> > > > > > >>> > > >>> > > > > - most of the code is isolated in the backup package > >>> > > >>> > > > > - all the backup code is client side > >>> > > >>> > > > > - there are few changes to the server side, mainly for > >>> > > cleaners, > >>> > > >>> wal > >>> > > >>> > > > > rolling and similar (which is ok) > >>> > > >>> > > > > - there is a good number of tests, and an integration > >>> test > >>> > > >>> > > > > > >>> > > >>> > > > > the code seems to have still some left overs from the > old > >>> > > >>> > > implementation, > >>> > > >>> > > > > and some stuff needs a cleanup. but I don't think this > >>> should > >>> > > be > >>> > > >>> used > >>> > > >>> > > as > >>> > > >>> > > > an > >>> > > >>> > > > > argument to block the merge. I think the guys will keep > >>> > working > >>> > > >>> on > >>> > > >>> > this > >>> > > >>> > > > and > >>> > > >>> > > > > they may also get help of others once the patch is in > >>> master. > >>> > > >>> > > > > > >>> > > >>> > > > > I still have my concerns about the current limitations, > >>> but > >>> > > >>> these are > >>> > > >>> > > > > things already planned for phase 3, so some of this > >>> stuff may > >>> > > >>> even be > >>> > > >>> > > in > >>> > > >>> > > > > the final 2.0. > >>> > > >>> > > > > but as long as we have a "current limitations" section > >>> in the > >>> > > >>> user > >>> > > >>> > > guide > >>> > > >>> > > > > mentioning important stuff like the ones below, I'm ok > >>> with > >>> > it. > >>> > > >>> > > > > - if you write to the table with Durability.SKIP_WALS > >>> your > >>> > > data > >>> > > >>> will > >>> > > >>> > > not > >>> > > >>> > > > > be in the incremental-backup > >>> > > >>> > > > > - if you bulkload files that data will not be in the > >>> > > incremental > >>> > > >>> > > backup > >>> > > >>> > > > > (HBASE-14417) > >>> > > >>> > > > > - the incremental backup will not only contains the > >>> data of > >>> > > the > >>> > > >>> > table > >>> > > >>> > > > you > >>> > > >>> > > > > specified but also the regions from other tables that > >>> are on > >>> > > the > >>> > > >>> same > >>> > > >>> > > set > >>> > > >>> > > > > of RSs (HBASE-14141) ...maybe a note about security > >>> around > >>> > this > >>> > > >>> topic > >>> > > >>> > > > > - the incremental backup will not contains just the > >>> "latest > >>> > > row" > >>> > > >>> > > between > >>> > > >>> > > > > backup A and B, but it will also contains all the > updates > >>> > > >>> occurred in > >>> > > >>> > > > > between. but the restore does not allow you to restore > >>> up to > >>> > a > >>> > > >>> > certain > >>> > > >>> > > > > point in time, the restore will always be up to the > >>> "latest > >>> > > >>> backup > >>> > > >>> > > > point". > >>> > > >>> > > > > - you should limit the number of "incremental" up to N > >>> (or > >>> > > maybe > >>> > > >>> > > SIZE), > >>> > > >>> > > > to > >>> > > >>> > > > > avoid replay time becoming the bottleneck. > (HBASE-14135) > >>> > > >>> > > > > > >>> > > >>> > > > > I'll be ok even with the above not being in the final > >>> 2.0, > >>> > > >>> > > > > but i'd like to see as blocker for the final 2.0 (not > the > >>> > > merge) > >>> > > >>> > > > > - the backup code moved in an hbase-backup module > >>> > > >>> > > > > - and some more work around tools, especially to try > to > >>> > unify > >>> > > >>> and > >>> > > >>> > make > >>> > > >>> > > > > simple the backup experience (simple example: in some > >>> case > >>> > > there > >>> > > >>> is a > >>> > > >>> > > > > backup_id argument in others a backupId argument. or > >>> things > >>> > > >>> like.. > >>> > > >>> > > > restore > >>> > > >>> > > > > is not clear if given an incremental id it will do the > >>> full > >>> > > >>> restore > >>> > > >>> > > from > >>> > > >>> > > > > full up to that point or if i need to apply manually > >>> > > everything). > >>> > > >>> > > > > > >>> > > >>> > > > > in conclusion, I think we can open a merge vote. I'll > be > >>> +1 > >>> > on > >>> > > >>> it, > >>> > > >>> > and > >>> > > >>> > > I > >>> > > >>> > > > > think we should try to reject -1 with just a "code > >>> cleanup" > >>> > > >>> > motivation, > >>> > > >>> > > > > since there will still be work going on on the code > >>> after the > >>> > > >>> merge. > >>> > > >>> > > > > > >>> > > >>> > > > > Matteo > >>> > > >>> > > > > > >>> > > >>> > > > > > >>> > > >>> > > > > On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das < > >>> > > >>> [email protected]> > >>> > > >>> > > > wrote: > >>> > > >>> > > > > > >>> > > >>> > > > > > Stack and others, anything else on the patch? Merge > to > >>> > master > >>> > > >>> now? > >>> > > >>> > > > > > > >>> > > >>> > > > > > >>> > > >>> > > > > >>> > > >>> > > > >>> > > >>> > > >>> > > >>> > >>> > > >> > >>> > > >> > >>> > > > > >>> > > > >>> > > >>> > >> > >> > > >
