Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Josh Elser]

On 9/11/17 11:52 PM, Stack wrote:

On Mon, Sep 11, 2017 at 11:07 AM, Vladimir Rodionov 
wrote:





My plan is to finish HBASE-17825 (further performance optimizations). This
will cut down number of MR jobs during incremental backup
from 2*N to 2  (N - number of tables). That will probably take 2-3 more
days


Unless you have this patch ready to go already, could I convince you to 
temporarily pause work on HBASE-17825 to take up the 3 points you've 
outlined immediately below?


We can treat the optimization as a one-off instead of lumping it in with 
the B feature at-large.



Then:

1. Address remaining two sub-tasks in HBASE-15227
2. Update Release notes for all relevant B JIRAs
3. Work on doc

After that we can call it feature full complete. Taking into account the
vast amount of efforts
spent on this feature (including QA testing) I would say that we are
probably quite close to GA right now, but only
after real testing is done (I do not anticipate significant issues, except
probably correct failure handling).

On a feature itself. We provide tools to fully automate backup and restore
tasks: create backup (full and incremental), restore
from image, delete backups, merge backups, history, history per table,
backup set management.

Hopefully, my write up addresses at least some of your concerns.



Thanks for updating us (community) w/ status. Completion of HA seems
important as is result of the scale testing.

As we're quickly approaching that beta-1 mark, I think it would be in 
our combined best-interest to knock out what we have identified as 
blockers. While these items aren't "significant" (read-as: not feature 
work), they are still risks to the beta "train". We should prioritize 
de-risking ourselves as much as possible.


After we finish that up (or in parallel), we can look at getting some 
scale testing done. Let me spin out another thread to discuss the 
concrete details on that one.


- Josh

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

ot* making assertions on my perception of quality) was removed
> > from the release line it was expected to land.
> >
> > > S
> > >
> > > On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <vladrodio...@gmail.com>
> > wrote:
> > >
> > >> >> Have I grasped the state of things correctly, Vlad?
> > >>
> > >> Josh, the only thing which is still pending is doc update. All other
> > >> features are good to have but not a blockers for 2.0 release.
> > >>
> > >> -Vlad
> > >>
> > >> On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <
> > vladrodio...@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > >> What testing and at what
> > >> > >> scale has testing been done?
> > >> >
> > >> > Do we have have that for other features?
> > >> >
> > >> >
> > >> > On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
> > >> vladrodio...@gmail.com
> > >> > > wrote:
> > >> >
> > >> >> >> It asks: "How do I figure what of backup/restore feature is
> going
> > to
> > >> >> be in
> > >> >> >>hbase-2.0.0?
> > >> >>
> > >> >> Hmm, wait for doc update.
> > >> >>
> > >> >>
> > >> >> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
> > >> >>
> > >> >>> HBASE-14414 is a JIRA with a list of random seeming issues w/
> > >> >>> non-descript
> > >> >>> summaries: "Add nonce support to TableBackupProcedure, BackupID
> must
> > >> >>> include backup set name, ...". The last comment in that issue is
> > from
> > >> >>> July.
> > >> >>> It asks: "How do I figure what of backup/restore feature is going
> > to be
> > >> >>> in
> > >> >>> hbase-2.0.0? Thanks Vladimir Rodionov
> > >> >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > name=vrodionov
> > >> >>> >."
> > >> >>> to which there is no answer.  Doc update is TODO.
> > >> >>>
> > >> >>> Where is the summary of the capability in hbase-2? What testing
> and
> > at
> > >> >>> what
> > >> >>> scale has testing been done? Is this 'stable or experimental'? If
> I
> > >> can't
> > >> >>> get basic info on this feature though I ask repeatedly, what hope
> > does
> > >> >>> the
> > >> >>> poor old operator have?
> > >> >>>
> > >> >>> St.Ack
> > >> >>>
> > >> >>>
> > >> >>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
> > >> >>> vladrodio...@gmail.com>
> > >> >>> wrote:
> > >> >>>
> > >> >>> > HBASE-14414
> > >> >>> >
> > >> >>> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
> > >> >>> >
> > >> >>> > > Where do I go to get the current status of this feature?
> > Looking in
> > >> >>> JIRA
> > >> >>> > I
> > >> >>> > > see loads of issues open against backup including some against
> > >> >>> > hbase-2.0.0
> > >> >>> > > and no progress being made that I can discern.
> > >> >>> > >
> > >> >>> > > Thanks,
> > >> >>> > > S
> > >> >>> > >
> > >> >>> > >
> > >> >>> > >
> > >> >>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net>
> > wrote:
> > >> >>> > >
> > >> >>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net>
> > wrote:
> > >> >>> > > >
> > >> >>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> > >> >>> > > >> vladrodio...@gmail.com> wrote:
> > >> >>> > > >>
> > >> >>&

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

gt; > >> What testing and at what
> >> > >> scale has testing been done?
> >> >
> >> > Do we have have that for other features?
> >> >
> >> >
> >> > On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
> >> vladrodio...@gmail.com
> >> > > wrote:
> >> >
> >> >> >> It asks: "How do I figure what of backup/restore feature is going
> to
> >> >> be in
> >> >> >>hbase-2.0.0?
> >> >>
> >> >> Hmm, wait for doc update.
> >> >>
> >> >>
> >> >> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
> >> >>
> >> >>> HBASE-14414 is a JIRA with a list of random seeming issues w/
> >> >>> non-descript
> >> >>> summaries: "Add nonce support to TableBackupProcedure, BackupID must
> >> >>> include backup set name, ...". The last comment in that issue is
> from
> >> >>> July.
> >> >>> It asks: "How do I figure what of backup/restore feature is going
> to be
> >> >>> in
> >> >>> hbase-2.0.0? Thanks Vladimir Rodionov
> >> >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> name=vrodionov
> >> >>> >."
> >> >>> to which there is no answer.  Doc update is TODO.
> >> >>>
> >> >>> Where is the summary of the capability in hbase-2? What testing and
> at
> >> >>> what
> >> >>> scale has testing been done? Is this 'stable or experimental'? If I
> >> can't
> >> >>> get basic info on this feature though I ask repeatedly, what hope
> does
> >> >>> the
> >> >>> poor old operator have?
> >> >>>
> >> >>> St.Ack
> >> >>>
> >> >>>
> >> >>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
> >> >>> vladrodio...@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>> > HBASE-14414
> >> >>> >
> >> >>> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
> >> >>> >
> >> >>> > > Where do I go to get the current status of this feature?
> Looking in
> >> >>> JIRA
> >> >>> > I
> >> >>> > > see loads of issues open against backup including some against
> >> >>> > hbase-2.0.0
> >> >>> > > and no progress being made that I can discern.
> >> >>> > >
> >> >>> > > Thanks,
> >> >>> > > S
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net>
> wrote:
> >> >>> > >
> >> >>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net>
> wrote:
> >> >>> > > >
> >> >>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> >> >>> > > >> vladrodio...@gmail.com> wrote:
> >> >>> > > >>
> >> >>> > > >>> >> and/or he answered most of the review feedback
> >> >>> > > >>>
> >> >>> > > >>> No, questions are still open, but I do not see any blockers
> and
> >> >>> we
> >> >>> > have
> >> >>> > > >>> HBASE-16940 to address these questions.
> >> >>> > > >>>
> >> >>> > > >>>
> >> >>> > > >> Agree. No blockers but stuff that should be dealt with (No
> one
> >> >>> will
> >> >>> > pay
> >> >>> > > >> me any attention once merge goes in -- smile).
> >> >>> > > >>
> >> >>> > > >>
> >> >>> > > > Let me clarify the above. I want review addressed before merge
> >> >>> happens.
> >> >>> > > > Sorry if any confusion.
> >> >>> > > > St.Ack
> >> >>> > > >
> >> >>> > > >
> &g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Josh Elser]

 > Where do I go to get the current status of this feature? Looking in
>> >>> JIRA
>> >>> > I
>> >>> > > see loads of issues open against backup including some against
>> >>> > hbase-2.0.0
>> >>> > > and no progress being made that I can discern.
>> >>> > >
>> >>> > > Thanks,
>> >>> > > S
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
>> >>> > >
>> >>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
>> >>> > > >
>> >>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>> >>> > > >> vladrodio...@gmail.com> wrote:
>> >>> > > >>
>> >>> > > >>> >> and/or he answered most of the review feedback
>> >>> > > >>>
>> >>> > > >>> No, questions are still open, but I do not see any blockers and
>> >>> we
>> >>> > have
>> >>> > > >>> HBASE-16940 to address these questions.
>> >>> > > >>>
>> >>> > > >>>
>> >>> > > >> Agree. No blockers but stuff that should be dealt with (No one
>> >>> will
>> >>> > pay
>> >>> > > >> me any attention once merge goes in -- smile).
>> >>> > > >>
>> >>> > > >>
>> >>> > > > Let me clarify the above. I want review addressed before merge
>> >>> happens.
>> >>> > > > Sorry if any confusion.
>> >>> > > > St.Ack
>> >>> > > >
>> >>> > > >
>> >>> > > >
>> >>> > > >
>> >>> > > >
>> >>> > > >
>> >>> > > >> St.Ack
>> >>> > > >>
>> >>> > > >>
>> >>> > > >>
>> >>> > > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <
>> >>> d...@hortonworks.com>
>> >>> > > >>> wrote:
>> >>> > > >>>
>> >>> > > >>> > Hi Stack, hats off to you for spending so much time on this!
>> >>> > Thanks!
>> >>> > > >>> From
>> >>> > > >>> > my understanding, Vlad has raised follow-up jiras for the
>> >>> issues
>> >>> > you
>> >>> > > >>> > raised, and/or he answered most of the review feedback. So,
>> do
>> >>> you
>> >>> > > >>> think we
>> >>> > > >>> > could do a merge vote now?
>> >>> > > >>> > Devaraj.
>> >>> > > >>> > 
>> >>> > > >>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
>> >>> > > >>> > Sent: Monday, November 21, 2016 8:34 PM
>> >>> > > >>> > To: dev@hbase.apache.org
>> >>> > > >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
>> >>> > HBASE-7912
>> >>> > > >>> >
>> >>> > > >>> > >> I have spent a good bit of time reviewing and testing this
>> >>> > > feature.
>> >>> > > >>> I
>> >>> > > >>> > would
>> >>> > > >>> > >> like my review and concerns addressed and I'd like it to
>> be
>> >>> > clear
>> >>> > > >>> how;
>> >>> > > >>> > >> either explicit follow-on issues, pointers to where in the
>> >>> patch
>> >>> > > or
>> >>> > > >>> doc
>> >>> > > >>> > my
>> >>> > > >>> > >> remarks have been catered to, etc. Until then, I am
>> against
>> >>> > > commit.
>> >>> > > >>> >
>> >&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

much as a year and more. I
> have
> > >> some
> > >>>> confidence these items basically work.  For backup/restore I have no
> > >> such
> > >>>> sense even after spending time in review and trying to use the
> > feature.
> > >>>>
> > >>>> As release manager, I have say over what makes it into a release.
> > >> Unless
> > >>>> the work is done to convince me that backup/restore is more than a
> > lump
> > >> of
> > >>>> code and a few unit tests that can pass on some fellows laptop, I am
> > >> going
> > >>>> to kick it out of branch-2.  Let the feature harden more in master
> > >> branch
> > >>>> before it ships in a release.
> > >>>>
> > >>>> S
> > >>>>
> > >>>> On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <
> vladrodio...@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>>>> Have I grasped the state of things correctly, Vlad?
> > >>>>>
> > >>>>> Josh, the only thing which is still pending is doc update. All
> other
> > >>>>> features are good to have but not a blockers for 2.0 release.
> > >>>>>
> > >>>>> -Vlad
> > >>>>>
> > >>>>> On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <
> > >>>> vladrodio...@gmail.com
> > >>>>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>>>> What testing and at what
> > >>>>>>>> scale has testing been done?
> > >>>>>>
> > >>>>>> Do we have have that for other features?
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
> > >>>>> vladrodio...@gmail.com
> > >>>>>>> wrote:
> > >>>>>>
> > >>>>>>>>> It asks: "How do I figure what of backup/restore feature is
> going
> > >>>> to
> > >>>>>>> be in
> > >>>>>>>>> hbase-2.0.0?
> > >>>>>>>
> > >>>>>>> Hmm, wait for doc update.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
> > >>>>>>>>
> > >>>>>>>> HBASE-14414 is a JIRA with a list of random seeming issues w/
> > >>>>>>>> non-descript
> > >>>>>>>> summaries: "Add nonce support to TableBackupProcedure, BackupID
> > must
> > >>>>>>>> include backup set name, ...". The last comment in that issue is
> > >> from
> > >>>>>>>> July.
> > >>>>>>>> It asks: "How do I figure what of backup/restore feature is
> going
> > to
> > >>>> be
> > >>>>>>>> in
> > >>>>>>>> hbase-2.0.0? Thanks Vladimir Rodionov
> > >>>>>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > >>>> name=vrodionov
> > >>>>>>>>> ."
> > >>>>>>>> to which there is no answer.  Doc update is TODO.
> > >>>>>>>>
> > >>>>>>>> Where is the summary of the capability in hbase-2? What testing
> > and
> > >>>> at
> > >>>>>>>> what
> > >>>>>>>> scale has testing been done? Is this 'stable or experimental'?
> If
> > I
> > >>>>> can't
> > >>>>>>>> get basic info on this feature though I ask repeatedly, what
> hope
> > >>>> does
> > >>>>>>>> the
> > >>>>>>>> poor old operator have?
> > >>>>>>>>
> > >>>>>>>> St.Ack
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
> > >>>>>>>> vladrodio...@gmail.com&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Andrew Purtell]

a release.
>>>>>> 
>>>>>> S
>>>>>> 
>>>>>> On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <vladrodio...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>>>> Have I grasped the state of things correctly, Vlad?
>>>>>>> 
>>>>>>> Josh, the only thing which is still pending is doc update. All other
>>>>>>> features are good to have but not a blockers for 2.0 release.
>>>>>>> 
>>>>>>> -Vlad
>>>>>>> 
>>>>>>> On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <
>>>>>> vladrodio...@gmail.com
>>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>>>> What testing and at what
>>>>>>>>>> scale has testing been done?
>>>>>>>> 
>>>>>>>> Do we have have that for other features?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
>>>>>>> vladrodio...@gmail.com
>>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>>>> It asks: "How do I figure what of backup/restore feature is going
>>>>>> to
>>>>>>>>> be in
>>>>>>>>>>> hbase-2.0.0?
>>>>>>>>> 
>>>>>>>>> Hmm, wait for doc update.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>> 
>>>>>>>>>> HBASE-14414 is a JIRA with a list of random seeming issues w/
>>>>>>>>>> non-descript
>>>>>>>>>> summaries: "Add nonce support to TableBackupProcedure, BackupID
>> must
>>>>>>>>>> include backup set name, ...". The last comment in that issue is
>>>> from
>>>>>>>>>> July.
>>>>>>>>>> It asks: "How do I figure what of backup/restore feature is going
>> to
>>>>>> be
>>>>>>>>>> in
>>>>>>>>>> hbase-2.0.0? Thanks Vladimir Rodionov
>>>>>>>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
>>>>>> name=vrodionov
>>>>>>>>>>> ."
>>>>>>>>>> to which there is no answer.  Doc update is TODO.
>>>>>>>>>> 
>>>>>>>>>> Where is the summary of the capability in hbase-2? What testing
>> and
>>>>>> at
>>>>>>>>>> what
>>>>>>>>>> scale has testing been done? Is this 'stable or experimental'? If
>> I
>>>>>>> can't
>>>>>>>>>> get basic info on this feature though I ask repeatedly, what hope
>>>>>> does
>>>>>>>>>> the
>>>>>>>>>> poor old operator have?
>>>>>>>>>> 
>>>>>>>>>> St.Ack
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
>>>>>>>>>> vladrodio...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> HBASE-14414
>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Where do I go to get the current status of this feature? Looking
>>>>>> in
>>>>>>>>>> JIRA
>>>>>>>>>>> I
>>>>>>>>>>>> see loads of issues open against backup including some against
>>>>>>>>>>> hbase-2.0.0
>>>>>>>>>>>> and no progress being made that I can discern.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

gt;>
> >>>>> wrote:
> >>>>>
> >>>>>>>> What testing and at what
> >>>>>>>> scale has testing been done?
> >>>>>>
> >>>>>> Do we have have that for other features?
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
> >>>>> vladrodio...@gmail.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>>>> It asks: "How do I figure what of backup/restore feature is going
> >>>> to
> >>>>>>> be in
> >>>>>>>>> hbase-2.0.0?
> >>>>>>>
> >>>>>>> Hmm, wait for doc update.
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
> >>>>>>>>
> >>>>>>>> HBASE-14414 is a JIRA with a list of random seeming issues w/
> >>>>>>>> non-descript
> >>>>>>>> summaries: "Add nonce support to TableBackupProcedure, BackupID
> must
> >>>>>>>> include backup set name, ...". The last comment in that issue is
> >> from
> >>>>>>>> July.
> >>>>>>>> It asks: "How do I figure what of backup/restore feature is going
> to
> >>>> be
> >>>>>>>> in
> >>>>>>>> hbase-2.0.0? Thanks Vladimir Rodionov
> >>>>>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> >>>> name=vrodionov
> >>>>>>>>> ."
> >>>>>>>> to which there is no answer.  Doc update is TODO.
> >>>>>>>>
> >>>>>>>> Where is the summary of the capability in hbase-2? What testing
> and
> >>>> at
> >>>>>>>> what
> >>>>>>>> scale has testing been done? Is this 'stable or experimental'? If
> I
> >>>>> can't
> >>>>>>>> get basic info on this feature though I ask repeatedly, what hope
> >>>> does
> >>>>>>>> the
> >>>>>>>> poor old operator have?
> >>>>>>>>
> >>>>>>>> St.Ack
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
> >>>>>>>> vladrodio...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> HBASE-14414
> >>>>>>>>>
> >>>>>>>>>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Where do I go to get the current status of this feature? Looking
> >>>> in
> >>>>>>>> JIRA
> >>>>>>>>> I
> >>>>>>>>>> see loads of issues open against backup including some against
> >>>>>>>>> hbase-2.0.0
> >>>>>>>>>> and no progress being made that I can discern.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> S
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net>
> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net>
> >>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> >>>>>>>>>>>> vladrodio...@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>> and/or he answered most of the review feedback
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> No, questions are still open, but I do not

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Andrew Purtell]

base-2.0.0?
>>>>>>> 
>>>>>>> Hmm, wait for doc update.
>>>>>>> 
>>>>>>> 
>>>>>>>> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
>>>>>>>> 
>>>>>>>> HBASE-14414 is a JIRA with a list of random seeming issues w/
>>>>>>>> non-descript
>>>>>>>> summaries: "Add nonce support to TableBackupProcedure, BackupID must
>>>>>>>> include backup set name, ...". The last comment in that issue is
>> from
>>>>>>>> July.
>>>>>>>> It asks: "How do I figure what of backup/restore feature is going to
>>>> be
>>>>>>>> in
>>>>>>>> hbase-2.0.0? Thanks Vladimir Rodionov
>>>>>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
>>>> name=vrodionov
>>>>>>>>> ."
>>>>>>>> to which there is no answer.  Doc update is TODO.
>>>>>>>> 
>>>>>>>> Where is the summary of the capability in hbase-2? What testing and
>>>> at
>>>>>>>> what
>>>>>>>> scale has testing been done? Is this 'stable or experimental'? If I
>>>>> can't
>>>>>>>> get basic info on this feature though I ask repeatedly, what hope
>>>> does
>>>>>>>> the
>>>>>>>> poor old operator have?
>>>>>>>> 
>>>>>>>> St.Ack
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
>>>>>>>> vladrodio...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> HBASE-14414
>>>>>>>>> 
>>>>>>>>>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
>>>>>>>>>> 
>>>>>>>>>> Where do I go to get the current status of this feature? Looking
>>>> in
>>>>>>>> JIRA
>>>>>>>>> I
>>>>>>>>>> see loads of issues open against backup including some against
>>>>>>>>> hbase-2.0.0
>>>>>>>>>> and no progress being made that I can discern.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> S
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>>>>>>>>>>>> vladrodio...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>>>> and/or he answered most of the review feedback
>>>>>>>>>>>>> 
>>>>>>>>>>>>> No, questions are still open, but I do not see any blockers
>>>> and
>>>>>>>> we
>>>>>>>>> have
>>>>>>>>>>>>> HBASE-16940 to address these questions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> Agree. No blockers but stuff that should be dealt with (No one
>>>>>>>> will
>>>>>>>>> pay
>>>>>>>>>>>> me any attention once merge goes in -- smile).
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> Let me clarify the above. I want review addressed before merge
>>>>>>>> happens.
>>>>>>>>>>> Sorry if any confusion.
>>>>>>>>>>> St.Ack
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

;>>> in
> >>>>>> hbase-2.0.0? Thanks Vladimir Rodionov
> >>>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> >> name=vrodionov
> >>>>>>> ."
> >>>>>> to which there is no answer.  Doc update is TODO.
> >>>>>>
> >>>>>> Where is the summary of the capability in hbase-2? What testing and
> >> at
> >>>>>> what
> >>>>>> scale has testing been done? Is this 'stable or experimental'? If I
> >>> can't
> >>>>>> get basic info on this feature though I ask repeatedly, what hope
> >> does
> >>>>>> the
> >>>>>> poor old operator have?
> >>>>>>
> >>>>>> St.Ack
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
> >>>>>> vladrodio...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> HBASE-14414
> >>>>>>>
> >>>>>>>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
> >>>>>>>>
> >>>>>>>> Where do I go to get the current status of this feature? Looking
> >> in
> >>>>>> JIRA
> >>>>>>> I
> >>>>>>>> see loads of issues open against backup including some against
> >>>>>>> hbase-2.0.0
> >>>>>>>> and no progress being made that I can discern.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> S
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
> >>>>>>>>>
> >>>>>>>>> On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>>> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> >>>>>>>>>> vladrodio...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>>> and/or he answered most of the review feedback
> >>>>>>>>>>>
> >>>>>>>>>>> No, questions are still open, but I do not see any blockers
> >> and
> >>>>>> we
> >>>>>>> have
> >>>>>>>>>>> HBASE-16940 to address these questions.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Agree. No blockers but stuff that should be dealt with (No one
> >>>>>> will
> >>>>>>> pay
> >>>>>>>>>> me any attention once merge goes in -- smile).
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> Let me clarify the above. I want review addressed before merge
> >>>>>> happens.
> >>>>>>>>> Sorry if any confusion.
> >>>>>>>>> St.Ack
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> St.Ack
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <
> >>>>>> d...@hortonworks.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Stack, hats off to you for spending so much time on
> >> this!
> >>>>>>> Thanks!
> >>>>>>>>>>> From
> >>>>>>>>>>>> my understanding, Vlad has raised follow-up jiras for the
> >>>>>> issues
> >>>>>>> you
> >>>>>>>>>>>> raised, and/or he answered most of the review feedback. S

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Andrew Purtell]

 St.Ack
>>>>>> 
>>>>>> 
>>>>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
>>>>>> vladrodio...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>>> HBASE-14414
>>>>>>> 
>>>>>>>> On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
>>>>>>>> 
>>>>>>>> Where do I go to get the current status of this feature? Looking
>> in
>>>>>> JIRA
>>>>>>> I
>>>>>>>> see loads of issues open against backup including some against
>>>>>>> hbase-2.0.0
>>>>>>>> and no progress being made that I can discern.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> S
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
>>>>>>>>> 
>>>>>>>>> On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net>
>> wrote:
>>>>>>>>> 
>>>>>>>>>> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>>>>>>>>>> vladrodio...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>>>> and/or he answered most of the review feedback
>>>>>>>>>>> 
>>>>>>>>>>> No, questions are still open, but I do not see any blockers
>> and
>>>>>> we
>>>>>>> have
>>>>>>>>>>> HBASE-16940 to address these questions.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Agree. No blockers but stuff that should be dealt with (No one
>>>>>> will
>>>>>>> pay
>>>>>>>>>> me any attention once merge goes in -- smile).
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> Let me clarify the above. I want review addressed before merge
>>>>>> happens.
>>>>>>>>> Sorry if any confusion.
>>>>>>>>> St.Ack
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> St.Ack
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <
>>>>>> d...@hortonworks.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Stack, hats off to you for spending so much time on
>> this!
>>>>>>> Thanks!
>>>>>>>>>>> From
>>>>>>>>>>>> my understanding, Vlad has raised follow-up jiras for the
>>>>>> issues
>>>>>>> you
>>>>>>>>>>>> raised, and/or he answered most of the review feedback. So,
>>> do
>>>>>> you
>>>>>>>>>>> think we
>>>>>>>>>>>> could do a merge vote now?
>>>>>>>>>>>> Devaraj.
>>>>>>>>>>>> 
>>>>>>>>>>>> From: Vladimir Rodionov <vladrodio...@gmail.com>
>>>>>>>>>>>> Sent: Monday, November 21, 2016 8:34 PM
>>>>>>>>>>>> To: dev@hbase.apache.org
>>>>>>>>>>>> Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
>>>>>>> HBASE-7912
>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have spent a good bit of time reviewing and testing
>> this
>>>>>>>> feature.
>>>>>>>>>>> I
>>>>>>>>>>>> would
>>>>>>>>>>>>>> like my review and concerns addressed and I'd like it to
>>> be
>>>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

> >
> > >>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> > >>> > > >> vladrodio...@gmail.com> wrote:
> > >>> > > >>
> > >>> > > >>> >> and/or he answered most of the review feedback
> > >>> > > >>>
> > >>> > > >>> No, questions are still open, but I do not see any blockers
> and
> > >>> we
> > >>> > have
> > >>> > > >>> HBASE-16940 to address these questions.
> > >>> > > >>>
> > >>> > > >>>
> > >>> > > >> Agree. No blockers but stuff that should be dealt with (No one
> > >>> will
> > >>> > pay
> > >>> > > >> me any attention once merge goes in -- smile).
> > >>> > > >>
> > >>> > > >>
> > >>> > > > Let me clarify the above. I want review addressed before merge
> > >>> happens.
> > >>> > > > Sorry if any confusion.
> > >>> > > > St.Ack
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > >> St.Ack
> > >>> > > >>
> > >>> > > >>
> > >>> > > >>
> > >>> > > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <
> > >>> d...@hortonworks.com>
> > >>> > > >>> wrote:
> > >>> > > >>>
> > >>> > > >>> > Hi Stack, hats off to you for spending so much time on
> this!
> > >>> > Thanks!
> > >>> > > >>> From
> > >>> > > >>> > my understanding, Vlad has raised follow-up jiras for the
> > >>> issues
> > >>> > you
> > >>> > > >>> > raised, and/or he answered most of the review feedback. So,
> > do
> > >>> you
> > >>> > > >>> think we
> > >>> > > >>> > could do a merge vote now?
> > >>> > > >>> > Devaraj.
> > >>> > > >>> > 
> > >>> > > >>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
> > >>> > > >>> > Sent: Monday, November 21, 2016 8:34 PM
> > >>> > > >>> > To: dev@hbase.apache.org
> > >>> > > >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
> > >>> > HBASE-7912
> > >>> > > >>> >
> > >>> > > >>> > >> I have spent a good bit of time reviewing and testing
> this
> > >>> > > feature.
> > >>> > > >>> I
> > >>> > > >>> > would
> > >>> > > >>> > >> like my review and concerns addressed and I'd like it to
> > be
> > >>> > clear
> > >>> > > >>> how;
> > >>> > > >>> > >> either explicit follow-on issues, pointers to where in
> the
> > >>> patch
> > >>> > > or
> > >>> > > >>> doc
> > >>> > > >>> > my
> > >>> > > >>> > >> remarks have been catered to, etc. Until then, I am
> > against
> > >>> > > commit.
> > >>> > > >>> >
> > >>> > > >>> > Stack, mega patch review comments will be addressed in the
> > >>> > dedicated
> > >>> > > >>> JIRA:
> > >>> > > >>> > HBASE-16940
> > >>> > > >>> > I have open several other JIRAs to address your other
> > comments
> > >>> (not
> > >>> > > on
> > >>> > > >>> > review board).
> > >>> > > >>> >
> > >>> > > >>> > Details are here (end of the thread):
> > >>> > > >>> > https://issues.apache.org/jira/browse/HBASE-14123
> > >>>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[stack]

In spite of repeated requests for eng summary of state of this feature --
summary of what is in 2.0, what is not, what the capabilities are, how well
it has been tested and at what scale -- all I get, when the requests are
not ignored, are pointers to lists of ill-describing jiras and some pending
user facing doc update.

For other features, mob or region server groups, I know that they have been
running at scale in production for as much as a year and more. I have some
confidence these items basically work.  For backup/restore I have no such
sense even after spending time in review and trying to use the feature.

As release manager, I have say over what makes it into a release.  Unless
the work is done to convince me that backup/restore is more than a lump of
code and a few unit tests that can pass on some fellows laptop, I am going
to kick it out of branch-2.  Let the feature harden more in master branch
before it ships in a release.

S

On Sep 8, 2017 10:59 PM, "Vladimir Rodionov" <vladrodio...@gmail.com> wrote:

> >> Have I grasped the state of things correctly, Vlad?
>
> Josh, the only thing which is still pending is doc update. All other
> features are good to have but not a blockers for 2.0 release.
>
> -Vlad
>
> On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <vladrodio...@gmail.com
> >
> wrote:
>
> > >> What testing and at what
> > >> scale has testing been done?
> >
> > Do we have have that for other features?
> >
> >
> > On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <
> vladrodio...@gmail.com
> > > wrote:
> >
> >> >> It asks: "How do I figure what of backup/restore feature is going to
> >> be in
> >> >>hbase-2.0.0?
> >>
> >> Hmm, wait for doc update.
> >>
> >>
> >> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
> >>
> >>> HBASE-14414 is a JIRA with a list of random seeming issues w/
> >>> non-descript
> >>> summaries: "Add nonce support to TableBackupProcedure, BackupID must
> >>> include backup set name, ...". The last comment in that issue is from
> >>> July.
> >>> It asks: "How do I figure what of backup/restore feature is going to be
> >>> in
> >>> hbase-2.0.0? Thanks Vladimir Rodionov
> >>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov
> >>> >."
> >>> to which there is no answer.  Doc update is TODO.
> >>>
> >>> Where is the summary of the capability in hbase-2? What testing and at
> >>> what
> >>> scale has testing been done? Is this 'stable or experimental'? If I
> can't
> >>> get basic info on this feature though I ask repeatedly, what hope does
> >>> the
> >>> poor old operator have?
> >>>
> >>> St.Ack
> >>>
> >>>
> >>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
> >>> vladrodio...@gmail.com>
> >>> wrote:
> >>>
> >>> > HBASE-14414
> >>> >
> >>> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
> >>> >
> >>> > > Where do I go to get the current status of this feature? Looking in
> >>> JIRA
> >>> > I
> >>> > > see loads of issues open against backup including some against
> >>> > hbase-2.0.0
> >>> > > and no progress being made that I can discern.
> >>> > >
> >>> > > Thanks,
> >>> > > S
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
> >>> > >
> >>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
> >>> > > >
> >>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> >>> > > >> vladrodio...@gmail.com> wrote:
> >>> > > >>
> >>> > > >>> >> and/or he answered most of the review feedback
> >>> > > >>>
> >>> > > >>> No, questions are still open, but I do not see any blockers and
> >>> we
> >>> > have
> >>> > > >>> HBASE-16940 to address these questions.
> >>> > > >>>
> >>> > > >>>
> >>> > > >> Agree. No blo

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> Have I grasped the state of things correctly, Vlad?

Josh, the only thing which is still pending is doc update. All other
features are good to have but not a blockers for 2.0 release.

-Vlad

On Fri, Sep 8, 2017 at 10:42 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> >> What testing and at what
> >> scale has testing been done?
>
> Do we have have that for other features?
>
>
> On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <vladrodio...@gmail.com
> > wrote:
>
>> >> It asks: "How do I figure what of backup/restore feature is going to
>> be in
>> >>hbase-2.0.0?
>>
>> Hmm, wait for doc update.
>>
>>
>> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
>>
>>> HBASE-14414 is a JIRA with a list of random seeming issues w/
>>> non-descript
>>> summaries: "Add nonce support to TableBackupProcedure, BackupID must
>>> include backup set name, ...". The last comment in that issue is from
>>> July.
>>> It asks: "How do I figure what of backup/restore feature is going to be
>>> in
>>> hbase-2.0.0? Thanks Vladimir Rodionov
>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov
>>> >."
>>> to which there is no answer.  Doc update is TODO.
>>>
>>> Where is the summary of the capability in hbase-2? What testing and at
>>> what
>>> scale has testing been done? Is this 'stable or experimental'? If I can't
>>> get basic info on this feature though I ask repeatedly, what hope does
>>> the
>>> poor old operator have?
>>>
>>> St.Ack
>>>
>>>
>>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <
>>> vladrodio...@gmail.com>
>>> wrote:
>>>
>>> > HBASE-14414
>>> >
>>> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
>>> >
>>> > > Where do I go to get the current status of this feature? Looking in
>>> JIRA
>>> > I
>>> > > see loads of issues open against backup including some against
>>> > hbase-2.0.0
>>> > > and no progress being made that I can discern.
>>> > >
>>> > > Thanks,
>>> > > S
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
>>> > >
>>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
>>> > > >
>>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>>> > > >> vladrodio...@gmail.com> wrote:
>>> > > >>
>>> > > >>> >> and/or he answered most of the review feedback
>>> > > >>>
>>> > > >>> No, questions are still open, but I do not see any blockers and
>>> we
>>> > have
>>> > > >>> HBASE-16940 to address these questions.
>>> > > >>>
>>> > > >>>
>>> > > >> Agree. No blockers but stuff that should be dealt with (No one
>>> will
>>> > pay
>>> > > >> me any attention once merge goes in -- smile).
>>> > > >>
>>> > > >>
>>> > > > Let me clarify the above. I want review addressed before merge
>>> happens.
>>> > > > Sorry if any confusion.
>>> > > > St.Ack
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >> St.Ack
>>> > > >>
>>> > > >>
>>> > > >>
>>> > > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <
>>> d...@hortonworks.com>
>>> > > >>> wrote:
>>> > > >>>
>>> > > >>> > Hi Stack, hats off to you for spending so much time on this!
>>> > Thanks!
>>> > > >>> From
>>> > > >>> > my understanding, Vlad has raised follow-up jiras for the
>>> issues
>>> > you
>>> > > >>> > raised, and/or he answered most of the review feedback. So, do
>>> you
>>> > > >>> think we
>>> > > >>> > could do a merge vote now?
&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> What testing and at what
>> scale has testing been done?

Do we have have that for other features?


On Fri, Sep 8, 2017 at 10:41 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> >> It asks: "How do I figure what of backup/restore feature is going to
> be in
> >>hbase-2.0.0?
>
> Hmm, wait for doc update.
>
>
> On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:
>
>> HBASE-14414 is a JIRA with a list of random seeming issues w/ non-descript
>> summaries: "Add nonce support to TableBackupProcedure, BackupID must
>> include backup set name, ...". The last comment in that issue is from
>> July.
>> It asks: "How do I figure what of backup/restore feature is going to be in
>> hbase-2.0.0? Thanks Vladimir Rodionov
>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov>."
>> to which there is no answer.  Doc update is TODO.
>>
>> Where is the summary of the capability in hbase-2? What testing and at
>> what
>> scale has testing been done? Is this 'stable or experimental'? If I can't
>> get basic info on this feature though I ask repeatedly, what hope does the
>> poor old operator have?
>>
>> St.Ack
>>
>>
>> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <vladrodio...@gmail.com
>> >
>> wrote:
>>
>> > HBASE-14414
>> >
>> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
>> >
>> > > Where do I go to get the current status of this feature? Looking in
>> JIRA
>> > I
>> > > see loads of issues open against backup including some against
>> > hbase-2.0.0
>> > > and no progress being made that I can discern.
>> > >
>> > > Thanks,
>> > > S
>> > >
>> > >
>> > >
>> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
>> > >
>> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
>> > > >
>> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>> > > >> vladrodio...@gmail.com> wrote:
>> > > >>
>> > > >>> >> and/or he answered most of the review feedback
>> > > >>>
>> > > >>> No, questions are still open, but I do not see any blockers and we
>> > have
>> > > >>> HBASE-16940 to address these questions.
>> > > >>>
>> > > >>>
>> > > >> Agree. No blockers but stuff that should be dealt with (No one will
>> > pay
>> > > >> me any attention once merge goes in -- smile).
>> > > >>
>> > > >>
>> > > > Let me clarify the above. I want review addressed before merge
>> happens.
>> > > > Sorry if any confusion.
>> > > > St.Ack
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >> St.Ack
>> > > >>
>> > > >>
>> > > >>
>> > > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <
>> d...@hortonworks.com>
>> > > >>> wrote:
>> > > >>>
>> > > >>> > Hi Stack, hats off to you for spending so much time on this!
>> > Thanks!
>> > > >>> From
>> > > >>> > my understanding, Vlad has raised follow-up jiras for the issues
>> > you
>> > > >>> > raised, and/or he answered most of the review feedback. So, do
>> you
>> > > >>> think we
>> > > >>> > could do a merge vote now?
>> > > >>> > Devaraj.
>> > > >>> > 
>> > > >>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
>> > > >>> > Sent: Monday, November 21, 2016 8:34 PM
>> > > >>> > To: dev@hbase.apache.org
>> > > >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
>> > HBASE-7912
>> > > >>> >
>> > > >>> > >> I have spent a good bit of time reviewing and testing this
>> > > feature.
>> > > >>> I
>> > > >>> > would
>> > > >>> > >> like my review and concerns addressed and 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> It asks: "How do I figure what of backup/restore feature is going to be
in
>>hbase-2.0.0?

Hmm, wait for doc update.


On Fri, Sep 8, 2017 at 2:39 PM, Stack <st...@duboce.net> wrote:

> HBASE-14414 is a JIRA with a list of random seeming issues w/ non-descript
> summaries: "Add nonce support to TableBackupProcedure, BackupID must
> include backup set name, ...". The last comment in that issue is from July.
> It asks: "How do I figure what of backup/restore feature is going to be in
> hbase-2.0.0? Thanks Vladimir Rodionov
> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov>."
> to which there is no answer.  Doc update is TODO.
>
> Where is the summary of the capability in hbase-2? What testing and at what
> scale has testing been done? Is this 'stable or experimental'? If I can't
> get basic info on this feature though I ask repeatedly, what hope does the
> poor old operator have?
>
> St.Ack
>
>
> On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <vladrodio...@gmail.com>
> wrote:
>
> > HBASE-14414
> >
> > On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
> >
> > > Where do I go to get the current status of this feature? Looking in
> JIRA
> > I
> > > see loads of issues open against backup including some against
> > hbase-2.0.0
> > > and no progress being made that I can discern.
> > >
> > > Thanks,
> > > S
> > >
> > >
> > >
> > > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> > > >> vladrodio...@gmail.com> wrote:
> > > >>
> > > >>> >> and/or he answered most of the review feedback
> > > >>>
> > > >>> No, questions are still open, but I do not see any blockers and we
> > have
> > > >>> HBASE-16940 to address these questions.
> > > >>>
> > > >>>
> > > >> Agree. No blockers but stuff that should be dealt with (No one will
> > pay
> > > >> me any attention once merge goes in -- smile).
> > > >>
> > > >>
> > > > Let me clarify the above. I want review addressed before merge
> happens.
> > > > Sorry if any confusion.
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com
> >
> > > >>> wrote:
> > > >>>
> > > >>> > Hi Stack, hats off to you for spending so much time on this!
> > Thanks!
> > > >>> From
> > > >>> > my understanding, Vlad has raised follow-up jiras for the issues
> > you
> > > >>> > raised, and/or he answered most of the review feedback. So, do
> you
> > > >>> think we
> > > >>> > could do a merge vote now?
> > > >>> > Devaraj.
> > > >>> > 
> > > >>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
> > > >>> > Sent: Monday, November 21, 2016 8:34 PM
> > > >>> > To: dev@hbase.apache.org
> > > >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
> > HBASE-7912
> > > >>> >
> > > >>> > >> I have spent a good bit of time reviewing and testing this
> > > feature.
> > > >>> I
> > > >>> > would
> > > >>> > >> like my review and concerns addressed and I'd like it to be
> > clear
> > > >>> how;
> > > >>> > >> either explicit follow-on issues, pointers to where in the
> patch
> > > or
> > > >>> doc
> > > >>> > my
> > > >>> > >> remarks have been catered to, etc. Until then, I am against
> > > commit.
> > > >>> >
> > > >>> > Stack, mega patch review comments will be addressed in the
> > dedicated
> > > >>> JIRA:
> > > >>> > HBASE-16940
> > > >>> &g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Josh Elser]

Based on the list of stuff on HBASE-14414 and offline-chats had with 
Vlad myself, I know of the following being needed


https://issues.apache.org/jira/browse/HBASE-15227 - Fault tolerance 
umbrella (no-op on its own)
https://issues.apache.org/jira/browse/HBASE-17852 - I believe Vlad is 
working on this one now
https://issues.apache.org/jira/browse/HBASE-16465 - disable 
splits/merges (marked as required for 15227 -- not sure if that's 
accurate anymore)

https://issues.apache.org/jira/browse/HBASE-17851 -- Just a unit test
https://issues.apache.org/jira/browse/HBASE-17133 - Docs

Echo'ing Stack, we're at the point where we need to draw a line in the 
sand for what will hit 2.0.0 to avoid mucking up testing.


Have I grasped the state of things correctly, Vlad?

On 9/8/17 5:39 PM, Stack wrote:

HBASE-14414 is a JIRA with a list of random seeming issues w/ non-descript
summaries: "Add nonce support to TableBackupProcedure, BackupID must
include backup set name, ...". The last comment in that issue is from July.
It asks: "How do I figure what of backup/restore feature is going to be in
hbase-2.0.0? Thanks Vladimir Rodionov
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov>."
to which there is no answer.  Doc update is TODO.

Where is the summary of the capability in hbase-2? What testing and at what
scale has testing been done? Is this 'stable or experimental'? If I can't
get basic info on this feature though I ask repeatedly, what hope does the
poor old operator have?

St.Ack


On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:


HBASE-14414

On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:


Where do I go to get the current status of this feature? Looking in JIRA

I

see loads of issues open against backup including some against

hbase-2.0.0

and no progress being made that I can discern.

Thanks,
S



On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:


On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:


On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
vladrodio...@gmail.com> wrote:


and/or he answered most of the review feedback


No, questions are still open, but I do not see any blockers and we

have

HBASE-16940 to address these questions.



Agree. No blockers but stuff that should be dealt with (No one will

pay

me any attention once merge goes in -- smile).



Let me clarify the above. I want review addressed before merge happens.
Sorry if any confusion.
St.Ack







St.Ack




On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com>
wrote:


Hi Stack, hats off to you for spending so much time on this!

Thanks!

From

my understanding, Vlad has raised follow-up jiras for the issues

you

raised, and/or he answered most of the review feedback. So, do you

think we

could do a merge vote now?
Devaraj.

From: Vladimir Rodionov <vladrodio...@gmail.com>
Sent: Monday, November 21, 2016 8:34 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch

HBASE-7912



I have spent a good bit of time reviewing and testing this

feature.

I

would

like my review and concerns addressed and I'd like it to be

clear

how;

either explicit follow-on issues, pointers to where in the patch

or

doc

my

remarks have been catered to, etc. Until then, I am against

commit.


Stack, mega patch review comments will be addressed in the

dedicated

JIRA:

HBASE-16940
I have open several other JIRAs to address your other comments (not

on

review board).

Details are here (end of the thread):
https://issues.apache.org/jira/browse/HBASE-14123

Let me know what else should we do to move merge forward.

-Vlad


On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:


On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com>

wrote:



Thanks, Matteo.

bq. restore is not clear if given an incremental id it will do

the

full

restore from full up to that point or if i need to apply

manually

everything

The restore takes into consideration of the dependent

backup(s).

So there is no need to apply preceding backup(s) manually.



I ask this question on the issue. It is not clear from the usage

or

doc

how

to run a restore from incremental. Can you fix in doc and usage

how

so I

can be clear and try it. Currently I am stuck verifying a round

trip

backup

restore made of incrementals.

Thanks,
S




On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi <

theo.berto...@gmail.com>

wrote:


I did one last pass to the mega patch. I don't see anything

major

that

should block the merge.

- most of the code is isolated in the backup package
- all the backup code is client side
- there are few changes to the server side, mainly for

cleaners,

wal

rolling and similar (which is ok)
- there is a good number of tests, and an integration test

the code s

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

HBASE-14414 is a JIRA with a list of random seeming issues w/ non-descript
summaries: "Add nonce support to TableBackupProcedure, BackupID must
include backup set name, ...". The last comment in that issue is from July.
It asks: "How do I figure what of backup/restore feature is going to be in
hbase-2.0.0? Thanks Vladimir Rodionov
<https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vrodionov>."
to which there is no answer.  Doc update is TODO.

Where is the summary of the capability in hbase-2? What testing and at what
scale has testing been done? Is this 'stable or experimental'? If I can't
get basic info on this feature though I ask repeatedly, what hope does the
poor old operator have?

St.Ack


On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> HBASE-14414
>
> On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
>
> > Where do I go to get the current status of this feature? Looking in JIRA
> I
> > see loads of issues open against backup including some against
> hbase-2.0.0
> > and no progress being made that I can discern.
> >
> > Thanks,
> > S
> >
> >
> >
> > On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> > >> vladrodio...@gmail.com> wrote:
> > >>
> > >>> >> and/or he answered most of the review feedback
> > >>>
> > >>> No, questions are still open, but I do not see any blockers and we
> have
> > >>> HBASE-16940 to address these questions.
> > >>>
> > >>>
> > >> Agree. No blockers but stuff that should be dealt with (No one will
> pay
> > >> me any attention once merge goes in -- smile).
> > >>
> > >>
> > > Let me clarify the above. I want review addressed before merge happens.
> > > Sorry if any confusion.
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > >
> > >> St.Ack
> > >>
> > >>
> > >>
> > >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com>
> > >>> wrote:
> > >>>
> > >>> > Hi Stack, hats off to you for spending so much time on this!
> Thanks!
> > >>> From
> > >>> > my understanding, Vlad has raised follow-up jiras for the issues
> you
> > >>> > raised, and/or he answered most of the review feedback. So, do you
> > >>> think we
> > >>> > could do a merge vote now?
> > >>> > Devaraj.
> > >>> > 
> > >>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
> > >>> > Sent: Monday, November 21, 2016 8:34 PM
> > >>> > To: dev@hbase.apache.org
> > >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
> HBASE-7912
> > >>> >
> > >>> > >> I have spent a good bit of time reviewing and testing this
> > feature.
> > >>> I
> > >>> > would
> > >>> > >> like my review and concerns addressed and I'd like it to be
> clear
> > >>> how;
> > >>> > >> either explicit follow-on issues, pointers to where in the patch
> > or
> > >>> doc
> > >>> > my
> > >>> > >> remarks have been catered to, etc. Until then, I am against
> > commit.
> > >>> >
> > >>> > Stack, mega patch review comments will be addressed in the
> dedicated
> > >>> JIRA:
> > >>> > HBASE-16940
> > >>> > I have open several other JIRAs to address your other comments (not
> > on
> > >>> > review board).
> > >>> >
> > >>> > Details are here (end of the thread):
> > >>> > https://issues.apache.org/jira/browse/HBASE-14123
> > >>> >
> > >>> > Let me know what else should we do to move merge forward.
> > >>> >
> > >>> > -Vlad
> > >>> >
> > >>> >
> > >>> > On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:
> > >>> >
> > >>> > > On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

All blockers have been resolved. Some remaining JIRAs are good to have but
are not a blockers

On Fri, Sep 8, 2017 at 1:59 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> HBASE-14414
>
> On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:
>
>> Where do I go to get the current status of this feature? Looking in JIRA I
>> see loads of issues open against backup including some against hbase-2.0.0
>> and no progress being made that I can discern.
>>
>> Thanks,
>> S
>>
>>
>>
>> On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
>>
>> > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>> >> vladrodio...@gmail.com> wrote:
>> >>
>> >>> >> and/or he answered most of the review feedback
>> >>>
>> >>> No, questions are still open, but I do not see any blockers and we
>> have
>> >>> HBASE-16940 to address these questions.
>> >>>
>> >>>
>> >> Agree. No blockers but stuff that should be dealt with (No one will pay
>> >> me any attention once merge goes in -- smile).
>> >>
>> >>
>> > Let me clarify the above. I want review addressed before merge happens.
>> > Sorry if any confusion.
>> > St.Ack
>> >
>> >
>> >
>> >
>> >
>> >
>> >> St.Ack
>> >>
>> >>
>> >>
>> >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com>
>> >>> wrote:
>> >>>
>> >>> > Hi Stack, hats off to you for spending so much time on this! Thanks!
>> >>> From
>> >>> > my understanding, Vlad has raised follow-up jiras for the issues you
>> >>> > raised, and/or he answered most of the review feedback. So, do you
>> >>> think we
>> >>> > could do a merge vote now?
>> >>> > Devaraj.
>> >>> > 
>> >>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
>> >>> > Sent: Monday, November 21, 2016 8:34 PM
>> >>> > To: dev@hbase.apache.org
>> >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
>> >>> >
>> >>> > >> I have spent a good bit of time reviewing and testing this
>> feature.
>> >>> I
>> >>> > would
>> >>> > >> like my review and concerns addressed and I'd like it to be clear
>> >>> how;
>> >>> > >> either explicit follow-on issues, pointers to where in the patch
>> or
>> >>> doc
>> >>> > my
>> >>> > >> remarks have been catered to, etc. Until then, I am against
>> commit.
>> >>> >
>> >>> > Stack, mega patch review comments will be addressed in the dedicated
>> >>> JIRA:
>> >>> > HBASE-16940
>> >>> > I have open several other JIRAs to address your other comments (not
>> on
>> >>> > review board).
>> >>> >
>> >>> > Details are here (end of the thread):
>> >>> > https://issues.apache.org/jira/browse/HBASE-14123
>> >>> >
>> >>> > Let me know what else should we do to move merge forward.
>> >>> >
>> >>> > -Vlad
>> >>> >
>> >>> >
>> >>> > On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:
>> >>> >
>> >>> > > On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com>
>> wrote:
>> >>> > >
>> >>> > > > Thanks, Matteo.
>> >>> > > >
>> >>> > > > bq. restore is not clear if given an incremental id it will do
>> the
>> >>> full
>> >>> > > > restore from full up to that point or if i need to apply
>> manually
>> >>> > > > everything
>> >>> > > >
>> >>> > > > The restore takes into consideration of the dependent backup(s).
>> >>> > > > So there is no need to apply preceding backup(s) manually.
>> >>> > > >
>> >>> > > >
>&

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

HBASE-14414

On Fri, Sep 8, 2017 at 1:14 PM, Stack <st...@duboce.net> wrote:

> Where do I go to get the current status of this feature? Looking in JIRA I
> see loads of issues open against backup including some against hbase-2.0.0
> and no progress being made that I can discern.
>
> Thanks,
> S
>
>
>
> On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:
>
> > On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
> >> vladrodio...@gmail.com> wrote:
> >>
> >>> >> and/or he answered most of the review feedback
> >>>
> >>> No, questions are still open, but I do not see any blockers and we have
> >>> HBASE-16940 to address these questions.
> >>>
> >>>
> >> Agree. No blockers but stuff that should be dealt with (No one will pay
> >> me any attention once merge goes in -- smile).
> >>
> >>
> > Let me clarify the above. I want review addressed before merge happens.
> > Sorry if any confusion.
> > St.Ack
> >
> >
> >
> >
> >
> >
> >> St.Ack
> >>
> >>
> >>
> >>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com>
> >>> wrote:
> >>>
> >>> > Hi Stack, hats off to you for spending so much time on this! Thanks!
> >>> From
> >>> > my understanding, Vlad has raised follow-up jiras for the issues you
> >>> > raised, and/or he answered most of the review feedback. So, do you
> >>> think we
> >>> > could do a merge vote now?
> >>> > Devaraj.
> >>> > 
> >>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
> >>> > Sent: Monday, November 21, 2016 8:34 PM
> >>> > To: dev@hbase.apache.org
> >>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
> >>> >
> >>> > >> I have spent a good bit of time reviewing and testing this
> feature.
> >>> I
> >>> > would
> >>> > >> like my review and concerns addressed and I'd like it to be clear
> >>> how;
> >>> > >> either explicit follow-on issues, pointers to where in the patch
> or
> >>> doc
> >>> > my
> >>> > >> remarks have been catered to, etc. Until then, I am against
> commit.
> >>> >
> >>> > Stack, mega patch review comments will be addressed in the dedicated
> >>> JIRA:
> >>> > HBASE-16940
> >>> > I have open several other JIRAs to address your other comments (not
> on
> >>> > review board).
> >>> >
> >>> > Details are here (end of the thread):
> >>> > https://issues.apache.org/jira/browse/HBASE-14123
> >>> >
> >>> > Let me know what else should we do to move merge forward.
> >>> >
> >>> > -Vlad
> >>> >
> >>> >
> >>> > On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:
> >>> >
> >>> > > On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> >>> > >
> >>> > > > Thanks, Matteo.
> >>> > > >
> >>> > > > bq. restore is not clear if given an incremental id it will do
> the
> >>> full
> >>> > > > restore from full up to that point or if i need to apply manually
> >>> > > > everything
> >>> > > >
> >>> > > > The restore takes into consideration of the dependent backup(s).
> >>> > > > So there is no need to apply preceding backup(s) manually.
> >>> > > >
> >>> > > >
> >>> > > I ask this question on the issue. It is not clear from the usage or
> >>> doc
> >>> > how
> >>> > > to run a restore from incremental. Can you fix in doc and usage how
> >>> so I
> >>> > > can be clear and try it. Currently I am stuck verifying a round
> trip
> >>> > backup
> >>> > > restore made of incrementals.
> >>> > >
> >>> > > Thanks,
> >>> > > S
> >>> > >
> >>> > >
> >>> > >
> >>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

Where do I go to get the current status of this feature? Looking in JIRA I
see loads of issues open against backup including some against hbase-2.0.0
and no progress being made that I can discern.

Thanks,
S



On Wed, Nov 23, 2016 at 8:52 AM, Stack <st...@duboce.net> wrote:

> On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:
>
>> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <
>> vladrodio...@gmail.com> wrote:
>>
>>> >> and/or he answered most of the review feedback
>>>
>>> No, questions are still open, but I do not see any blockers and we have
>>> HBASE-16940 to address these questions.
>>>
>>>
>> Agree. No blockers but stuff that should be dealt with (No one will pay
>> me any attention once merge goes in -- smile).
>>
>>
> Let me clarify the above. I want review addressed before merge happens.
> Sorry if any confusion.
> St.Ack
>
>
>
>
>
>
>> St.Ack
>>
>>
>>
>>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com>
>>> wrote:
>>>
>>> > Hi Stack, hats off to you for spending so much time on this! Thanks!
>>> From
>>> > my understanding, Vlad has raised follow-up jiras for the issues you
>>> > raised, and/or he answered most of the review feedback. So, do you
>>> think we
>>> > could do a merge vote now?
>>> > Devaraj.
>>> > 
>>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
>>> > Sent: Monday, November 21, 2016 8:34 PM
>>> > To: dev@hbase.apache.org
>>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
>>> >
>>> > >> I have spent a good bit of time reviewing and testing this feature.
>>> I
>>> > would
>>> > >> like my review and concerns addressed and I'd like it to be clear
>>> how;
>>> > >> either explicit follow-on issues, pointers to where in the patch or
>>> doc
>>> > my
>>> > >> remarks have been catered to, etc. Until then, I am against commit.
>>> >
>>> > Stack, mega patch review comments will be addressed in the dedicated
>>> JIRA:
>>> > HBASE-16940
>>> > I have open several other JIRAs to address your other comments (not on
>>> > review board).
>>> >
>>> > Details are here (end of the thread):
>>> > https://issues.apache.org/jira/browse/HBASE-14123
>>> >
>>> > Let me know what else should we do to move merge forward.
>>> >
>>> > -Vlad
>>> >
>>> >
>>> > On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:
>>> >
>>> > > On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>>> > >
>>> > > > Thanks, Matteo.
>>> > > >
>>> > > > bq. restore is not clear if given an incremental id it will do the
>>> full
>>> > > > restore from full up to that point or if i need to apply manually
>>> > > > everything
>>> > > >
>>> > > > The restore takes into consideration of the dependent backup(s).
>>> > > > So there is no need to apply preceding backup(s) manually.
>>> > > >
>>> > > >
>>> > > I ask this question on the issue. It is not clear from the usage or
>>> doc
>>> > how
>>> > > to run a restore from incremental. Can you fix in doc and usage how
>>> so I
>>> > > can be clear and try it. Currently I am stuck verifying a round trip
>>> > backup
>>> > > restore made of incrementals.
>>> > >
>>> > > Thanks,
>>> > > S
>>> > >
>>> > >
>>> > >
>>> > > > On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi <
>>> > > theo.berto...@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > I did one last pass to the mega patch. I don't see anything major
>>> > that
>>> > > > > should block the merge.
>>> > > > >
>>> > > > > - most of the code is isolated in the backup package
>>> > > > > - all the backup code is client side
>>> > > > > - there are few changes to the server side, mainly for cleaners,

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

On Tue, Nov 22, 2016 at 6:48 PM, Stack <st...@duboce.net> wrote:

> On Tue, Nov 22, 2016 at 3:17 PM, Vladimir Rodionov <vladrodio...@gmail.com
> > wrote:
>
>> >> and/or he answered most of the review feedback
>>
>> No, questions are still open, but I do not see any blockers and we have
>> HBASE-16940 to address these questions.
>>
>>
> Agree. No blockers but stuff that should be dealt with (No one will pay me
> any attention once merge goes in -- smile).
>
>
Let me clarify the above. I want review addressed before merge happens.
Sorry if any confusion.
St.Ack






> St.Ack
>
>
>
>> On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com>
>> wrote:
>>
>> > Hi Stack, hats off to you for spending so much time on this! Thanks!
>> From
>> > my understanding, Vlad has raised follow-up jiras for the issues you
>> > raised, and/or he answered most of the review feedback. So, do you
>> think we
>> > could do a merge vote now?
>> > Devaraj.
>> > ____________________
>> > From: Vladimir Rodionov <vladrodio...@gmail.com>
>> > Sent: Monday, November 21, 2016 8:34 PM
>> > To: dev@hbase.apache.org
>> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
>> >
>> > >> I have spent a good bit of time reviewing and testing this feature. I
>> > would
>> > >> like my review and concerns addressed and I'd like it to be clear
>> how;
>> > >> either explicit follow-on issues, pointers to where in the patch or
>> doc
>> > my
>> > >> remarks have been catered to, etc. Until then, I am against commit.
>> >
>> > Stack, mega patch review comments will be addressed in the dedicated
>> JIRA:
>> > HBASE-16940
>> > I have open several other JIRAs to address your other comments (not on
>> > review board).
>> >
>> > Details are here (end of the thread):
>> > https://issues.apache.org/jira/browse/HBASE-14123
>> >
>> > Let me know what else should we do to move merge forward.
>> >
>> > -Vlad
>> >
>> >
>> > On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:
>> >
>> > > On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > >
>> > > > Thanks, Matteo.
>> > > >
>> > > > bq. restore is not clear if given an incremental id it will do the
>> full
>> > > > restore from full up to that point or if i need to apply manually
>> > > > everything
>> > > >
>> > > > The restore takes into consideration of the dependent backup(s).
>> > > > So there is no need to apply preceding backup(s) manually.
>> > > >
>> > > >
>> > > I ask this question on the issue. It is not clear from the usage or
>> doc
>> > how
>> > > to run a restore from incremental. Can you fix in doc and usage how
>> so I
>> > > can be clear and try it. Currently I am stuck verifying a round trip
>> > backup
>> > > restore made of incrementals.
>> > >
>> > > Thanks,
>> > > S
>> > >
>> > >
>> > >
>> > > > On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi <
>> > > theo.berto...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > I did one last pass to the mega patch. I don't see anything major
>> > that
>> > > > > should block the merge.
>> > > > >
>> > > > > - most of the code is isolated in the backup package
>> > > > > - all the backup code is client side
>> > > > > - there are few changes to the server side, mainly for cleaners,
>> wal
>> > > > > rolling and similar (which is ok)
>> > > > > - there is a good number of tests, and an integration test
>> > > > >
>> > > > > the code seems to have still some left overs from the old
>> > > implementation,
>> > > > > and some stuff needs a cleanup. but I don't think this should be
>> used
>> > > as
>> > > > an
>> > > > > argument to block the merge. I think the guys will keep working on
>> > this
>> > > > and
>> > > > > they may also get help of others once the patch is in master.
>> > > &

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> and/or he answered most of the review feedback

No, questions are still open, but I do not see any blockers and we have
HBASE-16940 to address these questions.

On Tue, Nov 22, 2016 at 3:04 PM, Devaraj Das <d...@hortonworks.com> wrote:

> Hi Stack, hats off to you for spending so much time on this! Thanks! From
> my understanding, Vlad has raised follow-up jiras for the issues you
> raised, and/or he answered most of the review feedback. So, do you think we
> could do a merge vote now?
> Devaraj.
> 
> From: Vladimir Rodionov <vladrodio...@gmail.com>
> Sent: Monday, November 21, 2016 8:34 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
>
> >> I have spent a good bit of time reviewing and testing this feature. I
> would
> >> like my review and concerns addressed and I'd like it to be clear how;
> >> either explicit follow-on issues, pointers to where in the patch or doc
> my
> >> remarks have been catered to, etc. Until then, I am against commit.
>
> Stack, mega patch review comments will be addressed in the dedicated JIRA:
> HBASE-16940
> I have open several other JIRAs to address your other comments (not on
> review board).
>
> Details are here (end of the thread):
> https://issues.apache.org/jira/browse/HBASE-14123
>
> Let me know what else should we do to move merge forward.
>
> -Vlad
>
>
> On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:
>
> > On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > Thanks, Matteo.
> > >
> > > bq. restore is not clear if given an incremental id it will do the full
> > > restore from full up to that point or if i need to apply manually
> > > everything
> > >
> > > The restore takes into consideration of the dependent backup(s).
> > > So there is no need to apply preceding backup(s) manually.
> > >
> > >
> > I ask this question on the issue. It is not clear from the usage or doc
> how
> > to run a restore from incremental. Can you fix in doc and usage how so I
> > can be clear and try it. Currently I am stuck verifying a round trip
> backup
> > restore made of incrementals.
> >
> > Thanks,
> > S
> >
> >
> >
> > > On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi <
> > theo.berto...@gmail.com>
> > > wrote:
> > >
> > > > I did one last pass to the mega patch. I don't see anything major
> that
> > > > should block the merge.
> > > >
> > > > - most of the code is isolated in the backup package
> > > > - all the backup code is client side
> > > > - there are few changes to the server side, mainly for cleaners, wal
> > > > rolling and similar (which is ok)
> > > > - there is a good number of tests, and an integration test
> > > >
> > > > the code seems to have still some left overs from the old
> > implementation,
> > > > and some stuff needs a cleanup. but I don't think this should be used
> > as
> > > an
> > > > argument to block the merge. I think the guys will keep working on
> this
> > > and
> > > > they may also get help of others once the patch is in master.
> > > >
> > > > I still have my concerns about the current limitations, but these are
> > > > things already planned for phase 3, so some of this stuff may even be
> > in
> > > > the final 2.0.
> > > > but as long as we have a "current limitations" section in the user
> > guide
> > > > mentioning important stuff like the ones below, I'm ok with it.
> > > >  - if you write to the table with Durability.SKIP_WALS your data will
> > not
> > > > be in the incremental-backup
> > > >  - if you bulkload files that data will not be in the incremental
> > backup
> > > > (HBASE-14417)
> > > >  - the incremental backup will not only contains the data of the
> table
> > > you
> > > > specified but also the regions from other tables that are on the same
> > set
> > > > of RSs (HBASE-14141) ...maybe a note about security around this topic
> > > >  - the incremental backup will not contains just the "latest row"
> > between
> > > > backup A and B, but it will also contains all the updates occurred in
> > > > between. but the restore does not allow you to restore up to a
> certain
> > > &

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Devaraj Das]

Hi Stack, hats off to you for spending so much time on this! Thanks! From my 
understanding, Vlad has raised follow-up jiras for the issues you raised, 
and/or he answered most of the review feedback. So, do you think we could do a 
merge vote now?
Devaraj.

From: Vladimir Rodionov <vladrodio...@gmail.com>
Sent: Monday, November 21, 2016 8:34 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

>> I have spent a good bit of time reviewing and testing this feature. I
would
>> like my review and concerns addressed and I'd like it to be clear how;
>> either explicit follow-on issues, pointers to where in the patch or doc
my
>> remarks have been catered to, etc. Until then, I am against commit.

Stack, mega patch review comments will be addressed in the dedicated JIRA:
HBASE-16940
I have open several other JIRAs to address your other comments (not on
review board).

Details are here (end of the thread):
https://issues.apache.org/jira/browse/HBASE-14123

Let me know what else should we do to move merge forward.

-Vlad


On Fri, Nov 18, 2016 at 4:54 PM, Stack <st...@duboce.net> wrote:

> On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Thanks, Matteo.
> >
> > bq. restore is not clear if given an incremental id it will do the full
> > restore from full up to that point or if i need to apply manually
> > everything
> >
> > The restore takes into consideration of the dependent backup(s).
> > So there is no need to apply preceding backup(s) manually.
> >
> >
> I ask this question on the issue. It is not clear from the usage or doc how
> to run a restore from incremental. Can you fix in doc and usage how so I
> can be clear and try it. Currently I am stuck verifying a round trip backup
> restore made of incrementals.
>
> Thanks,
> S
>
>
>
> > On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi <
> theo.berto...@gmail.com>
> > wrote:
> >
> > > I did one last pass to the mega patch. I don't see anything major that
> > > should block the merge.
> > >
> > > - most of the code is isolated in the backup package
> > > - all the backup code is client side
> > > - there are few changes to the server side, mainly for cleaners, wal
> > > rolling and similar (which is ok)
> > > - there is a good number of tests, and an integration test
> > >
> > > the code seems to have still some left overs from the old
> implementation,
> > > and some stuff needs a cleanup. but I don't think this should be used
> as
> > an
> > > argument to block the merge. I think the guys will keep working on this
> > and
> > > they may also get help of others once the patch is in master.
> > >
> > > I still have my concerns about the current limitations, but these are
> > > things already planned for phase 3, so some of this stuff may even be
> in
> > > the final 2.0.
> > > but as long as we have a "current limitations" section in the user
> guide
> > > mentioning important stuff like the ones below, I'm ok with it.
> > >  - if you write to the table with Durability.SKIP_WALS your data will
> not
> > > be in the incremental-backup
> > >  - if you bulkload files that data will not be in the incremental
> backup
> > > (HBASE-14417)
> > >  - the incremental backup will not only contains the data of the table
> > you
> > > specified but also the regions from other tables that are on the same
> set
> > > of RSs (HBASE-14141) ...maybe a note about security around this topic
> > >  - the incremental backup will not contains just the "latest row"
> between
> > > backup A and B, but it will also contains all the updates occurred in
> > > between. but the restore does not allow you to restore up to a certain
> > > point in time, the restore will always be up to the "latest backup
> > point".
> > >  - you should limit the number of "incremental" up to N (or maybe
> SIZE),
> > to
> > > avoid replay time becoming the bottleneck. (HBASE-14135)
> > >
> > > I'll be ok even with the above not being in the final 2.0,
> > > but i'd like to see as blocker for the final 2.0 (not the merge)
> > >  - the backup code moved in an hbase-backup module
> > >  - and some more work around tools, especially to try to unify and make
> > > simple the backup experience (simple example: in some case there is a
> > > backup_id argument in others a backupId argument. or things like..
> > restore
> > > is not clear if given an incremental id it will do the full restore
> from
> > > full up to that point or if i need to apply manually everything).
> > >
> > > in conclusion, I think we can open a merge vote. I'll be +1 on it, and
> I
> > > think we should try to reject -1 with just a "code cleanup" motivation,
> > > since there will still be work going on on the code after the merge.
> > >
> > > Matteo
> > >
> > >
> > > On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das <d...@hortonworks.com>
> > wrote:
> > >
> > > > Stack and others, anything else on the patch? Merge to master now?
> > > >
> > >
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> I have spent a good bit of time reviewing and testing this feature. I
would
>> like my review and concerns addressed and I'd like it to be clear how;
>> either explicit follow-on issues, pointers to where in the patch or doc
my
>> remarks have been catered to, etc. Until then, I am against commit.

Stack, mega patch review comments will be addressed in the dedicated JIRA:
HBASE-16940
I have open several other JIRAs to address your other comments (not on
review board).

Details are here (end of the thread):
https://issues.apache.org/jira/browse/HBASE-14123

Let me know what else should we do to move merge forward.

-Vlad


On Fri, Nov 18, 2016 at 4:54 PM, Stack  wrote:

> On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu  wrote:
>
> > Thanks, Matteo.
> >
> > bq. restore is not clear if given an incremental id it will do the full
> > restore from full up to that point or if i need to apply manually
> > everything
> >
> > The restore takes into consideration of the dependent backup(s).
> > So there is no need to apply preceding backup(s) manually.
> >
> >
> I ask this question on the issue. It is not clear from the usage or doc how
> to run a restore from incremental. Can you fix in doc and usage how so I
> can be clear and try it. Currently I am stuck verifying a round trip backup
> restore made of incrementals.
>
> Thanks,
> S
>
>
>
> > On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi <
> theo.berto...@gmail.com>
> > wrote:
> >
> > > I did one last pass to the mega patch. I don't see anything major that
> > > should block the merge.
> > >
> > > - most of the code is isolated in the backup package
> > > - all the backup code is client side
> > > - there are few changes to the server side, mainly for cleaners, wal
> > > rolling and similar (which is ok)
> > > - there is a good number of tests, and an integration test
> > >
> > > the code seems to have still some left overs from the old
> implementation,
> > > and some stuff needs a cleanup. but I don't think this should be used
> as
> > an
> > > argument to block the merge. I think the guys will keep working on this
> > and
> > > they may also get help of others once the patch is in master.
> > >
> > > I still have my concerns about the current limitations, but these are
> > > things already planned for phase 3, so some of this stuff may even be
> in
> > > the final 2.0.
> > > but as long as we have a "current limitations" section in the user
> guide
> > > mentioning important stuff like the ones below, I'm ok with it.
> > >  - if you write to the table with Durability.SKIP_WALS your data will
> not
> > > be in the incremental-backup
> > >  - if you bulkload files that data will not be in the incremental
> backup
> > > (HBASE-14417)
> > >  - the incremental backup will not only contains the data of the table
> > you
> > > specified but also the regions from other tables that are on the same
> set
> > > of RSs (HBASE-14141) ...maybe a note about security around this topic
> > >  - the incremental backup will not contains just the "latest row"
> between
> > > backup A and B, but it will also contains all the updates occurred in
> > > between. but the restore does not allow you to restore up to a certain
> > > point in time, the restore will always be up to the "latest backup
> > point".
> > >  - you should limit the number of "incremental" up to N (or maybe
> SIZE),
> > to
> > > avoid replay time becoming the bottleneck. (HBASE-14135)
> > >
> > > I'll be ok even with the above not being in the final 2.0,
> > > but i'd like to see as blocker for the final 2.0 (not the merge)
> > >  - the backup code moved in an hbase-backup module
> > >  - and some more work around tools, especially to try to unify and make
> > > simple the backup experience (simple example: in some case there is a
> > > backup_id argument in others a backupId argument. or things like..
> > restore
> > > is not clear if given an incremental id it will do the full restore
> from
> > > full up to that point or if i need to apply manually everything).
> > >
> > > in conclusion, I think we can open a merge vote. I'll be +1 on it, and
> I
> > > think we should try to reject -1 with just a "code cleanup" motivation,
> > > since there will still be work going on on the code after the merge.
> > >
> > > Matteo
> > >
> > >
> > > On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das 
> > wrote:
> > >
> > > > Stack and others, anything else on the patch? Merge to master now?
> > > >
> > >
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu  wrote:

> Thanks, Matteo.
>
> bq. restore is not clear if given an incremental id it will do the full
> restore from full up to that point or if i need to apply manually
> everything
>
> The restore takes into consideration of the dependent backup(s).
> So there is no need to apply preceding backup(s) manually.
>
>
I ask this question on the issue. It is not clear from the usage or doc how
to run a restore from incremental. Can you fix in doc and usage how so I
can be clear and try it. Currently I am stuck verifying a round trip backup
restore made of incrementals.

Thanks,
S



> On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi 
> wrote:
>
> > I did one last pass to the mega patch. I don't see anything major that
> > should block the merge.
> >
> > - most of the code is isolated in the backup package
> > - all the backup code is client side
> > - there are few changes to the server side, mainly for cleaners, wal
> > rolling and similar (which is ok)
> > - there is a good number of tests, and an integration test
> >
> > the code seems to have still some left overs from the old implementation,
> > and some stuff needs a cleanup. but I don't think this should be used as
> an
> > argument to block the merge. I think the guys will keep working on this
> and
> > they may also get help of others once the patch is in master.
> >
> > I still have my concerns about the current limitations, but these are
> > things already planned for phase 3, so some of this stuff may even be in
> > the final 2.0.
> > but as long as we have a "current limitations" section in the user guide
> > mentioning important stuff like the ones below, I'm ok with it.
> >  - if you write to the table with Durability.SKIP_WALS your data will not
> > be in the incremental-backup
> >  - if you bulkload files that data will not be in the incremental backup
> > (HBASE-14417)
> >  - the incremental backup will not only contains the data of the table
> you
> > specified but also the regions from other tables that are on the same set
> > of RSs (HBASE-14141) ...maybe a note about security around this topic
> >  - the incremental backup will not contains just the "latest row" between
> > backup A and B, but it will also contains all the updates occurred in
> > between. but the restore does not allow you to restore up to a certain
> > point in time, the restore will always be up to the "latest backup
> point".
> >  - you should limit the number of "incremental" up to N (or maybe SIZE),
> to
> > avoid replay time becoming the bottleneck. (HBASE-14135)
> >
> > I'll be ok even with the above not being in the final 2.0,
> > but i'd like to see as blocker for the final 2.0 (not the merge)
> >  - the backup code moved in an hbase-backup module
> >  - and some more work around tools, especially to try to unify and make
> > simple the backup experience (simple example: in some case there is a
> > backup_id argument in others a backupId argument. or things like..
> restore
> > is not clear if given an incremental id it will do the full restore from
> > full up to that point or if i need to apply manually everything).
> >
> > in conclusion, I think we can open a merge vote. I'll be +1 on it, and I
> > think we should try to reject -1 with just a "code cleanup" motivation,
> > since there will still be work going on on the code after the merge.
> >
> > Matteo
> >
> >
> > On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das 
> wrote:
> >
> > > Stack and others, anything else on the patch? Merge to master now?
> > >
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

Nice writeup Matteo. I'm trying to get to where you are at but am not there
yet. Almost. Agree that it should be a blocker that all gets hoisted out of
core into a backup module and that the limitations are spelled out clearly
in doc.

I have spent a good bit of time reviewing and testing this feature. I would
like my review and concerns addressed and I'd like it to be clear how;
either explicit follow-on issues, pointers to where in the patch or doc my
remarks have been catered to, etc. Until then, I am against commit.

St.Ack


On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi 
wrote:

> I did one last pass to the mega patch. I don't see anything major that
> should block the merge.
>
> - most of the code is isolated in the backup package
> - all the backup code is client side
> - there are few changes to the server side, mainly for cleaners, wal
> rolling and similar (which is ok)
> - there is a good number of tests, and an integration test
>
> the code seems to have still some left overs from the old implementation,
> and some stuff needs a cleanup. but I don't think this should be used as an
> argument to block the merge. I think the guys will keep working on this and
> they may also get help of others once the patch is in master.
>
> I still have my concerns about the current limitations, but these are
> things already planned for phase 3, so some of this stuff may even be in
> the final 2.0.
> but as long as we have a "current limitations" section in the user guide
> mentioning important stuff like the ones below, I'm ok with it.
>  - if you write to the table with Durability.SKIP_WALS your data will not
> be in the incremental-backup
>  - if you bulkload files that data will not be in the incremental backup
> (HBASE-14417)
>  - the incremental backup will not only contains the data of the table you
> specified but also the regions from other tables that are on the same set
> of RSs (HBASE-14141) ...maybe a note about security around this topic
>  - the incremental backup will not contains just the "latest row" between
> backup A and B, but it will also contains all the updates occurred in
> between. but the restore does not allow you to restore up to a certain
> point in time, the restore will always be up to the "latest backup point".
>  - you should limit the number of "incremental" up to N (or maybe SIZE), to
> avoid replay time becoming the bottleneck. (HBASE-14135)
>
> I'll be ok even with the above not being in the final 2.0,
> but i'd like to see as blocker for the final 2.0 (not the merge)
>  - the backup code moved in an hbase-backup module
>  - and some more work around tools, especially to try to unify and make
> simple the backup experience (simple example: in some case there is a
> backup_id argument in others a backupId argument. or things like.. restore
> is not clear if given an incremental id it will do the full restore from
> full up to that point or if i need to apply manually everything).
>
> in conclusion, I think we can open a merge vote. I'll be +1 on it, and I
> think we should try to reject -1 with just a "code cleanup" motivation,
> since there will still be work going on on the code after the merge.
>
> Matteo
>
>
> On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das  wrote:
>
> > Stack and others, anything else on the patch? Merge to master now?
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

Thanks, Matteo

It is a good write up.

HBASE-14417 (bulk load support) is pretty close to code complete now. There
is already work in progress on
HBASE-14141 (Filtering WAL on backup). I would say they will be complete in
a 2-3 weeks.

and I agree that "code clean up before merge" requests should be ignored.
This is work in progress and we are doing clean up all the time.

-Vlad

On Fri, Nov 18, 2016 at 3:53 PM, Ted Yu  wrote:

> Thanks, Matteo.
>
> bq. restore is not clear if given an incremental id it will do the full
> restore from full up to that point or if i need to apply manually
> everything
>
> The restore takes into consideration of the dependent backup(s).
> So there is no need to apply preceding backup(s) manually.
>
> On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi 
> wrote:
>
> > I did one last pass to the mega patch. I don't see anything major that
> > should block the merge.
> >
> > - most of the code is isolated in the backup package
> > - all the backup code is client side
> > - there are few changes to the server side, mainly for cleaners, wal
> > rolling and similar (which is ok)
> > - there is a good number of tests, and an integration test
> >
> > the code seems to have still some left overs from the old implementation,
> > and some stuff needs a cleanup. but I don't think this should be used as
> an
> > argument to block the merge. I think the guys will keep working on this
> and
> > they may also get help of others once the patch is in master.
> >
> > I still have my concerns about the current limitations, but these are
> > things already planned for phase 3, so some of this stuff may even be in
> > the final 2.0.
> > but as long as we have a "current limitations" section in the user guide
> > mentioning important stuff like the ones below, I'm ok with it.
> >  - if you write to the table with Durability.SKIP_WALS your data will not
> > be in the incremental-backup
> >  - if you bulkload files that data will not be in the incremental backup
> > (HBASE-14417)
> >  - the incremental backup will not only contains the data of the table
> you
> > specified but also the regions from other tables that are on the same set
> > of RSs (HBASE-14141) ...maybe a note about security around this topic
> >  - the incremental backup will not contains just the "latest row" between
> > backup A and B, but it will also contains all the updates occurred in
> > between. but the restore does not allow you to restore up to a certain
> > point in time, the restore will always be up to the "latest backup
> point".
> >  - you should limit the number of "incremental" up to N (or maybe SIZE),
> to
> > avoid replay time becoming the bottleneck. (HBASE-14135)
> >
> > I'll be ok even with the above not being in the final 2.0,
> > but i'd like to see as blocker for the final 2.0 (not the merge)
> >  - the backup code moved in an hbase-backup module
> >  - and some more work around tools, especially to try to unify and make
> > simple the backup experience (simple example: in some case there is a
> > backup_id argument in others a backupId argument. or things like..
> restore
> > is not clear if given an incremental id it will do the full restore from
> > full up to that point or if i need to apply manually everything).
> >
> > in conclusion, I think we can open a merge vote. I'll be +1 on it, and I
> > think we should try to reject -1 with just a "code cleanup" motivation,
> > since there will still be work going on on the code after the merge.
> >
> > Matteo
> >
> >
> > On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das 
> wrote:
> >
> > > Stack and others, anything else on the patch? Merge to master now?
> > >
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Thanks, Matteo.

bq. restore is not clear if given an incremental id it will do the full
restore from full up to that point or if i need to apply manually everything

The restore takes into consideration of the dependent backup(s).
So there is no need to apply preceding backup(s) manually.

On Fri, Nov 18, 2016 at 3:48 PM, Matteo Bertozzi 
wrote:

> I did one last pass to the mega patch. I don't see anything major that
> should block the merge.
>
> - most of the code is isolated in the backup package
> - all the backup code is client side
> - there are few changes to the server side, mainly for cleaners, wal
> rolling and similar (which is ok)
> - there is a good number of tests, and an integration test
>
> the code seems to have still some left overs from the old implementation,
> and some stuff needs a cleanup. but I don't think this should be used as an
> argument to block the merge. I think the guys will keep working on this and
> they may also get help of others once the patch is in master.
>
> I still have my concerns about the current limitations, but these are
> things already planned for phase 3, so some of this stuff may even be in
> the final 2.0.
> but as long as we have a "current limitations" section in the user guide
> mentioning important stuff like the ones below, I'm ok with it.
>  - if you write to the table with Durability.SKIP_WALS your data will not
> be in the incremental-backup
>  - if you bulkload files that data will not be in the incremental backup
> (HBASE-14417)
>  - the incremental backup will not only contains the data of the table you
> specified but also the regions from other tables that are on the same set
> of RSs (HBASE-14141) ...maybe a note about security around this topic
>  - the incremental backup will not contains just the "latest row" between
> backup A and B, but it will also contains all the updates occurred in
> between. but the restore does not allow you to restore up to a certain
> point in time, the restore will always be up to the "latest backup point".
>  - you should limit the number of "incremental" up to N (or maybe SIZE), to
> avoid replay time becoming the bottleneck. (HBASE-14135)
>
> I'll be ok even with the above not being in the final 2.0,
> but i'd like to see as blocker for the final 2.0 (not the merge)
>  - the backup code moved in an hbase-backup module
>  - and some more work around tools, especially to try to unify and make
> simple the backup experience (simple example: in some case there is a
> backup_id argument in others a backupId argument. or things like.. restore
> is not clear if given an incremental id it will do the full restore from
> full up to that point or if i need to apply manually everything).
>
> in conclusion, I think we can open a merge vote. I'll be +1 on it, and I
> think we should try to reject -1 with just a "code cleanup" motivation,
> since there will still be work going on on the code after the merge.
>
> Matteo
>
>
> On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das  wrote:
>
> > Stack and others, anything else on the patch? Merge to master now?
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Matteo Bertozzi]

I did one last pass to the mega patch. I don't see anything major that
should block the merge.

- most of the code is isolated in the backup package
- all the backup code is client side
- there are few changes to the server side, mainly for cleaners, wal
rolling and similar (which is ok)
- there is a good number of tests, and an integration test

the code seems to have still some left overs from the old implementation,
and some stuff needs a cleanup. but I don't think this should be used as an
argument to block the merge. I think the guys will keep working on this and
they may also get help of others once the patch is in master.

I still have my concerns about the current limitations, but these are
things already planned for phase 3, so some of this stuff may even be in
the final 2.0.
but as long as we have a "current limitations" section in the user guide
mentioning important stuff like the ones below, I'm ok with it.
 - if you write to the table with Durability.SKIP_WALS your data will not
be in the incremental-backup
 - if you bulkload files that data will not be in the incremental backup
(HBASE-14417)
 - the incremental backup will not only contains the data of the table you
specified but also the regions from other tables that are on the same set
of RSs (HBASE-14141) ...maybe a note about security around this topic
 - the incremental backup will not contains just the "latest row" between
backup A and B, but it will also contains all the updates occurred in
between. but the restore does not allow you to restore up to a certain
point in time, the restore will always be up to the "latest backup point".
 - you should limit the number of "incremental" up to N (or maybe SIZE), to
avoid replay time becoming the bottleneck. (HBASE-14135)

I'll be ok even with the above not being in the final 2.0,
but i'd like to see as blocker for the final 2.0 (not the merge)
 - the backup code moved in an hbase-backup module
 - and some more work around tools, especially to try to unify and make
simple the backup experience (simple example: in some case there is a
backup_id argument in others a backupId argument. or things like.. restore
is not clear if given an incremental id it will do the full restore from
full up to that point or if i need to apply manually everything).

in conclusion, I think we can open a merge vote. I'll be +1 on it, and I
think we should try to reject -1 with just a "code cleanup" motivation,
since there will still be work going on on the code after the merge.

Matteo


On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das  wrote:

> Stack and others, anything else on the patch? Merge to master now?
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

New version of patch was pushed end of last week. I was going to give it a
review.  Takes a couple of hours.  Sorry for delay.
S

On Sun, Nov 6, 2016 at 10:54 PM, Devaraj Das <d...@hortonworks.com> wrote:

> Stack and others, anything else on the patch? Merge to master now?
> 
> From: Ted Yu <yuzhih...@gmail.com>
> Sent: Thursday, October 13, 2016 3:56 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
>
> It is not rebased - but all backup / restore functionality is there.
>
> You can use mega patch v29 over HBASE-14123 which applies to master branch.
>
> Cheers
>
> On Thu, Oct 13, 2016 at 3:51 PM, Apekshit Sharma <a...@cloudera.com>
> wrote:
>
> > Is HBASE-7912 the feature branch?
> > If yes, has it not been rebased to incorporate latest master changes yet?
> >
> > ~/apache/hbase  (master) → g ck apache/HBASE-7912
> > warning: refname 'apache/HBASE-7912' is ambiguous.
> > Switched to branch 'apache/HBASE-7912'
> > ~/apache/hbase  (apache/HBASE-7912) → g log HEAD..apache/master | grep
> > "^commit" | wc -l
> >  917
> >
> >
> > On Thu, Oct 13, 2016 at 7:01 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > Once Stack's comments are addressed, we can change the master build to
> > > using Maven 3.3.3
> > >
> > > I will log JIRA for modifying refguide on the upgrade.
> > >
> > > Cheers
> > >
> > > On Thu, Oct 13, 2016 at 6:42 AM, Sean Busbey <bus...@cloudera.com>
> > wrote:
> > >
> > > > Are we requiring maven 3.3.z now? historically our required maven
> > > > version has been 3.0.5. I don't have an objection to changing it, per
> > > > se, but I'd like to make sure our docs and CI builds get updated
> > > > properly.
> > > >
> > > > On Wed, Oct 12, 2016 at 4:03 PM, Stack <st...@duboce.net> wrote:
> > > > > On Wed, Oct 12, 2016 at 1:58 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > >> Fails again when I do this:
> > > > >>
> > > > >> $ mvn clean install -DskipTests assembly:single
> > > > >> 
> > > > >>
> > > > >>
> > > > > Ok. Works if I use mvn 3.3.x.
> > > > > St.Ack
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >>
> > > > >> [INFO] --
> --
> > > > >> 
> > > > >> [INFO] Building Apache HBase - Server 2.0.0-SNAPSHOT
> > > > >> [INFO] --
> --
> > > > >> 
> > > > >> Downloading: file:/home/stack/hbase.git/
> hbase-server/src/main/site/
> > > > >> resources/repo/org/apache/hadoop/hadoop-distcp/2.7.1/
> > > > >> hadoop-distcp-2.7.1.pom
> > > > >> Exception in thread "pool-1-thread-1"
> --
> > > > >> -
> > > > >> constituent[0]: file:/usr/share/maven/lib/aether-spi.jar
> > > > >> constituent[1]: file:/usr/share/maven/lib/
> > > > maven-settings-builder-3.x.jar
> > > > >> constituent[2]: file:/usr/share/maven/lib/maven-artifact-3.x.jar
> > > > >> constituent[3]: file:/usr/share/maven/lib/commons-httpclient.jar
> > > > >> constituent[4]: file:/usr/share/maven/lib/wagon-provider-api.jar
> > > > >> constituent[5]: file:/usr/share/maven/lib/maven-core-3.x.jar
> > > > >> constituent[6]: file:/usr/share/maven/lib/aether-impl.jar
> > > > >> constituent[7]: file:/usr/share/maven/lib/sisu-inject-bean.jar
> > > > >> constituent[8]: file:/usr/share/maven/lib/sisu-inject-plexus.jar
> > > > >> constituent[9]: file:/usr/share/maven/lib/wagon-file.jar
> > > > >> constituent[10]: file:/usr/share/maven/lib/guava.jar
> > > > >> constituent[11]: file:/usr/share/maven/lib/maven-compat-3.x.jar
> > > > >> constituent[12]: file:/usr/share/maven/lib/maven-model-3.x.jar
> > > > >> constituent[13]: file:/usr/share/maven/lib/
> plexus-interpolation.jar
> > > > >> constituent[14]: file:/usr/share/maven/lib/commons-codec.jar
> > > > >> constituent[15]: file:/usr/s

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Devaraj Das]

Stack and others, anything else on the patch? Merge to master now?

From: Ted Yu <yuzhih...@gmail.com>
Sent: Thursday, October 13, 2016 3:56 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

It is not rebased - but all backup / restore functionality is there.

You can use mega patch v29 over HBASE-14123 which applies to master branch.

Cheers

On Thu, Oct 13, 2016 at 3:51 PM, Apekshit Sharma <a...@cloudera.com> wrote:

> Is HBASE-7912 the feature branch?
> If yes, has it not been rebased to incorporate latest master changes yet?
>
> ~/apache/hbase  (master) → g ck apache/HBASE-7912
> warning: refname 'apache/HBASE-7912' is ambiguous.
> Switched to branch 'apache/HBASE-7912'
> ~/apache/hbase  (apache/HBASE-7912) → g log HEAD..apache/master | grep
> "^commit" | wc -l
>  917
>
>
> On Thu, Oct 13, 2016 at 7:01 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Once Stack's comments are addressed, we can change the master build to
> > using Maven 3.3.3
> >
> > I will log JIRA for modifying refguide on the upgrade.
> >
> > Cheers
> >
> > On Thu, Oct 13, 2016 at 6:42 AM, Sean Busbey <bus...@cloudera.com>
> wrote:
> >
> > > Are we requiring maven 3.3.z now? historically our required maven
> > > version has been 3.0.5. I don't have an objection to changing it, per
> > > se, but I'd like to make sure our docs and CI builds get updated
> > > properly.
> > >
> > > On Wed, Oct 12, 2016 at 4:03 PM, Stack <st...@duboce.net> wrote:
> > > > On Wed, Oct 12, 2016 at 1:58 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > >> Fails again when I do this:
> > > >>
> > > >> $ mvn clean install -DskipTests assembly:single
> > > >> 
> > > >>
> > > >>
> > > > Ok. Works if I use mvn 3.3.x.
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >>
> > > >> [INFO] 
> > > >> 
> > > >> [INFO] Building Apache HBase - Server 2.0.0-SNAPSHOT
> > > >> [INFO] 
> > > >> 
> > > >> Downloading: file:/home/stack/hbase.git/hbase-server/src/main/site/
> > > >> resources/repo/org/apache/hadoop/hadoop-distcp/2.7.1/
> > > >> hadoop-distcp-2.7.1.pom
> > > >> Exception in thread "pool-1-thread-1" --
> > > >> -
> > > >> constituent[0]: file:/usr/share/maven/lib/aether-spi.jar
> > > >> constituent[1]: file:/usr/share/maven/lib/
> > > maven-settings-builder-3.x.jar
> > > >> constituent[2]: file:/usr/share/maven/lib/maven-artifact-3.x.jar
> > > >> constituent[3]: file:/usr/share/maven/lib/commons-httpclient.jar
> > > >> constituent[4]: file:/usr/share/maven/lib/wagon-provider-api.jar
> > > >> constituent[5]: file:/usr/share/maven/lib/maven-core-3.x.jar
> > > >> constituent[6]: file:/usr/share/maven/lib/aether-impl.jar
> > > >> constituent[7]: file:/usr/share/maven/lib/sisu-inject-bean.jar
> > > >> constituent[8]: file:/usr/share/maven/lib/sisu-inject-plexus.jar
> > > >> constituent[9]: file:/usr/share/maven/lib/wagon-file.jar
> > > >> constituent[10]: file:/usr/share/maven/lib/guava.jar
> > > >> constituent[11]: file:/usr/share/maven/lib/maven-compat-3.x.jar
> > > >> constituent[12]: file:/usr/share/maven/lib/maven-model-3.x.jar
> > > >> constituent[13]: file:/usr/share/maven/lib/plexus-interpolation.jar
> > > >> constituent[14]: file:/usr/share/maven/lib/commons-codec.jar
> > > >> constituent[15]: file:/usr/share/maven/lib/aether-util.jar
> > > >> constituent[16]: file:/usr/share/maven/lib/aether-api.jar
> > > >> constituent[17]: file:/usr/share/maven/lib/
> > maven-model-builder-3.x.jar
> > > >> constituent[18]: file:/usr/share/maven/lib/
> > > maven-repository-metadata-3.x.
> > > >> jar
> > > >> constituent[19]: file:/usr/share/maven/lib/maven-settings-3.x.jar
> > > >> constituent[20]: file:/usr/share/maven/lib/
> > > plexus-component-annotations.
> > > >> jar
> > > >> constituent[21]: file:/usr/share/maven/lib/commons

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

2016 at 3:52 PM, Vladimir Rodionov
> <
> > > >>>> >>> >> > > > vladrodio...@gmail.com
> > > >>>> >>> >> > > > > >
> > > >>>> >>> >> > > > > wrote:
> > > >>>> >>> >> > > > >
> > > >>>> >>> >> > > > > > >> How hard to put in an hbase-backup module?
> > > >>>> hbase-server
> > > >>>> >>> is
> > > >>>> >>> >> fat
> > > >>>> >>> >> > > > enough
> > > >>>> >>> >> > > > > > >> already. Could be done as a follow-up.
> > > >>>> >>> >> > > > > >
> > > >>>> >>> >> > > > > > https://issues.apache.org/
> jira/browse/HBASE-16727?
> > > >>>> >>> >> > > > > > focusedCommentId=15531237&
> page=com.atlassian.jira.
> > > >>>> >>> >> > > > > > plugin.system.issuetabpanels:c
> > > >>>> >>> omment-tabpanel#comment-15531237
> > > >>>> >>> >> > > > > >
> > > >>>> >>> >> > > > > > Can we do merge first? Then we can discuss
> separate
> > > >>>> module.
> > > >>>> >>> >> > > > > >
> > > >>>> >>> >> > > > > > -Vlad
> > > >>>> >>> >> > > > > >
> > > >>>> >>> >> > > > > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu <
> > > >>>> >>> yuzhih...@gmail.com>
> > > >>>> >>> >> > wrote:
> > > >>>> >>> >> > > > > >
> > > >>>> >>> >> > > > > >> Looks like the first quote was cut off.
> > > >>>> >>> >> > > > > >> The original sentence was:
> > > >>>> >>> >> > > > > >>
> > > >>>> >>> >> > > > > >> bq. no mapreduce job launched from master or
> > region
> > > >>>> server.
> > > >>>> >>> >> > > > > >>
> > > >>>> >>> >> > > > > >> mapreduce job is launched from the node where
> > > command
> > > >>>> line
> > > >>>> >>> >> tool is
> > > >>>> >>> >> > > > run.
> > > >>>> >>> >> > > > > >>
> > > >>>> >>> >> > > > > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack <
> > > >>>> st...@duboce.net>
> > > >>>> >>> >> wrote:
> > > >>>> >>> >> > > > > >>
> > > >>>> >>> >> > > > > >> > bq. launched from master or region server.
> > > >>>> >>> >> > > > > >> >
> > > >>>> >>> >> > > > > >> > What does this mean please? Has to be run from
> > > >>>> Master or
> > > >>>> >>> >> > > > RegionServer?
> > > >>>> >>> >> > > > > >> Can
> > > >>>> >>> >> > > > > >> > it be run from another node altogether?
> > > >>>> >>> >> > > > > >> >
> > > >>>> >>> >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir
> > > Rodionov <
> > > >>>> >>> >> > > > > >> vladrodio...@gmail.com
> > > >>>> >>> >> > > > > >> > >
> > > >>>> >>> >> > > > > >> > wrote:
> > > >>>> >>> >> > > > > >> >
> > > >>>> >>> >> > > > > >> > > >> mapreduce dependency has been moved to
> > client
> > > >&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Apekshit Sharma]

de where
> > command
> > >>>> line
> > >>>> >>> >> tool is
> > >>>> >>> >> > > > run.
> > >>>> >>> >> > > > > >>
> > >>>> >>> >> > > > > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack <
> > >>>> st...@duboce.net>
> > >>>> >>> >> wrote:
> > >>>> >>> >> > > > > >>
> > >>>> >>> >> > > > > >> > bq. launched from master or region server.
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> > What does this mean please? Has to be run from
> > >>>> Master or
> > >>>> >>> >> > > > RegionServer?
> > >>>> >>> >> > > > > >> Can
> > >>>> >>> >> > > > > >> > it be run from another node altogether?
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir
> > Rodionov <
> > >>>> >>> >> > > > > >> vladrodio...@gmail.com
> > >>>> >>> >> > > > > >> > >
> > >>>> >>> >> > > > > >> > wrote:
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> > > >> mapreduce dependency has been moved to
> client
> > >>>> side
> > >>>> >>> - no
> > >>>> >>> >> > > > mapreduce
> > >>>> >>> >> > > > > >> job
> > >>>> >>> >> > > > > >> > >
> > >>>> >>> >> > > > > >> > > 1. We have no code in the client module
> anymore,
> > >>>> due to
> > >>>> >>> >> > > dependency
> > >>>> >>> >> > > > > on
> > >>>> >>> >> > > > > >> > > internal server API (HFile and WAL access).
> > >>>> >>> >> > > > > >> > > 2. Backup/ restore are client - driven
> > operations,
> > >>>> but
> > >>>> >>> all
> > >>>> >>> >> the
> > >>>> >>> >> > > > code
> > >>>> >>> >> > > > > >> > resides
> > >>>> >>> >> > > > > >> > > in the server module
> > >>>> >>> >> > > > > >> > >
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> > How hard to put in an hbase-backup module?
> > >>>> hbase-server
> > >>>> >>> is
> > >>>> >>> >> fat
> > >>>> >>> >> > > > enough
> > >>>> >>> >> > > > > >> > already. Could be done as a follow-up.
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> > Thanks,
> > >>>> >>> >> > > > > >> > St.Ack
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> >
> > >>>> >>> >> > > > > >> > > 3. No MR in Master, no procedure - driven
> > >>>> execution.
> > >>>> >>> >> > > > > >> > > 4. Old good MR from command-line.
> > >>>> >>> >> > > > > >> > > 5. Security was simplified and now only
> > super-user
> > >>>> is
> > >>>> >>> >> allowed
> > >>>> >>> >>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

> > >> How hard to put in an hbase-backup module?
> >>>> hbase-server
> >>>> >>> is
> >>>> >>> >> fat
> >>>> >>> >> > > > enough
> >>>> >>> >> > > > > > >> already. Could be done as a follow-up.
> >>>> >>> >> > > > > >
> >>>> >>> >> > > > > > https://issues.apache.org/jira/browse/HBASE-16727?
> >>>> >>> >> > > > > > focusedCommentId=15531237=com.atlassian.jira.
> >>>> >>> >> > > > > > plugin.system.issuetabpanels:c
> >>>> >>> omment-tabpanel#comment-15531237
> >>>> >>> >> > > > > >
> >>>> >>> >> > > > > > Can we do merge first? Then we can discuss separate
> >>>> module.
> >>>> >>> >> > > > > >
> >>>> >>> >> > > > > > -Vlad
> >>>> >>> >> > > > > >
> >>>> >>> >> > > > > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu <
> >>>> >>> yuzhih...@gmail.com>
> >>>> >>> >> > wrote:
> >>>> >>> >> > > > > >
> >>>> >>> >> > > > > >> Looks like the first quote was cut off.
> >>>> >>> >> > > > > >> The original sentence was:
> >>>> >>> >> > > > > >>
> >>>> >>> >> > > > > >> bq. no mapreduce job launched from master or region
> >>>> server.
> >>>> >>> >> > > > > >>
> >>>> >>> >> > > > > >> mapreduce job is launched from the node where
> command
> >>>> line
> >>>> >>> >> tool is
> >>>> >>> >> > > > run.
> >>>> >>> >> > > > > >>
> >>>> >>> >> > > > > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack <
> >>>> st...@duboce.net>
> >>>> >>> >> wrote:
> >>>> >>> >> > > > > >>
> >>>> >>> >> > > > > >> > bq. launched from master or region server.
> >>>> >>> >> > > > > >> >
> >>>> >>> >> > > > > >> > What does this mean please? Has to be run from
> >>>> Master or
> >>>> >>> >> > > > RegionServer?
> >>>> >>> >> > > > > >> Can
> >>>> >>> >> > > > > >> > it be run from another node altogether?
> >>>> >>> >> > > > > >> >
> >>>> >>> >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir
> Rodionov <
> >>>> >>> >> > > > > >> vladrodio...@gmail.com
> >>>> >>> >> > > > > >> > >
> >>>> >>> >> > > > > >> > wrote:
> >>>> >>> >> > > > > >> >
> >>>> >>> >> > > > > >> > > >> mapreduce dependency has been moved to client
> >>>> side
> >>>> >>> - no
> >>>> >>> >> > > > mapreduce
> >>>> >>> >> > > > > >> job
> >>>> >>> >> > > > > >> > >
> >>>> >>> >> > > > > >> > > 1. We have no code in the client module anymore,
> >>>> due to
> >>>> >>> >> > > dependency
> >>>> >>> >> > > > > on
> >>>> >>> >> > > > > >> > > internal server API (HFile and WAL access).
> >>>> >>> >> > > > > >> > > 2. Backup/ restore are client - driven
> operations,
> >>>> but
> >>>> >>> all
> >>>> >>> >> the
> >>>> >>> >> > > > code
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Sean Busbey]

t;
>>>> >>> >> wrote:
>>>> >>> >> > > > > >>
>>>> >>> >> > > > > >> > bq. launched from master or region server.
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > What does this mean please? Has to be run from
>>>> Master or
>>>> >>> >> > > > RegionServer?
>>>> >>> >> > > > > >> Can
>>>> >>> >> > > > > >> > it be run from another node altogether?
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <
>>>> >>> >> > > > > >> vladrodio...@gmail.com
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > wrote:
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > > >> mapreduce dependency has been moved to client
>>>> side
>>>> >>> - no
>>>> >>> >> > > > mapreduce
>>>> >>> >> > > > > >> job
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > > 1. We have no code in the client module anymore,
>>>> due to
>>>> >>> >> > > dependency
>>>> >>> >> > > > > on
>>>> >>> >> > > > > >> > > internal server API (HFile and WAL access).
>>>> >>> >> > > > > >> > > 2. Backup/ restore are client - driven operations,
>>>> but
>>>> >>> all
>>>> >>> >> the
>>>> >>> >> > > > code
>>>> >>> >> > > > > >> > resides
>>>> >>> >> > > > > >> > > in the server module
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > How hard to put in an hbase-backup module?
>>>> hbase-server
>>>> >>> is
>>>> >>> >> fat
>>>> >>> >> > > > enough
>>>> >>> >> > > > > >> > already. Could be done as a follow-up.
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > Thanks,
>>>> >>> >> > > > > >> > St.Ack
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > > 3. No MR in Master, no procedure - driven
>>>> execution.
>>>> >>> >> > > > > >> > > 4. Old good MR from command-line.
>>>> >>> >> > > > > >> > > 5. Security was simplified and now only super-user
>>>> is
>>>> >>> >> allowed
>>>> >>> >> > to
>>>> >>> >> > > > run
>>>> >>> >> > > > > >> > > backup/restores.
>>>> >>> >> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only
>>>> >>> >> command-line
>>>> >>> >> > > > access
>>>> >>> >> > > > > to
>>>> >>> >> > > > > >> > > backup tools.
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > > These consequences of refactoring has been
>>>> discussed in
>>>> >>> >> > > > HBASE-16727.
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > > -Vlad
>>>&

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

gt;>> >> > > > > >>
>>>> >>> >> > > > > >> > bq. launched from master or region server.
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > What does this mean please? Has to be run from
>>>> Master or
>>>> >>> >> > > > RegionServer?
>>>> >>> >> > > > > >> Can
>>>> >>> >> > > > > >> > it be run from another node altogether?
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <
>>>> >>> >> > > > > >> vladrodio...@gmail.com
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > wrote:
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > > >> mapreduce dependency has been moved to client
>>>> side
>>>> >>> - no
>>>> >>> >> > > > mapreduce
>>>> >>> >> > > > > >> job
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > > 1. We have no code in the client module anymore,
>>>> due to
>>>> >>> >> > > dependency
>>>> >>> >> > > > > on
>>>> >>> >> > > > > >> > > internal server API (HFile and WAL access).
>>>> >>> >> > > > > >> > > 2. Backup/ restore are client - driven
>>>> operations, but
>>>> >>> all
>>>> >>> >> the
>>>> >>> >> > > > code
>>>> >>> >> > > > > >> > resides
>>>> >>> >> > > > > >> > > in the server module
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > How hard to put in an hbase-backup module?
>>>> hbase-server
>>>> >>> is
>>>> >>> >> fat
>>>> >>> >> > > > enough
>>>> >>> >> > > > > >> > already. Could be done as a follow-up.
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > Thanks,
>>>> >>> >> > > > > >> > St.Ack
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> >
>>>> >>> >> > > > > >> > > 3. No MR in Master, no procedure - driven
>>>> execution.
>>>> >>> >> > > > > >> > > 4. Old good MR from command-line.
>>>> >>> >> > > > > >> > > 5. Security was simplified and now only
>>>> super-user is
>>>> >>> >> allowed
>>>> >>> >> > to
>>>> >>> >> > > > run
>>>> >>> >> > > > > >> > > backup/restores.
>>>> >>> >> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only
>>>> >>> >> command-line
>>>> >>> >> > > > access
>>>> >>> >> > > > > to
>>>> >>> >> > > > > >> > > backup tools.
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > > These consequences of refactoring has been
>>>> discussed in
>>>> >>> >> > > > HBASE-16727.
>>>> >>> >> > > > > >> > >
>>>> >>> >> > > > > >> > > -Vlad
>>>> >>> >> > > > > >> > >
>>>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

; > > >> > > internal server API (HFile and WAL access).
>>> >>> >> > > > > >> > > 2. Backup/ restore are client - driven operations,
>>> but
>>> >>> all
>>> >>> >> the
>>> >>> >> > > > code
>>> >>> >> > > > > >> > resides
>>> >>> >> > > > > >> > > in the server module
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> >
>>> >>> >> > > > > >> > How hard to put in an hbase-backup module?
>>> hbase-server
>>> >>> is
>>> >>> >> fat
>>> >>> >> > > > enough
>>> >>> >> > > > > >> > already. Could be done as a follow-up.
>>> >>> >> > > > > >> >
>>> >>> >> > > > > >> > Thanks,
>>> >>> >> > > > > >> > St.Ack
>>> >>> >> > > > > >> >
>>> >>> >> > > > > >> >
>>> >>> >> > > > > >> >
>>> >>> >> > > > > >> > > 3. No MR in Master, no procedure - driven
>>> execution.
>>> >>> >> > > > > >> > > 4. Old good MR from command-line.
>>> >>> >> > > > > >> > > 5. Security was simplified and now only super-user
>>> is
>>> >>> >> allowed
>>> >>> >> > to
>>> >>> >> > > > run
>>> >>> >> > > > > >> > > backup/restores.
>>> >>> >> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only
>>> >>> >> command-line
>>> >>> >> > > > access
>>> >>> >> > > > > to
>>> >>> >> > > > > >> > > backup tools.
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> > > These consequences of refactoring has been
>>> discussed in
>>> >>> >> > > > HBASE-16727.
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> > > -Vlad
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <
>>> >>> >> yuzhih...@gmail.com>
>>> >>> >> > > > > wrote:
>>> >>> >> > > > > >> > >
>>> >>> >> > > > > >> > > > Reviving this thread.
>>> >>> >> > > > > >> > > >
>>> >>> >> > > > > >> > > > The following has taken place:
>>> >>> >> > > > > >> > > >
>>> >>> >> > > > > >> > > > mapreduce dependency has been moved to client
>>> side -
>>> >>> no
>>> >>> >> > > > mapreduce
>>> >>> >> > > > > >> job
>>> >>> >> > > > > >> > > > launched from master or region server.
>>> >>> >> > > > > >> > > > document patch (HBASE-16574) has been integrated.
>>> >>> >> > > > > >> > > > Updated mega patch has been attached to
>>> HBASE-14123:
>>> >>> this
>>> >>> >> > > covers
>>> >>> >> > > > > the
>>> >>> >> > > > > >> > > > refactor in #1 above and the protobuf 3 merge.
>>> >>> >> > > > > >> > > >
>>> >>> >> > > > > >> > > > If community has more feedback on the merge
>>> >>> proposal, I
>>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

; module.
>> >>> >> > > > > >
>> >>> >> > > > > > -Vlad
>> >>> >> > > > > >
>> >>> >> > > > > > On Mon, Oct 10, 2016 at 3:44 PM, Ted Yu <
>> >>> yuzhih...@gmail.com>
>> >>> >> > wrote:
>> >>> >> > > > > >
>> >>> >> > > > > >> Looks like the first quote was cut off.
>> >>> >> > > > > >> The original sentence was:
>> >>> >> > > > > >>
>> >>> >> > > > > >> bq. no mapreduce job launched from master or region
>> server.
>> >>> >> > > > > >>
>> >>> >> > > > > >> mapreduce job is launched from the node where command
>> line
>> >>> >> tool is
>> >>> >> > > > run.
>> >>> >> > > > > >>
>> >>> >> > > > > >> On Mon, Oct 10, 2016 at 3:38 PM, Stack <
>> st...@duboce.net>
>> >>> >> wrote:
>> >>> >> > > > > >>
>> >>> >> > > > > >> > bq. launched from master or region server.
>> >>> >> > > > > >> >
>> >>> >> > > > > >> > What does this mean please? Has to be run from Master
>> or
>> >>> >> > > > RegionServer?
>> >>> >> > > > > >> Can
>> >>> >> > > > > >> > it be run from another node altogether?
>> >>> >> > > > > >> >
>> >>> >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <
>> >>> >> > > > > >> vladrodio...@gmail.com
>> >>> >> > > > > >> > >
>> >>> >> > > > > >> > wrote:
>> >>> >> > > > > >> >
>> >>> >> > > > > >> > > >> mapreduce dependency has been moved to client
>> side
>> >>> - no
>> >>> >> > > > mapreduce
>> >>> >> > > > > >> job
>> >>> >> > > > > >> > >
>> >>> >> > > > > >> > > 1. We have no code in the client module anymore,
>> due to
>> >>> >> > > dependency
>> >>> >> > > > > on
>> >>> >> > > > > >> > > internal server API (HFile and WAL access).
>> >>> >> > > > > >> > > 2. Backup/ restore are client - driven operations,
>> but
>> >>> all
>> >>> >> the
>> >>> >> > > > code
>> >>> >> > > > > >> > resides
>> >>> >> > > > > >> > > in the server module
>> >>> >> > > > > >> > >
>> >>> >> > > > > >> >
>> >>> >> > > > > >> > How hard to put in an hbase-backup module?
>> hbase-server
>> >>> is
>> >>> >> fat
>> >>> >> > > > enough
>> >>> >> > > > > >> > already. Could be done as a follow-up.
>> >>> >> > > > > >> >
>> >>> >> > > > > >> > Thanks,
>> >>> >> > > > > >> > St.Ack
>> >>> >> > > > > >> >
>> >>> >> > > > > >> >
>> >>> >> > > > > >> >
>> >>> >> > > > > >> > > 3. No MR in Master, no procedure - driven execution.
>> >>> >> > > > > >> > > 4. Old good MR from command-line.
>> >>> >> > > > > >> > > 5. Security was simplified and now only super-user
>> is
>> >>> >> allowed
>> >>> >> > to
>> >>> >> > > > run
>> >>> >> > > > > >> > > backup/restores.
>> >>> >> > > > > >> > > 6. HBase B

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

:
> >>> >> > > > > >>
> >>> >> > > > > >> > bq. launched from master or region server.
> >>> >> > > > > >> >
> >>> >> > > > > >> > What does this mean please? Has to be run from Master
> or
> >>> >> > > > RegionServer?
> >>> >> > > > > >> Can
> >>> >> > > > > >> > it be run from another node altogether?
> >>> >> > > > > >> >
> >>> >> > > > > >> > On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <
> >>> >> > > > > >> vladrodio...@gmail.com
> >>> >> > > > > >> > >
> >>> >> > > > > >> > wrote:
> >>> >> > > > > >> >
> >>> >> > > > > >> > > >> mapreduce dependency has been moved to client side
> >>> - no
> >>> >> > > > mapreduce
> >>> >> > > > > >> job
> >>> >> > > > > >> > >
> >>> >> > > > > >> > > 1. We have no code in the client module anymore, due
> to
> >>> >> > > dependency
> >>> >> > > > > on
> >>> >> > > > > >> > > internal server API (HFile and WAL access).
> >>> >> > > > > >> > > 2. Backup/ restore are client - driven operations,
> but
> >>> all
> >>> >> the
> >>> >> > > > code
> >>> >> > > > > >> > resides
> >>> >> > > > > >> > > in the server module
> >>> >> > > > > >> > >
> >>> >> > > > > >> >
> >>> >> > > > > >> > How hard to put in an hbase-backup module? hbase-server
> >>> is
> >>> >> fat
> >>> >> > > > enough
> >>> >> > > > > >> > already. Could be done as a follow-up.
> >>> >> > > > > >> >
> >>> >> > > > > >> > Thanks,
> >>> >> > > > > >> > St.Ack
> >>> >> > > > > >> >
> >>> >> > > > > >> >
> >>> >> > > > > >> >
> >>> >> > > > > >> > > 3. No MR in Master, no procedure - driven execution.
> >>> >> > > > > >> > > 4. Old good MR from command-line.
> >>> >> > > > > >> > > 5. Security was simplified and now only super-user is
> >>> >> allowed
> >>> >> > to
> >>> >> > > > run
> >>> >> > > > > >> > > backup/restores.
> >>> >> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only
> >>> >> command-line
> >>> >> > > > access
> >>> >> > > > > to
> >>> >> > > > > >> > > backup tools.
> >>> >> > > > > >> > >
> >>> >> > > > > >> > > These consequences of refactoring has been discussed
> in
> >>> >> > > > HBASE-16727.
> >>> >> > > > > >> > >
> >>> >> > > > > >> > > -Vlad
> >>> >> > > > > >> > >
> >>> >> > > > > >> > >
> >>> >> > > > > >> > >
> >>> >> > > > > >> > >
> >>> >> > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <
> >>> >> yuzhih...@gmail.com>
> >>> >> > > > > wrote:
> >>> >> > > > > >> > >
> >>> >> > > > > >> > > > Reviving this thread.
> >>> >> > > > > >> > > >
> >>> >> > > > > >> > > > The following has taken place:
> >>> >> > > > > &g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Devaraj Das]

Ditto. Worked for me:

$ java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)



From: Vladimir Rodionov <vladrodio...@gmail.com>
Sent: Wednesday, October 12, 2016 1:10 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

checked out HBASE-7912

ran:

mvn clean install -DskipTests

successfully.

-Vlad

On Wed, Oct 12, 2016 at 1:01 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> I usually use:
>
> mvn clean install -DskipTests
>
> and it works.
>
> On Wed, Oct 12, 2016 at 1:01 PM, Vladimir Rodionov <vladrodio...@gmail.com
> > wrote:
>
>> Michael,
>>
>> you can try master + latest patch on HBASE-14123 (v29). No need to use
>> HBASE-7912 branch. I will double check HBASE-7912.
>>
>> -Vlad
>>
>> On Wed, Oct 12, 2016 at 12:31 PM, Stack <st...@duboce.net> wrote:
>>
>>> More info:
>>>
>>> stack@ve0524:~/hbase.git$ git checkout origin/HBASE-7912 -b 7912v2
>>> Branch 7912v2 set up to track remote branch HBASE-7912 from origin.
>>> Switched to a new branch '7912v2'
>>> stack@ve0524:~/hbase.git$ java -version
>>> java version "1.8.0_101"
>>> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
>>> stack@ve0524:~/hbase.git$ mvn clean install -DskipTests &> /tmp/out.txt
>>>
>>> ...
>>>
>>> St.Ack
>>>
>>>
>>> On Wed, Oct 12, 2016 at 12:29 PM, Stack <st...@duboce.net> wrote:
>>>
>>> > Interesting. When I try it fails w/ below:
>>> >
>>> > [INFO] 26 warnings
>>> > 322 [INFO] --
>>> ---
>>> > 323 [INFO] --
>>> ---
>>> > 324 [ERROR] COMPILATION ERROR :
>>> > 325 [INFO] --
>>> ---
>>> > 326 [ERROR] /home/stack/hbase.git/hbase-common/src/main/java/org/
>>> > apache/hadoop/hbase/io/encoding/rowindexV2/RowIndexCodecV2.java:[48,8]
>>> > org.apache.hadoop.hbase.io.encoding.rowindexV2.RowIndexCodecV2 is not
>>> > abstract and does not override abstract method createSeeker(org.ap
>>> >  ache.hadoop.hbase.CellComparator,org.apache.hadoop.hbase.io.
>>> encoding.HFileBlockDecodingContext)
>>> > in org.apache.hadoop.hbase.io.encoding.DataBlockEncoder
>>> > 327 [ERROR] /home/stack/hbase.git/hbase-common/src/main/java/org/
>>> > apache/hadoop/hbase/io/encoding/rowindexV2/RowIndexCodecV2.j
>>> ava:[143,3]
>>> > method does not override or implement a method from a supertype
>>> > 328 [ERROR] /home/stack/hbase.git/hbase-common/src/main/java/org/
>>> > apache/hadoop/hbase/io/encoding/rowindexV2/RowIndexCodecV2.j
>>> ava:[147,29]
>>> > incompatible types: java.nio.ByteBuffer cannot be converted to
>>> > org.apache.hadoop.hbase.nio.ByteBuff
>>> > 329 [ERROR] /home/stack/hbase.git/hbase-common/src/main/java/org/
>>> > apache/hadoop/hbase/io/encoding/rowindexV2/RowIndexCodecV2.j
>>> ava:[148,33]
>>> > cannot find symbol
>>> > 330   symbol:   method getKeyDeepCopy()
>>> > 331   location: variable seeker of type org.apache.hadoop.hbase.io.
>>> > encoding.DataBlockEncoder.EncodedSeeker
>>> > 332 [ERROR] /home/stack/hbase.git/hbase-common/src/main/java/org/
>>> > apache/hadoop/hbase/io/encoding/rowindexV2/RowIndexCodecV2.j
>>> ava:[153,3]
>>> > method does not override or implement a method from a supertype
>>> > 333 [ERROR] /home/stack/hbase.git/hbase-common/src/main/java/org/
>>> > apache/hadoop/hbase/io/encoding/rowindexV1/RowIndexCodecV1.java:[45,8]
>>> > org.apache.hadoop.hbase.io.encoding.rowindexV1.RowIndexCodecV1 is not
>>> > abstract and does not override abstract method createSeeker(org.ap
>>> >  ache.hadoop.hbase.CellComparator,org.apache.hadoop.hbase.io.
>>> encoding.HFileBlockDecodingContext)
>>> > in org.apache.hadoop.hbase.io.encoding.DataBlockEncoder
>>> > 334 [ERROR] /home/stack/hbase.git/hbase-common/src/main/java/org/
>>> > apache/hadoop/hbase/io/encoding/rowindexV1/RowIndexCodecV1.j
>>> ava:[145,3]
>>> > method does not override or implement a method from a supertype

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

 (HFile and WAL access).
>>> >> > > > > >> > > 2. Backup/ restore are client - driven operations, but
>>> all
>>> >> the
>>> >> > > > code
>>> >> > > > > >> > resides
>>> >> > > > > >> > > in the server module
>>> >> > > > > >> > >
>>> >> > > > > >> >
>>> >> > > > > >> > How hard to put in an hbase-backup module? hbase-server
>>> is
>>> >> fat
>>> >> > > > enough
>>> >> > > > > >> > already. Could be done as a follow-up.
>>> >> > > > > >> >
>>> >> > > > > >> > Thanks,
>>> >> > > > > >> > St.Ack
>>> >> > > > > >> >
>>> >> > > > > >> >
>>> >> > > > > >> >
>>> >> > > > > >> > > 3. No MR in Master, no procedure - driven execution.
>>> >> > > > > >> > > 4. Old good MR from command-line.
>>> >> > > > > >> > > 5. Security was simplified and now only super-user is
>>> >> allowed
>>> >> > to
>>> >> > > > run
>>> >> > > > > >> > > backup/restores.
>>> >> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only
>>> >> command-line
>>> >> > > > access
>>> >> > > > > to
>>> >> > > > > >> > > backup tools.
>>> >> > > > > >> > >
>>> >> > > > > >> > > These consequences of refactoring has been discussed in
>>> >> > > > HBASE-16727.
>>> >> > > > > >> > >
>>> >> > > > > >> > > -Vlad
>>> >> > > > > >> > >
>>> >> > > > > >> > >
>>> >> > > > > >> > >
>>> >> > > > > >> > >
>>> >> > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <
>>> >> yuzhih...@gmail.com>
>>> >> > > > > wrote:
>>> >> > > > > >> > >
>>> >> > > > > >> > > > Reviving this thread.
>>> >> > > > > >> > > >
>>> >> > > > > >> > > > The following has taken place:
>>> >> > > > > >> > > >
>>> >> > > > > >> > > > mapreduce dependency has been moved to client side -
>>> no
>>> >> > > > mapreduce
>>> >> > > > > >> job
>>> >> > > > > >> > > > launched from master or region server.
>>> >> > > > > >> > > > document patch (HBASE-16574) has been integrated.
>>> >> > > > > >> > > > Updated mega patch has been attached to HBASE-14123:
>>> this
>>> >> > > covers
>>> >> > > > > the
>>> >> > > > > >> > > > refactor in #1 above and the protobuf 3 merge.
>>> >> > > > > >> > > >
>>> >> > > > > >> > > > If community has more feedback on the merge
>>> proposal, I
>>> >> > would
>>> >> > > > love
>>> >> > > > > >> to
>>> >> > > > > >> > > hear
>>> >> > > > > >> > > > it.
>>> >> > > > > >> > > >
>>> >> > > > > >> > > > Thanks
>>> >> > > > > >> > > >
>>> >> > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
>>> >> > > > > bus...@cloudera.com>
>>> >> > > > > >> > > wrote:
>>> >> > > > > >> > > >
>>> >> > > > > >> > > > &g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

as simplified and now only super-user is
>> >> allowed
>> >> > to
>> >> > > > run
>> >> > > > > >> > > backup/restores.
>> >> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only
>> >> command-line
>> >> > > > access
>> >> > > > > to
>> >> > > > > >> > > backup tools.
>> >> > > > > >> > >
>> >> > > > > >> > > These consequences of refactoring has been discussed in
>> >> > > > HBASE-16727.
>> >> > > > > >> > >
>> >> > > > > >> > > -Vlad
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > >
>> >> > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <
>> >> yuzhih...@gmail.com>
>> >> > > > > wrote:
>> >> > > > > >> > >
>> >> > > > > >> > > > Reviving this thread.
>> >> > > > > >> > > >
>> >> > > > > >> > > > The following has taken place:
>> >> > > > > >> > > >
>> >> > > > > >> > > > mapreduce dependency has been moved to client side -
>> no
>> >> > > > mapreduce
>> >> > > > > >> job
>> >> > > > > >> > > > launched from master or region server.
>> >> > > > > >> > > > document patch (HBASE-16574) has been integrated.
>> >> > > > > >> > > > Updated mega patch has been attached to HBASE-14123:
>> this
>> >> > > covers
>> >> > > > > the
>> >> > > > > >> > > > refactor in #1 above and the protobuf 3 merge.
>> >> > > > > >> > > >
>> >> > > > > >> > > > If community has more feedback on the merge proposal,
>> I
>> >> > would
>> >> > > > love
>> >> > > > > >> to
>> >> > > > > >> > > hear
>> >> > > > > >> > > > it.
>> >> > > > > >> > > >
>> >> > > > > >> > > > Thanks
>> >> > > > > >> > > >
>> >> > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
>> >> > > > > bus...@cloudera.com>
>> >> > > > > >> > > wrote:
>> >> > > > > >> > > >
>> >> > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574
>> >> > integrated
>> >> > > > into
>> >> > > > > >> our
>> >> > > > > >> > > > > project's documentation prior to merge.
>> >> > > > > >> > > > >
>> >> > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <
>> >> > > yuzhih...@gmail.com>
>> >> > > > > >> wrote:
>> >> > > > > >> > > > > > This feature can be marked experimental due to
>> some
>> >> > > > > limitations
>> >> > > > > >> > such
>> >> > > > > >> > > as
>> >> > > > > >> > > > > > security.
>> >> > > > > >> > > > > >
>> >> > > > > >> > > > > > Your previous round of comments have been
>> addressed.
>> >> > > > > >> > > > > > Command line tool has gone through:
>> >> > > > > >> > > > > >
>> >> > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability
>> >> > issues
>> >> > > > > >> > > > > > HBASE-16655 hbase backup describe with incorrect
>> >> back

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

/restores.
> >> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only
> >> command-line
> >> > > > access
> >> > > > > to
> >> > > > > >> > > backup tools.
> >> > > > > >> > >
> >> > > > > >> > > These consequences of refactoring has been discussed in
> >> > > > HBASE-16727.
> >> > > > > >> > >
> >> > > > > >> > > -Vlad
> >> > > > > >> > >
> >> > > > > >> > >
> >> > > > > >> > >
> >> > > > > >> > >
> >> > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <
> >> yuzhih...@gmail.com>
> >> > > > > wrote:
> >> > > > > >> > >
> >> > > > > >> > > > Reviving this thread.
> >> > > > > >> > > >
> >> > > > > >> > > > The following has taken place:
> >> > > > > >> > > >
> >> > > > > >> > > > mapreduce dependency has been moved to client side - no
> >> > > > mapreduce
> >> > > > > >> job
> >> > > > > >> > > > launched from master or region server.
> >> > > > > >> > > > document patch (HBASE-16574) has been integrated.
> >> > > > > >> > > > Updated mega patch has been attached to HBASE-14123:
> this
> >> > > covers
> >> > > > > the
> >> > > > > >> > > > refactor in #1 above and the protobuf 3 merge.
> >> > > > > >> > > >
> >> > > > > >> > > > If community has more feedback on the merge proposal, I
> >> > would
> >> > > > love
> >> > > > > >> to
> >> > > > > >> > > hear
> >> > > > > >> > > > it.
> >> > > > > >> > > >
> >> > > > > >> > > > Thanks
> >> > > > > >> > > >
> >> > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
> >> > > > > bus...@cloudera.com>
> >> > > > > >> > > wrote:
> >> > > > > >> > > >
> >> > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574
> >> > integrated
> >> > > > into
> >> > > > > >> our
> >> > > > > >> > > > > project's documentation prior to merge.
> >> > > > > >> > > > >
> >> > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <
> >> > > yuzhih...@gmail.com>
> >> > > > > >> wrote:
> >> > > > > >> > > > > > This feature can be marked experimental due to some
> >> > > > > limitations
> >> > > > > >> > such
> >> > > > > >> > > as
> >> > > > > >> > > > > > security.
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > Your previous round of comments have been
> addressed.
> >> > > > > >> > > > > > Command line tool has gone through:
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability
> >> > issues
> >> > > > > >> > > > > > HBASE-16655 hbase backup describe with incorrect
> >> backup
> >> > id
> >> > > > > >> results
> >> > > > > >> > in
> >> > > > > >> > > > NPE
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > The updated doc has been attached to HBASE-16574.
> >> > > > > >> > > > > >
> >> > > > > >> > > > >

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

gt; >> > > > Reviving this thread.
> >> > > > > >> > > >
> >> > > > > >> > > > The following has taken place:
> >> > > > > >> > > >
> >> > > > > >> > > > mapreduce dependency has been moved to client side - no
> >> > > > mapreduce
> >> > > > > >> job
> >> > > > > >> > > > launched from master or region server.
> >> > > > > >> > > > document patch (HBASE-16574) has been integrated.
> >> > > > > >> > > > Updated mega patch has been attached to HBASE-14123:
> this
> >> > > covers
> >> > > > > the
> >> > > > > >> > > > refactor in #1 above and the protobuf 3 merge.
> >> > > > > >> > > >
> >> > > > > >> > > > If community has more feedback on the merge proposal, I
> >> > would
> >> > > > love
> >> > > > > >> to
> >> > > > > >> > > hear
> >> > > > > >> > > > it.
> >> > > > > >> > > >
> >> > > > > >> > > > Thanks
> >> > > > > >> > > >
> >> > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
> >> > > > > bus...@cloudera.com>
> >> > > > > >> > > wrote:
> >> > > > > >> > > >
> >> > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574
> >> > integrated
> >> > > > into
> >> > > > > >> our
> >> > > > > >> > > > > project's documentation prior to merge.
> >> > > > > >> > > > >
> >> > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <
> >> > > yuzhih...@gmail.com>
> >> > > > > >> wrote:
> >> > > > > >> > > > > > This feature can be marked experimental due to some
> >> > > > > limitations
> >> > > > > >> > such
> >> > > > > >> > > as
> >> > > > > >> > > > > > security.
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > Your previous round of comments have been
> addressed.
> >> > > > > >> > > > > > Command line tool has gone through:
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability
> >> > issues
> >> > > > > >> > > > > > HBASE-16655 hbase backup describe with incorrect
> >> backup
> >> > id
> >> > > > > >> results
> >> > > > > >> > in
> >> > > > > >> > > > NPE
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > The updated doc has been attached to HBASE-16574.
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > Cheers
> >> > > > > >> > > > > >
> >> > > > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <
> >> > st...@duboce.net>
> >> > > > > >> wrote:
> >> > > > > >> > > > > >
> >> > > > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <
> >> > > > yuzhih...@gmail.com
> >> > > > > >
> >> > > > > >> > > wrote:
> >> > > > > >> > > > > >>
> >> > > > > >> > > > > >> > Are there more (review) comments ?
> >> > > > > >> > > > > >> >
> >> > > > > >> > > > > >> >
> >> > > > > >> > > > > >> Are outstanding comments addressed?
> >> > > > > >> > > >

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

gt;
>> > > > > >> > > wrote:
>> > > > > >> > > >
>> > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574
>> > integrated
>> > > > into
>> > > > > >> our
>> > > > > >> > > > > project's documentation prior to merge.
>> > > > > >> > > > >
>> > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <
>> > > yuzhih...@gmail.com>
>> > > > > >> wrote:
>> > > > > >> > > > > > This feature can be marked experimental due to some
>> > > > > limitations
>> > > > > >> > such
>> > > > > >> > > as
>> > > > > >> > > > > > security.
>> > > > > >> > > > > >
>> > > > > >> > > > > > Your previous round of comments have been addressed.
>> > > > > >> > > > > > Command line tool has gone through:
>> > > > > >> > > > > >
>> > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability
>> > issues
>> > > > > >> > > > > > HBASE-16655 hbase backup describe with incorrect
>> backup
>> > id
>> > > > > >> results
>> > > > > >> > in
>> > > > > >> > > > NPE
>> > > > > >> > > > > >
>> > > > > >> > > > > > The updated doc has been attached to HBASE-16574.
>> > > > > >> > > > > >
>> > > > > >> > > > > > Cheers
>> > > > > >> > > > > >
>> > > > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <
>> > st...@duboce.net>
>> > > > > >> wrote:
>> > > > > >> > > > > >
>> > > > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <
>> > > > yuzhih...@gmail.com
>> > > > > >
>> > > > > >> > > wrote:
>> > > > > >> > > > > >>
>> > > > > >> > > > > >> > Are there more (review) comments ?
>> > > > > >> > > > > >> >
>> > > > > >> > > > > >> >
>> > > > > >> > > > > >> Are outstanding comments addressed?
>> > > > > >> > > > > >>
>> > > > > >> > > > > >> I don't see answer to my 'is this experimental/will
>> it
>> > be
>> > > > > >> marked
>> > > > > >> > > > > >> experimental' question.
>> > > > > >> > > > > >>
>> > > > > >> > > > > >> I ran into some issues trying to use the feature and
>> > > > > suggested
>> > > > > >> > that
>> > > > > >> > > a
>> > > > > >> > > > > >> feature likes this needs polish else it'll just rot,
>> > > > unused.
>> > > > > >> Has
>> > > > > >> > > > polish
>> > > > > >> > > > > >> been applied? All ready for another 'user' test?
>> > Suggest
>> > > > that
>> > > > > >> you
>> > > > > >> > > > update
>> > > > > >> > > > > >> here going forward for the benefit of those trying
>> to
>> > > > follow
>> > > > > >> along
>> > > > > >> > > and
>> > > > > >> > > > > who
>> > > > > >> > > > > >> are not watching JIRA change fly-by.
>> > > > > >> > > > > >>
>> > > > > >> > > > > >> It looks like doc got a revision -- I have to check
>> --
>> > to
>> > > > > take
>> > > > > >> on
>> > > > > >> 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

gt; > > >> > >
> > > > > >> >
> > > > > >> > How hard to put in an hbase-backup module? hbase-server is fat
> > > > enough
> > > > > >> > already. Could be done as a follow-up.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > St.Ack
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > > 3. No MR in Master, no procedure - driven execution.
> > > > > >> > > 4. Old good MR from command-line.
> > > > > >> > > 5. Security was simplified and now only super-user is
> allowed
> > to
> > > > run
> > > > > >> > > backup/restores.
> > > > > >> > > 6. HBase Backup API was gone due to 1. Now only command-line
> > > > access
> > > > > to
> > > > > >> > > backup tools.
> > > > > >> > >
> > > > > >> > > These consequences of refactoring has been discussed in
> > > > HBASE-16727.
> > > > > >> > >
> > > > > >> > > -Vlad
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <
> yuzhih...@gmail.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Reviving this thread.
> > > > > >> > > >
> > > > > >> > > > The following has taken place:
> > > > > >> > > >
> > > > > >> > > > mapreduce dependency has been moved to client side - no
> > > > mapreduce
> > > > > >> job
> > > > > >> > > > launched from master or region server.
> > > > > >> > > > document patch (HBASE-16574) has been integrated.
> > > > > >> > > > Updated mega patch has been attached to HBASE-14123: this
> > > covers
> > > > > the
> > > > > >> > > > refactor in #1 above and the protobuf 3 merge.
> > > > > >> > > >
> > > > > >> > > > If community has more feedback on the merge proposal, I
> > would
> > > > love
> > > > > >> to
> > > > > >> > > hear
> > > > > >> > > > it.
> > > > > >> > > >
> > > > > >> > > > Thanks
> > > > > >> > > >
> > > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
> > > > > bus...@cloudera.com>
> > > > > >> > > wrote:
> > > > > >> > > >
> > > > > >> > > > > I'd like to see the docs proposed on HBASE-16574
> > integrated
> > > > into
> > > > > >> our
> > > > > >> > > > > project's documentation prior to merge.
> > > > > >> > > > >
> > > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <
> > > yuzhih...@gmail.com>
> > > > > >> wrote:
> > > > > >> > > > > > This feature can be marked experimental due to some
> > > > > limitations
> > > > > >> > such
> > > > > >> > > as
> > > > > >> > > > > > security.
> > > > > >> > > > > >
> > > > > >> > > > > > Your previous round of comments have been addressed.
> > > > > >> > > > > > Command line tool has gone through:
> > > > > >> > > > > >
> > > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability
> > issues
> > > > > >> > > > > > HBASE-16655 hbase backup describe with incorrect
> backup
> > id
> > > > > >> results
> > > > > >> > in
> > > > > >> > > > NPE
> > > > > >> > > > > >
> > > > >

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Apekshit Sharma]

> > > 3. No MR in Master, no procedure - driven execution.
> > > > >> > > 4. Old good MR from command-line.
> > > > >> > > 5. Security was simplified and now only super-user is allowed
> to
> > > run
> > > > >> > > backup/restores.
> > > > >> > > 6. HBase Backup API was gone due to 1. Now only command-line
> > > access
> > > > to
> > > > >> > > backup tools.
> > > > >> > >
> > > > >> > > These consequences of refactoring has been discussed in
> > > HBASE-16727.
> > > > >> > >
> > > > >> > > -Vlad
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > Reviving this thread.
> > > > >> > > >
> > > > >> > > > The following has taken place:
> > > > >> > > >
> > > > >> > > > mapreduce dependency has been moved to client side - no
> > > mapreduce
> > > > >> job
> > > > >> > > > launched from master or region server.
> > > > >> > > > document patch (HBASE-16574) has been integrated.
> > > > >> > > > Updated mega patch has been attached to HBASE-14123: this
> > covers
> > > > the
> > > > >> > > > refactor in #1 above and the protobuf 3 merge.
> > > > >> > > >
> > > > >> > > > If community has more feedback on the merge proposal, I
> would
> > > love
> > > > >> to
> > > > >> > > hear
> > > > >> > > > it.
> > > > >> > > >
> > > > >> > > > Thanks
> > > > >> > > >
> > > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
> > > > bus...@cloudera.com>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > I'd like to see the docs proposed on HBASE-16574
> integrated
> > > into
> > > > >> our
> > > > >> > > > > project's documentation prior to merge.
> > > > >> > > > >
> > > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <
> > yuzhih...@gmail.com>
> > > > >> wrote:
> > > > >> > > > > > This feature can be marked experimental due to some
> > > > limitations
> > > > >> > such
> > > > >> > > as
> > > > >> > > > > > security.
> > > > >> > > > > >
> > > > >> > > > > > Your previous round of comments have been addressed.
> > > > >> > > > > > Command line tool has gone through:
> > > > >> > > > > >
> > > > >> > > > > > HBASE-16620 Fix backup command-line tool usability
> issues
> > > > >> > > > > > HBASE-16655 hbase backup describe with incorrect backup
> id
> > > > >> results
> > > > >> > in
> > > > >> > > > NPE
> > > > >> > > > > >
> > > > >> > > > > > The updated doc has been attached to HBASE-16574.
> > > > >> > > > > >
> > > > >> > > > > > Cheers
> > > > >> > > > > >
> > > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <
> st...@duboce.net>
> > > > >> wrote:
> > > > >> > > > > >
> > > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <
> > > yuzhih...@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > > > >>
> > > > >> > > > > >> > Are there more (review) comments ?
> > > > >> > > > > >> >
> > > > >> > > > > >> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

> >> > >
> > > >> > >
> > > >> > > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com>
> > > wrote:
> > > >> > >
> > > >> > > > Reviving this thread.
> > > >> > > >
> > > >> > > > The following has taken place:
> > > >> > > >
> > > >> > > > mapreduce dependency has been moved to client side - no
> > mapreduce
> > > >> job
> > > >> > > > launched from master or region server.
> > > >> > > > document patch (HBASE-16574) has been integrated.
> > > >> > > > Updated mega patch has been attached to HBASE-14123: this
> covers
> > > the
> > > >> > > > refactor in #1 above and the protobuf 3 merge.
> > > >> > > >
> > > >> > > > If community has more feedback on the merge proposal, I would
> > love
> > > >> to
> > > >> > > hear
> > > >> > > > it.
> > > >> > > >
> > > >> > > > Thanks
> > > >> > > >
> > > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
> > > bus...@cloudera.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated
> > into
> > > >> our
> > > >> > > > > project's documentation prior to merge.
> > > >> > > > >
> > > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <
> yuzhih...@gmail.com>
> > > >> wrote:
> > > >> > > > > > This feature can be marked experimental due to some
> > > limitations
> > > >> > such
> > > >> > > as
> > > >> > > > > > security.
> > > >> > > > > >
> > > >> > > > > > Your previous round of comments have been addressed.
> > > >> > > > > > Command line tool has gone through:
> > > >> > > > > >
> > > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues
> > > >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id
> > > >> results
> > > >> > in
> > > >> > > > NPE
> > > >> > > > > >
> > > >> > > > > > The updated doc has been attached to HBASE-16574.
> > > >> > > > > >
> > > >> > > > > > Cheers
> > > >> > > > > >
> > > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net>
> > > >> wrote:
> > > >> > > > > >
> > > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <
> > yuzhih...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > > > >>
> > > >> > > > > >> > Are there more (review) comments ?
> > > >> > > > > >> >
> > > >> > > > > >> >
> > > >> > > > > >> Are outstanding comments addressed?
> > > >> > > > > >>
> > > >> > > > > >> I don't see answer to my 'is this experimental/will it be
> > > >> marked
> > > >> > > > > >> experimental' question.
> > > >> > > > > >>
> > > >> > > > > >> I ran into some issues trying to use the feature and
> > > suggested
> > > >> > that
> > > >> > > a
> > > >> > > > > >> feature likes this needs polish else it'll just rot,
> > unused.
> > > >> Has
> > > >> > > > polish
> > > >> > > > > >> been applied? All ready for another 'user' test? Suggest
> > that
> > > >> you
> > > >> > > > update
> > > >> > > > > >> here going forward for the benefit of those trying to
> > follow
> > > >> along
> > > >> > > and
> > > >&

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

:31 PM, Ted Yu <yuzhih...@gmail.com>
>> > wrote:
>> > >> > >
>> > >> > > > Reviving this thread.
>> > >> > > >
>> > >> > > > The following has taken place:
>> > >> > > >
>> > >> > > > mapreduce dependency has been moved to client side - no
>> mapreduce
>> > >> job
>> > >> > > > launched from master or region server.
>> > >> > > > document patch (HBASE-16574) has been integrated.
>> > >> > > > Updated mega patch has been attached to HBASE-14123: this
>> covers
>> > the
>> > >> > > > refactor in #1 above and the protobuf 3 merge.
>> > >> > > >
>> > >> > > > If community has more feedback on the merge proposal, I would
>> love
>> > >> to
>> > >> > > hear
>> > >> > > > it.
>> > >> > > >
>> > >> > > > Thanks
>> > >> > > >
>> > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
>> > bus...@cloudera.com>
>> > >> > > wrote:
>> > >> > > >
>> > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated
>> into
>> > >> our
>> > >> > > > > project's documentation prior to merge.
>> > >> > > > >
>> > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com
>> >
>> > >> wrote:
>> > >> > > > > > This feature can be marked experimental due to some
>> > limitations
>> > >> > such
>> > >> > > as
>> > >> > > > > > security.
>> > >> > > > > >
>> > >> > > > > > Your previous round of comments have been addressed.
>> > >> > > > > > Command line tool has gone through:
>> > >> > > > > >
>> > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues
>> > >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id
>> > >> results
>> > >> > in
>> > >> > > > NPE
>> > >> > > > > >
>> > >> > > > > > The updated doc has been attached to HBASE-16574.
>> > >> > > > > >
>> > >> > > > > > Cheers
>> > >> > > > > >
>> > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net>
>> > >> wrote:
>> > >> > > > > >
>> > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <
>> yuzhih...@gmail.com
>> > >
>> > >> > > wrote:
>> > >> > > > > >>
>> > >> > > > > >> > Are there more (review) comments ?
>> > >> > > > > >> >
>> > >> > > > > >> >
>> > >> > > > > >> Are outstanding comments addressed?
>> > >> > > > > >>
>> > >> > > > > >> I don't see answer to my 'is this experimental/will it be
>> > >> marked
>> > >> > > > > >> experimental' question.
>> > >> > > > > >>
>> > >> > > > > >> I ran into some issues trying to use the feature and
>> > suggested
>> > >> > that
>> > >> > > a
>> > >> > > > > >> feature likes this needs polish else it'll just rot,
>> unused.
>> > >> Has
>> > >> > > > polish
>> > >> > > > > >> been applied? All ready for another 'user' test? Suggest
>> that
>> > >> you
>> > >> > > > update
>> > >> > > > > >> here going forward for the benefit of those trying to
>> follow
>> > >> along
>> > >> > > and
>> > >> > > > > who
>> > >> > > > > >> are not watching JIRA change fly-by.
>> > >> > > > > >>
>> > >>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

 the
> > >> > > > refactor in #1 above and the protobuf 3 merge.
> > >> > > >
> > >> > > > If community has more feedback on the merge proposal, I would
> love
> > >> to
> > >> > > hear
> > >> > > > it.
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <
> > bus...@cloudera.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > I'd like to see the docs proposed on HBASE-16574 integrated
> into
> > >> our
> > >> > > > > project's documentation prior to merge.
> > >> > > > >
> > >> > > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com>
> > >> wrote:
> > >> > > > > > This feature can be marked experimental due to some
> > limitations
> > >> > such
> > >> > > as
> > >> > > > > > security.
> > >> > > > > >
> > >> > > > > > Your previous round of comments have been addressed.
> > >> > > > > > Command line tool has gone through:
> > >> > > > > >
> > >> > > > > > HBASE-16620 Fix backup command-line tool usability issues
> > >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id
> > >> results
> > >> > in
> > >> > > > NPE
> > >> > > > > >
> > >> > > > > > The updated doc has been attached to HBASE-16574.
> > >> > > > > >
> > >> > > > > > Cheers
> > >> > > > > >
> > >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net>
> > >> wrote:
> > >> > > > > >
> > >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <
> yuzhih...@gmail.com
> > >
> > >> > > wrote:
> > >> > > > > >>
> > >> > > > > >> > Are there more (review) comments ?
> > >> > > > > >> >
> > >> > > > > >> >
> > >> > > > > >> Are outstanding comments addressed?
> > >> > > > > >>
> > >> > > > > >> I don't see answer to my 'is this experimental/will it be
> > >> marked
> > >> > > > > >> experimental' question.
> > >> > > > > >>
> > >> > > > > >> I ran into some issues trying to use the feature and
> > suggested
> > >> > that
> > >> > > a
> > >> > > > > >> feature likes this needs polish else it'll just rot,
> unused.
> > >> Has
> > >> > > > polish
> > >> > > > > >> been applied? All ready for another 'user' test? Suggest
> that
> > >> you
> > >> > > > update
> > >> > > > > >> here going forward for the benefit of those trying to
> follow
> > >> along
> > >> > > and
> > >> > > > > who
> > >> > > > > >> are not watching JIRA change fly-by.
> > >> > > > > >>
> > >> > > > > >> It looks like doc got a revision -- I have to check -- to
> > take
> > >> on
> > >> > > > > >> suggestion made above but again, suggest, that this thread
> > gets
> > >> > > > updated.
> > >> > > > > >>
> > >> > > > > >> Thanks,
> > >> > > > > >> St.Ack
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >>
> > >> > > > > >> > Thanks
> > >> > > > > >> >
> > >> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <
> > >> > > d...@hortonworks.com
> > >> > > > >
> > >> > > > > >> > wrote:
> > >> > > > > >> >
> > >> > > > > >&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

; > security.
> >> > > > > >
> >> > > > > > Your previous round of comments have been addressed.
> >> > > > > > Command line tool has gone through:
> >> > > > > >
> >> > > > > > HBASE-16620 Fix backup command-line tool usability issues
> >> > > > > > HBASE-16655 hbase backup describe with incorrect backup id
> >> results
> >> > in
> >> > > > NPE
> >> > > > > >
> >> > > > > > The updated doc has been attached to HBASE-16574.
> >> > > > > >
> >> > > > > > Cheers
> >> > > > > >
> >> > > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net>
> >> wrote:
> >> > > > > >
> >> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com
> >
> >> > > wrote:
> >> > > > > >>
> >> > > > > >> > Are there more (review) comments ?
> >> > > > > >> >
> >> > > > > >> >
> >> > > > > >> Are outstanding comments addressed?
> >> > > > > >>
> >> > > > > >> I don't see answer to my 'is this experimental/will it be
> >> marked
> >> > > > > >> experimental' question.
> >> > > > > >>
> >> > > > > >> I ran into some issues trying to use the feature and
> suggested
> >> > that
> >> > > a
> >> > > > > >> feature likes this needs polish else it'll just rot, unused.
> >> Has
> >> > > > polish
> >> > > > > >> been applied? All ready for another 'user' test? Suggest that
> >> you
> >> > > > update
> >> > > > > >> here going forward for the benefit of those trying to follow
> >> along
> >> > > and
> >> > > > > who
> >> > > > > >> are not watching JIRA change fly-by.
> >> > > > > >>
> >> > > > > >> It looks like doc got a revision -- I have to check -- to
> take
> >> on
> >> > > > > >> suggestion made above but again, suggest, that this thread
> gets
> >> > > > updated.
> >> > > > > >>
> >> > > > > >> Thanks,
> >> > > > > >> St.Ack
> >> > > > > >>
> >> > > > > >>
> >> > > > > >>
> >> > > > > >> > Thanks
> >> > > > > >> >
> >> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <
> >> > > d...@hortonworks.com
> >> > > > >
> >> > > > > >> > wrote:
> >> > > > > >> >
> >> > > > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and
> >> > others
> >> > > > for
> >> > > > > the
> >> > > > > >> > > thorough reviews and testing. Thanks Ted and Vlad for
> >> taking
> >> > > care
> >> > > > of
> >> > > > > >> the
> >> > > > > >> > > feedback. Are we all good to do the merge now? Rather do
> >> > sooner
> >> > > > than
> >> > > > > >> > later.
> >> > > > > >> > > 
> >> > > > > >> > > From: saint@gmail.com <saint@gmail.com> on
> behalf
> >> of
> >> > > > Stack
> >> > > > > <
> >> > > > > >> > > st...@duboce.net>
> >> > > > > >> > > Sent: Monday, September 12, 2016 1:18 PM
> >> > > > > >> > > To: HBase Dev List
> >> > > > > >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
> >> > > > HBASE-7912
> >> > > > > >> > >
> >> > > > > >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <
> >> yuzhih...@gmail.com
> >> > >
> >> &

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

> > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net>
>> wrote:
>> > > > > >
>> > > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com>
>> > > wrote:
>> > > > > >>
>> > > > > >> > Are there more (review) comments ?
>> > > > > >> >
>> > > > > >> >
>> > > > > >> Are outstanding comments addressed?
>> > > > > >>
>> > > > > >> I don't see answer to my 'is this experimental/will it be
>> marked
>> > > > > >> experimental' question.
>> > > > > >>
>> > > > > >> I ran into some issues trying to use the feature and suggested
>> > that
>> > > a
>> > > > > >> feature likes this needs polish else it'll just rot, unused.
>> Has
>> > > > polish
>> > > > > >> been applied? All ready for another 'user' test? Suggest that
>> you
>> > > > update
>> > > > > >> here going forward for the benefit of those trying to follow
>> along
>> > > and
>> > > > > who
>> > > > > >> are not watching JIRA change fly-by.
>> > > > > >>
>> > > > > >> It looks like doc got a revision -- I have to check -- to take
>> on
>> > > > > >> suggestion made above but again, suggest, that this thread gets
>> > > > updated.
>> > > > > >>
>> > > > > >> Thanks,
>> > > > > >> St.Ack
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > > > >> > Thanks
>> > > > > >> >
>> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <
>> > > d...@hortonworks.com
>> > > > >
>> > > > > >> > wrote:
>> > > > > >> >
>> > > > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and
>> > others
>> > > > for
>> > > > > the
>> > > > > >> > > thorough reviews and testing. Thanks Ted and Vlad for
>> taking
>> > > care
>> > > > of
>> > > > > >> the
>> > > > > >> > > feedback. Are we all good to do the merge now? Rather do
>> > sooner
>> > > > than
>> > > > > >> > later.
>> > > > > >> > > 
>> > > > > >> > > From: saint@gmail.com <saint@gmail.com> on behalf
>> of
>> > > > Stack
>> > > > > <
>> > > > > >> > > st...@duboce.net>
>> > > > > >> > > Sent: Monday, September 12, 2016 1:18 PM
>> > > > > >> > > To: HBase Dev List
>> > > > > >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
>> > > > HBASE-7912
>> > > > > >> > >
>> > > > > >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <
>> yuzhih...@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >> > >
>> > > > > >> > > > Mega patch (rev 18) is on HBASE-14123.
>> > > > > >> > > >
>> > > > > >> > > > Please comment on HBASE-14123 on how you want to review.
>> > > > > >> > > >
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > Yeah. That was my lost tab. Last rb was 6 months ago.
>> Suggest
>> > > > > updating
>> > > > > >> > it.
>> > > > > >> > > RB is pretty good for review. Patch is only 1.5M so should
>> be
>> > > > fine.
>> > > > > >> > >
>> > > > > >> > > St.Ack
>> > > > > >> > >
>> > > > > >> > >
>> > > > > >> > > >
>> > > > > >> > > > Thanks
>> > > > > >> > > >
&g

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

gt; > >> experimental' question.
> > > > > >>
> > > > > >> I ran into some issues trying to use the feature and suggested
> > that
> > > a
> > > > > >> feature likes this needs polish else it'll just rot, unused. Has
> > > > polish
> > > > > >> been applied? All ready for another 'user' test? Suggest that
> you
> > > > update
> > > > > >> here going forward for the benefit of those trying to follow
> along
> > > and
> > > > > who
> > > > > >> are not watching JIRA change fly-by.
> > > > > >>
> > > > > >> It looks like doc got a revision -- I have to check -- to take
> on
> > > > > >> suggestion made above but again, suggest, that this thread gets
> > > > updated.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> St.Ack
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > Thanks
> > > > > >> >
> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <
> > > d...@hortonworks.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and
> > others
> > > > for
> > > > > the
> > > > > >> > > thorough reviews and testing. Thanks Ted and Vlad for taking
> > > care
> > > > of
> > > > > >> the
> > > > > >> > > feedback. Are we all good to do the merge now? Rather do
> > sooner
> > > > than
> > > > > >> > later.
> > > > > >> > > 
> > > > > >> > > From: saint@gmail.com <saint@gmail.com> on behalf
> of
> > > > Stack
> > > > > <
> > > > > >> > > st...@duboce.net>
> > > > > >> > > Sent: Monday, September 12, 2016 1:18 PM
> > > > > >> > > To: HBase Dev List
> > > > > >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
> > > > HBASE-7912
> > > > > >> > >
> > > > > >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <
> yuzhih...@gmail.com
> > >
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Mega patch (rev 18) is on HBASE-14123.
> > > > > >> > > >
> > > > > >> > > > Please comment on HBASE-14123 on how you want to review.
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Yeah. That was my lost tab. Last rb was 6 months ago.
> Suggest
> > > > > updating
> > > > > >> > it.
> > > > > >> > > RB is pretty good for review. Patch is only 1.5M so should
> be
> > > > fine.
> > > > > >> > >
> > > > > >> > > St.Ack
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > Thanks
> > > > > >> > > >
> > > > > >> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net
> >
> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > On review of the 'patch', do I just compare the branch
> to
> > > > > master or
> > > > > >> > is
> > > > > >> > > > > there a megapatch posted somewhere (I think I saw one
> but
> > it
> > > > > seemed
> > > > > >> > > stale
> > > > > >> > > > > and then I 'lost' the tab). Sorry for dumb question.
> > > > > >> > > > > St.Ack
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <
> st...@duboce.net
> > >
> > > > > wrote:
> > > > > >> > > > >
> > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Looks like the first quote was cut off.
The original sentence was:

bq. no mapreduce job launched from master or region server.

mapreduce job is launched from the node where command line tool is run.

On Mon, Oct 10, 2016 at 3:38 PM, Stack <st...@duboce.net> wrote:

> bq. launched from master or region server.
>
> What does this mean please? Has to be run from Master or RegionServer? Can
> it be run from another node altogether?
>
> On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <vladrodio...@gmail.com
> >
> wrote:
>
> > >> mapreduce dependency has been moved to client side - no mapreduce job
> >
> > 1. We have no code in the client module anymore, due to dependency on
> > internal server API (HFile and WAL access).
> > 2. Backup/ restore are client - driven operations, but all the code
> resides
> > in the server module
> >
>
> How hard to put in an hbase-backup module? hbase-server is fat enough
> already. Could be done as a follow-up.
>
> Thanks,
> St.Ack
>
>
>
> > 3. No MR in Master, no procedure - driven execution.
> > 4. Old good MR from command-line.
> > 5. Security was simplified and now only super-user is allowed to run
> > backup/restores.
> > 6. HBase Backup API was gone due to 1. Now only command-line access to
> > backup tools.
> >
> > These consequences of refactoring has been discussed in HBASE-16727.
> >
> > -Vlad
> >
> >
> >
> >
> > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > Reviving this thread.
> > >
> > > The following has taken place:
> > >
> > > mapreduce dependency has been moved to client side - no mapreduce job
> > > launched from master or region server.
> > > document patch (HBASE-16574) has been integrated.
> > > Updated mega patch has been attached to HBASE-14123: this covers the
> > > refactor in #1 above and the protobuf 3 merge.
> > >
> > > If community has more feedback on the merge proposal, I would love to
> > hear
> > > it.
> > >
> > > Thanks
> > >
> > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com>
> > wrote:
> > >
> > > > I'd like to see the docs proposed on HBASE-16574 integrated into our
> > > > project's documentation prior to merge.
> > > >
> > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > > > > This feature can be marked experimental due to some limitations
> such
> > as
> > > > > security.
> > > > >
> > > > > Your previous round of comments have been addressed.
> > > > > Command line tool has gone through:
> > > > >
> > > > > HBASE-16620 Fix backup command-line tool usability issues
> > > > > HBASE-16655 hbase backup describe with incorrect backup id results
> in
> > > NPE
> > > > >
> > > > > The updated doc has been attached to HBASE-16574.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > Are there more (review) comments ?
> > > > >> >
> > > > >> >
> > > > >> Are outstanding comments addressed?
> > > > >>
> > > > >> I don't see answer to my 'is this experimental/will it be marked
> > > > >> experimental' question.
> > > > >>
> > > > >> I ran into some issues trying to use the feature and suggested
> that
> > a
> > > > >> feature likes this needs polish else it'll just rot, unused. Has
> > > polish
> > > > >> been applied? All ready for another 'user' test? Suggest that you
> > > update
> > > > >> here going forward for the benefit of those trying to follow along
> > and
> > > > who
> > > > >> are not watching JIRA change fly-by.
> > > > >>
> > > > >> It looks like doc got a revision -- I have to check -- to take on
> > > > >> suggestion made above but again, suggest, that this thread gets
> > > updated.
> > > > >>
> > > > >> Thanks,
> > > > >> St.Ack
> &

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

bq. launched from master or region server.

What does this mean please? Has to be run from Master or RegionServer? Can
it be run from another node altogether?

On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> >> mapreduce dependency has been moved to client side - no mapreduce job
>
> 1. We have no code in the client module anymore, due to dependency on
> internal server API (HFile and WAL access).
> 2. Backup/ restore are client - driven operations, but all the code resides
> in the server module
>

How hard to put in an hbase-backup module? hbase-server is fat enough
already. Could be done as a follow-up.

Thanks,
St.Ack



> 3. No MR in Master, no procedure - driven execution.
> 4. Old good MR from command-line.
> 5. Security was simplified and now only super-user is allowed to run
> backup/restores.
> 6. HBase Backup API was gone due to 1. Now only command-line access to
> backup tools.
>
> These consequences of refactoring has been discussed in HBASE-16727.
>
> -Vlad
>
>
>
>
> On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Reviving this thread.
> >
> > The following has taken place:
> >
> > mapreduce dependency has been moved to client side - no mapreduce job
> > launched from master or region server.
> > document patch (HBASE-16574) has been integrated.
> > Updated mega patch has been attached to HBASE-14123: this covers the
> > refactor in #1 above and the protobuf 3 merge.
> >
> > If community has more feedback on the merge proposal, I would love to
> hear
> > it.
> >
> > Thanks
> >
> > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com>
> wrote:
> >
> > > I'd like to see the docs proposed on HBASE-16574 integrated into our
> > > project's documentation prior to merge.
> > >
> > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > > > This feature can be marked experimental due to some limitations such
> as
> > > > security.
> > > >
> > > > Your previous round of comments have been addressed.
> > > > Command line tool has gone through:
> > > >
> > > > HBASE-16620 Fix backup command-line tool usability issues
> > > > HBASE-16655 hbase backup describe with incorrect backup id results in
> > NPE
> > > >
> > > > The updated doc has been attached to HBASE-16574.
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:
> > > >
> > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> > > >>
> > > >> > Are there more (review) comments ?
> > > >> >
> > > >> >
> > > >> Are outstanding comments addressed?
> > > >>
> > > >> I don't see answer to my 'is this experimental/will it be marked
> > > >> experimental' question.
> > > >>
> > > >> I ran into some issues trying to use the feature and suggested that
> a
> > > >> feature likes this needs polish else it'll just rot, unused. Has
> > polish
> > > >> been applied? All ready for another 'user' test? Suggest that you
> > update
> > > >> here going forward for the benefit of those trying to follow along
> and
> > > who
> > > >> are not watching JIRA change fly-by.
> > > >>
> > > >> It looks like doc got a revision -- I have to check -- to take on
> > > >> suggestion made above but again, suggest, that this thread gets
> > updated.
> > > >>
> > > >> Thanks,
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >> > Thanks
> > > >> >
> > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <
> d...@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others
> > for
> > > the
> > > >> > > thorough reviews and testing. Thanks Ted and Vlad for taking
> care
> > of
> > > >> the
> > > >> > > feedback. Are we all good to do the merge now? Rather do sooner
> > than
> > > >> > later.
> > > >> > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Enis Söztutar]

kes this needs polish else it'll just rot, unused. Has
> > > > polish
> > > > > >> been applied? All ready for another 'user' test? Suggest that
> you
> > > > update
> > > > > >> here going forward for the benefit of those trying to follow
> along
> > > and
> > > > > who
> > > > > >> are not watching JIRA change fly-by.
> > > > > >>
> > > > > >> It looks like doc got a revision -- I have to check -- to take
> on
> > > > > >> suggestion made above but again, suggest, that this thread gets
> > > > updated.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> St.Ack
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > Thanks
> > > > > >> >
> > > > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <
> > > d...@hortonworks.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and
> > others
> > > > for
> > > > > the
> > > > > >> > > thorough reviews and testing. Thanks Ted and Vlad for taking
> > > care
> > > > of
> > > > > >> the
> > > > > >> > > feedback. Are we all good to do the merge now? Rather do
> > sooner
> > > > than
> > > > > >> > later.
> > > > > >> > > 
> > > > > >> > > From: saint@gmail.com <saint@gmail.com> on behalf
> of
> > > > Stack
> > > > > <
> > > > > >> > > st...@duboce.net>
> > > > > >> > > Sent: Monday, September 12, 2016 1:18 PM
> > > > > >> > > To: HBase Dev List
> > > > > >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
> > > > HBASE-7912
> > > > > >> > >
> > > > > >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <
> yuzhih...@gmail.com
> > >
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Mega patch (rev 18) is on HBASE-14123.
> > > > > >> > > >
> > > > > >> > > > Please comment on HBASE-14123 on how you want to review.
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Yeah. That was my lost tab. Last rb was 6 months ago.
> Suggest
> > > > > updating
> > > > > >> > it.
> > > > > >> > > RB is pretty good for review. Patch is only 1.5M so should
> be
> > > > fine.
> > > > > >> > >
> > > > > >> > > St.Ack
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > Thanks
> > > > > >> > > >
> > > > > >> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net
> >
> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > On review of the 'patch', do I just compare the branch
> to
> > > > > master or
> > > > > >> > is
> > > > > >> > > > > there a megapatch posted somewhere (I think I saw one
> but
> > it
> > > > > seemed
> > > > > >> > > stale
> > > > > >> > > > > and then I 'lost' the tab). Sorry for dumb question.
> > > > > >> > > > > St.Ack
> > > > > >> > > > >
> > > > > >> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <
> st...@duboce.net
> > >
> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Late to the game. A few comments after rereading this
> > > thread
> > > > > as a
> > > > > >> > > > 'user'.
> > > > > >>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

bq. Does the branch ready for merge

Yes - see the mega patch.

We can discuss the actual procedure of merging if the community gives green
light to merging.

A vote thread can be started if there is no further comment.

On Mon, Oct 10, 2016 at 2:42 PM, Enis Söztutar <e...@apache.org> wrote:

> Great work.
>
> So what are the next steps? It seems from the discussion that all
> outstanding requests have been addressed. Does the branch ready for merge
> or, you have to rebase the branch as well?
>
> Time to start a merge vote?
> Enis
>
> On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <vladrodio...@gmail.com
> >
> wrote:
>
> > >> mapreduce dependency has been moved to client side - no mapreduce job
> >
> > 1. We have no code in the client module anymore, due to dependency on
> > internal server API (HFile and WAL access).
> > 2. Backup/ restore are client - driven operations, but all the code
> resides
> > in the server module
> > 3. No MR in Master, no procedure - driven execution.
> > 4. Old good MR from command-line.
> > 5. Security was simplified and now only super-user is allowed to run
> > backup/restores.
> > 6. HBase Backup API was gone due to 1. Now only command-line access to
> > backup tools.
> >
> > These consequences of refactoring has been discussed in HBASE-16727.
> >
> > -Vlad
> >
> >
> >
> >
> > On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > Reviving this thread.
> > >
> > > The following has taken place:
> > >
> > > mapreduce dependency has been moved to client side - no mapreduce job
> > > launched from master or region server.
> > > document patch (HBASE-16574) has been integrated.
> > > Updated mega patch has been attached to HBASE-14123: this covers the
> > > refactor in #1 above and the protobuf 3 merge.
> > >
> > > If community has more feedback on the merge proposal, I would love to
> > hear
> > > it.
> > >
> > > Thanks
> > >
> > > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com>
> > wrote:
> > >
> > > > I'd like to see the docs proposed on HBASE-16574 integrated into our
> > > > project's documentation prior to merge.
> > > >
> > > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > > > > This feature can be marked experimental due to some limitations
> such
> > as
> > > > > security.
> > > > >
> > > > > Your previous round of comments have been addressed.
> > > > > Command line tool has gone through:
> > > > >
> > > > > HBASE-16620 Fix backup command-line tool usability issues
> > > > > HBASE-16655 hbase backup describe with incorrect backup id results
> in
> > > NPE
> > > > >
> > > > > The updated doc has been attached to HBASE-16574.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > Are there more (review) comments ?
> > > > >> >
> > > > >> >
> > > > >> Are outstanding comments addressed?
> > > > >>
> > > > >> I don't see answer to my 'is this experimental/will it be marked
> > > > >> experimental' question.
> > > > >>
> > > > >> I ran into some issues trying to use the feature and suggested
> that
> > a
> > > > >> feature likes this needs polish else it'll just rot, unused. Has
> > > polish
> > > > >> been applied? All ready for another 'user' test? Suggest that you
> > > update
> > > > >> here going forward for the benefit of those trying to follow along
> > and
> > > > who
> > > > >> are not watching JIRA change fly-by.
> > > > >>
> > > > >> It looks like doc got a revision -- I have to check -- to take on
> > > > >> suggestion made above but again, suggest, that this thread gets
> > > updated.
> > > > >>
> > > > >> Thanks,
> > > > >> St.Ack
> > > > >>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Enis Söztutar]

Great work.

So what are the next steps? It seems from the discussion that all
outstanding requests have been addressed. Does the branch ready for merge
or, you have to rebase the branch as well?

Time to start a merge vote?
Enis

On Mon, Oct 10, 2016 at 1:44 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> >> mapreduce dependency has been moved to client side - no mapreduce job
>
> 1. We have no code in the client module anymore, due to dependency on
> internal server API (HFile and WAL access).
> 2. Backup/ restore are client - driven operations, but all the code resides
> in the server module
> 3. No MR in Master, no procedure - driven execution.
> 4. Old good MR from command-line.
> 5. Security was simplified and now only super-user is allowed to run
> backup/restores.
> 6. HBase Backup API was gone due to 1. Now only command-line access to
> backup tools.
>
> These consequences of refactoring has been discussed in HBASE-16727.
>
> -Vlad
>
>
>
>
> On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Reviving this thread.
> >
> > The following has taken place:
> >
> > mapreduce dependency has been moved to client side - no mapreduce job
> > launched from master or region server.
> > document patch (HBASE-16574) has been integrated.
> > Updated mega patch has been attached to HBASE-14123: this covers the
> > refactor in #1 above and the protobuf 3 merge.
> >
> > If community has more feedback on the merge proposal, I would love to
> hear
> > it.
> >
> > Thanks
> >
> > On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com>
> wrote:
> >
> > > I'd like to see the docs proposed on HBASE-16574 integrated into our
> > > project's documentation prior to merge.
> > >
> > > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > > > This feature can be marked experimental due to some limitations such
> as
> > > > security.
> > > >
> > > > Your previous round of comments have been addressed.
> > > > Command line tool has gone through:
> > > >
> > > > HBASE-16620 Fix backup command-line tool usability issues
> > > > HBASE-16655 hbase backup describe with incorrect backup id results in
> > NPE
> > > >
> > > > The updated doc has been attached to HBASE-16574.
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:
> > > >
> > > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> > > >>
> > > >> > Are there more (review) comments ?
> > > >> >
> > > >> >
> > > >> Are outstanding comments addressed?
> > > >>
> > > >> I don't see answer to my 'is this experimental/will it be marked
> > > >> experimental' question.
> > > >>
> > > >> I ran into some issues trying to use the feature and suggested that
> a
> > > >> feature likes this needs polish else it'll just rot, unused. Has
> > polish
> > > >> been applied? All ready for another 'user' test? Suggest that you
> > update
> > > >> here going forward for the benefit of those trying to follow along
> and
> > > who
> > > >> are not watching JIRA change fly-by.
> > > >>
> > > >> It looks like doc got a revision -- I have to check -- to take on
> > > >> suggestion made above but again, suggest, that this thread gets
> > updated.
> > > >>
> > > >> Thanks,
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >> > Thanks
> > > >> >
> > > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <
> d...@hortonworks.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others
> > for
> > > the
> > > >> > > thorough reviews and testing. Thanks Ted and Vlad for taking
> care
> > of
> > > >> the
> > > >> > > feedback. Are we all good to do the merge now? Rather do sooner
> > than
> > > >> > later.
> > > >> > > 
> > > >> > > From: saint@gmail.com <saint.

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> mapreduce dependency has been moved to client side - no mapreduce job

1. We have no code in the client module anymore, due to dependency on
internal server API (HFile and WAL access).
2. Backup/ restore are client - driven operations, but all the code resides
in the server module
3. No MR in Master, no procedure - driven execution.
4. Old good MR from command-line.
5. Security was simplified and now only super-user is allowed to run
backup/restores.
6. HBase Backup API was gone due to 1. Now only command-line access to
backup tools.

These consequences of refactoring has been discussed in HBASE-16727.

-Vlad




On Mon, Oct 10, 2016 at 1:31 PM, Ted Yu <yuzhih...@gmail.com> wrote:

> Reviving this thread.
>
> The following has taken place:
>
> mapreduce dependency has been moved to client side - no mapreduce job
> launched from master or region server.
> document patch (HBASE-16574) has been integrated.
> Updated mega patch has been attached to HBASE-14123: this covers the
> refactor in #1 above and the protobuf 3 merge.
>
> If community has more feedback on the merge proposal, I would love to hear
> it.
>
> Thanks
>
> On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com> wrote:
>
> > I'd like to see the docs proposed on HBASE-16574 integrated into our
> > project's documentation prior to merge.
> >
> > On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > > This feature can be marked experimental due to some limitations such as
> > > security.
> > >
> > > Your previous round of comments have been addressed.
> > > Command line tool has gone through:
> > >
> > > HBASE-16620 Fix backup command-line tool usability issues
> > > HBASE-16655 hbase backup describe with incorrect backup id results in
> NPE
> > >
> > > The updated doc has been attached to HBASE-16574.
> > >
> > > Cheers
> > >
> > > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:
> > >
> > >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > >>
> > >> > Are there more (review) comments ?
> > >> >
> > >> >
> > >> Are outstanding comments addressed?
> > >>
> > >> I don't see answer to my 'is this experimental/will it be marked
> > >> experimental' question.
> > >>
> > >> I ran into some issues trying to use the feature and suggested that a
> > >> feature likes this needs polish else it'll just rot, unused. Has
> polish
> > >> been applied? All ready for another 'user' test? Suggest that you
> update
> > >> here going forward for the benefit of those trying to follow along and
> > who
> > >> are not watching JIRA change fly-by.
> > >>
> > >> It looks like doc got a revision -- I have to check -- to take on
> > >> suggestion made above but again, suggest, that this thread gets
> updated.
> > >>
> > >> Thanks,
> > >> St.Ack
> > >>
> > >>
> > >>
> > >> > Thanks
> > >> >
> > >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com
> >
> > >> > wrote:
> > >> >
> > >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others
> for
> > the
> > >> > > thorough reviews and testing. Thanks Ted and Vlad for taking care
> of
> > >> the
> > >> > > feedback. Are we all good to do the merge now? Rather do sooner
> than
> > >> > later.
> > >> > > 
> > >> > > From: saint@gmail.com <saint@gmail.com> on behalf of
> Stack
> > <
> > >> > > st...@duboce.net>
> > >> > > Sent: Monday, September 12, 2016 1:18 PM
> > >> > > To: HBase Dev List
> > >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch
> HBASE-7912
> > >> > >
> > >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com>
> > wrote:
> > >> > >
> > >> > > > Mega patch (rev 18) is on HBASE-14123.
> > >> > > >
> > >> > > > Please comment on HBASE-14123 on how you want to review.
> > >> > > >
> > >> > >
> > >> > >
> > >> > > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest
&

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Reviving this thread.

The following has taken place:

mapreduce dependency has been moved to client side - no mapreduce job
launched from master or region server.
document patch (HBASE-16574) has been integrated.
Updated mega patch has been attached to HBASE-14123: this covers the
refactor in #1 above and the protobuf 3 merge.

If community has more feedback on the merge proposal, I would love to hear
it.

Thanks

On Thu, Sep 22, 2016 at 10:31 AM, Sean Busbey <bus...@cloudera.com> wrote:

> I'd like to see the docs proposed on HBASE-16574 integrated into our
> project's documentation prior to merge.
>
> On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > This feature can be marked experimental due to some limitations such as
> > security.
> >
> > Your previous round of comments have been addressed.
> > Command line tool has gone through:
> >
> > HBASE-16620 Fix backup command-line tool usability issues
> > HBASE-16655 hbase backup describe with incorrect backup id results in NPE
> >
> > The updated doc has been attached to HBASE-16574.
> >
> > Cheers
> >
> > On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:
> >
> >> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>
> >> > Are there more (review) comments ?
> >> >
> >> >
> >> Are outstanding comments addressed?
> >>
> >> I don't see answer to my 'is this experimental/will it be marked
> >> experimental' question.
> >>
> >> I ran into some issues trying to use the feature and suggested that a
> >> feature likes this needs polish else it'll just rot, unused. Has polish
> >> been applied? All ready for another 'user' test? Suggest that you update
> >> here going forward for the benefit of those trying to follow along and
> who
> >> are not watching JIRA change fly-by.
> >>
> >> It looks like doc got a revision -- I have to check -- to take on
> >> suggestion made above but again, suggest, that this thread gets updated.
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >>
> >> > Thanks
> >> >
> >> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com>
> >> > wrote:
> >> >
> >> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others for
> the
> >> > > thorough reviews and testing. Thanks Ted and Vlad for taking care of
> >> the
> >> > > feedback. Are we all good to do the merge now? Rather do sooner than
> >> > later.
> >> > > 
> >> > > From: saint@gmail.com <saint@gmail.com> on behalf of Stack
> <
> >> > > st...@duboce.net>
> >> > > Sent: Monday, September 12, 2016 1:18 PM
> >> > > To: HBase Dev List
> >> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
> >> > >
> >> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> >> > >
> >> > > > Mega patch (rev 18) is on HBASE-14123.
> >> > > >
> >> > > > Please comment on HBASE-14123 on how you want to review.
> >> > > >
> >> > >
> >> > >
> >> > > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest
> updating
> >> > it.
> >> > > RB is pretty good for review. Patch is only 1.5M so should be fine.
> >> > >
> >> > > St.Ack
> >> > >
> >> > >
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
> >> > > >
> >> > > > > On review of the 'patch', do I just compare the branch to
> master or
> >> > is
> >> > > > > there a megapatch posted somewhere (I think I saw one but it
> seemed
> >> > > stale
> >> > > > > and then I 'lost' the tab). Sorry for dumb question.
> >> > > > > St.Ack
> >> > > > >
> >> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net>
> wrote:
> >> > > > >
> >> > > > > > Late to the game. A few comments after rereading this thread
> as a
> >> > > > 'user'.
> >> > > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Sean Busbey]

I'd like to see the docs proposed on HBASE-16574 integrated into our
project's documentation prior to merge.

On Thu, Sep 22, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> This feature can be marked experimental due to some limitations such as
> security.
>
> Your previous round of comments have been addressed.
> Command line tool has gone through:
>
> HBASE-16620 Fix backup command-line tool usability issues
> HBASE-16655 hbase backup describe with incorrect backup id results in NPE
>
> The updated doc has been attached to HBASE-16574.
>
> Cheers
>
> On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:
>
>> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>> > Are there more (review) comments ?
>> >
>> >
>> Are outstanding comments addressed?
>>
>> I don't see answer to my 'is this experimental/will it be marked
>> experimental' question.
>>
>> I ran into some issues trying to use the feature and suggested that a
>> feature likes this needs polish else it'll just rot, unused. Has polish
>> been applied? All ready for another 'user' test? Suggest that you update
>> here going forward for the benefit of those trying to follow along and who
>> are not watching JIRA change fly-by.
>>
>> It looks like doc got a revision -- I have to check -- to take on
>> suggestion made above but again, suggest, that this thread gets updated.
>>
>> Thanks,
>> St.Ack
>>
>>
>>
>> > Thanks
>> >
>> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com>
>> > wrote:
>> >
>> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others for the
>> > > thorough reviews and testing. Thanks Ted and Vlad for taking care of
>> the
>> > > feedback. Are we all good to do the merge now? Rather do sooner than
>> > later.
>> > > 
>> > > From: saint@gmail.com <saint@gmail.com> on behalf of Stack <
>> > > st...@duboce.net>
>> > > Sent: Monday, September 12, 2016 1:18 PM
>> > > To: HBase Dev List
>> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
>> > >
>> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > >
>> > > > Mega patch (rev 18) is on HBASE-14123.
>> > > >
>> > > > Please comment on HBASE-14123 on how you want to review.
>> > > >
>> > >
>> > >
>> > > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating
>> > it.
>> > > RB is pretty good for review. Patch is only 1.5M so should be fine.
>> > >
>> > > St.Ack
>> > >
>> > >
>> > > >
>> > > > Thanks
>> > > >
>> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
>> > > >
>> > > > > On review of the 'patch', do I just compare the branch to master or
>> > is
>> > > > > there a megapatch posted somewhere (I think I saw one but it seemed
>> > > stale
>> > > > > and then I 'lost' the tab). Sorry for dumb question.
>> > > > > St.Ack
>> > > > >
>> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> wrote:
>> > > > >
>> > > > > > Late to the game. A few comments after rereading this thread as a
>> > > > 'user'.
>> > > > > >
>> > > > > > + Before merge, a user-facing feature like this should work (If
>> > this
>> > > is
>> > > > > "higher-bar
>> > > > > > for new features", bring it on -- smile).
>> > > > > > + As a user, I tried the branch with tools after reviewing the
>> > > > > just-posted
>> > > > > > doc. I had an 'interesting' experience (left comments up on
>> > issue). I
>> > > > > think
>> > > > > > the tooling/doc. important to get right. If it breaks easily or
>> is
>> > > > > > inconsistent (or lacks 'polish'), operators will judge the whole
>> > > > > > backup/restore tooling chain as not trustworthy and abandon it.
>> > Lets
>> > > > not
>> > > >

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

This feature can be marked experimental due to some limitations such as
security.

Your previous round of comments have been addressed.
Command line tool has gone through:

HBASE-16620 Fix backup command-line tool usability issues
HBASE-16655 hbase backup describe with incorrect backup id results in NPE

The updated doc has been attached to HBASE-16574.

Cheers

On Thu, Sep 22, 2016 at 8:53 AM, Stack <st...@duboce.net> wrote:

> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Are there more (review) comments ?
> >
> >
> Are outstanding comments addressed?
>
> I don't see answer to my 'is this experimental/will it be marked
> experimental' question.
>
> I ran into some issues trying to use the feature and suggested that a
> feature likes this needs polish else it'll just rot, unused. Has polish
> been applied? All ready for another 'user' test? Suggest that you update
> here going forward for the benefit of those trying to follow along and who
> are not watching JIRA change fly-by.
>
> It looks like doc got a revision -- I have to check -- to take on
> suggestion made above but again, suggest, that this thread gets updated.
>
> Thanks,
> St.Ack
>
>
>
> > Thanks
> >
> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com>
> > wrote:
> >
> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others for the
> > > thorough reviews and testing. Thanks Ted and Vlad for taking care of
> the
> > > feedback. Are we all good to do the merge now? Rather do sooner than
> > later.
> > > 
> > > From: saint....@gmail.com <saint....@gmail.com> on behalf of Stack <
> > > st...@duboce.net>
> > > Sent: Monday, September 12, 2016 1:18 PM
> > > To: HBase Dev List
> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
> > >
> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > >
> > > > Mega patch (rev 18) is on HBASE-14123.
> > > >
> > > > Please comment on HBASE-14123 on how you want to review.
> > > >
> > >
> > >
> > > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating
> > it.
> > > RB is pretty good for review. Patch is only 1.5M so should be fine.
> > >
> > > St.Ack
> > >
> > >
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On review of the 'patch', do I just compare the branch to master or
> > is
> > > > > there a megapatch posted somewhere (I think I saw one but it seemed
> > > stale
> > > > > and then I 'lost' the tab). Sorry for dumb question.
> > > > > St.Ack
> > > > >
> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > Late to the game. A few comments after rereading this thread as a
> > > > 'user'.
> > > > > >
> > > > > > + Before merge, a user-facing feature like this should work (If
> > this
> > > is
> > > > > "higher-bar
> > > > > > for new features", bring it on -- smile).
> > > > > > + As a user, I tried the branch with tools after reviewing the
> > > > > just-posted
> > > > > > doc. I had an 'interesting' experience (left comments up on
> > issue). I
> > > > > think
> > > > > > the tooling/doc. important to get right. If it breaks easily or
> is
> > > > > > inconsistent (or lacks 'polish'), operators will judge the whole
> > > > > > backup/restore tooling chain as not trustworthy and abandon it.
> > Lets
> > > > not
> > > > > > have this happen to this feature.
> > > > > > + Matteo's suggestion (with a helpful starter list) that there
> > needs
> > > to
> > > > > be
> > > > > > explicit qualification on what is actually being delivered --
> > > > including a
> > > > > > listing of limitations (some look serious such as data bleed from
> > > other
> > > > > > regions in WALs, but maybe I don't care for my use case...) --
> > needs
> > > to
> > > > > > accompany the merge. Lets fold them into the user doc. in the
> > > te

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> wrote:

> Are there more (review) comments ?
>
>
Are outstanding comments addressed?

I don't see answer to my 'is this experimental/will it be marked
experimental' question.

I ran into some issues trying to use the feature and suggested that a
feature likes this needs polish else it'll just rot, unused. Has polish
been applied? All ready for another 'user' test? Suggest that you update
here going forward for the benefit of those trying to follow along and who
are not watching JIRA change fly-by.

It looks like doc got a revision -- I have to check -- to take on
suggestion made above but again, suggest, that this thread gets updated.

Thanks,
St.Ack



> Thanks
>
> On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com>
> wrote:
>
> > Just reviving this thread. Thanks Sean, Stack, Dima, and others for the
> > thorough reviews and testing. Thanks Ted and Vlad for taking care of the
> > feedback. Are we all good to do the merge now? Rather do sooner than
> later.
> > 
> > From: saint@gmail.com <saint@gmail.com> on behalf of Stack <
> > st...@duboce.net>
> > Sent: Monday, September 12, 2016 1:18 PM
> > To: HBase Dev List
> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
> >
> > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > Mega patch (rev 18) is on HBASE-14123.
> > >
> > > Please comment on HBASE-14123 on how you want to review.
> > >
> >
> >
> > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating
> it.
> > RB is pretty good for review. Patch is only 1.5M so should be fine.
> >
> > St.Ack
> >
> >
> > >
> > > Thanks
> > >
> > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > On review of the 'patch', do I just compare the branch to master or
> is
> > > > there a megapatch posted somewhere (I think I saw one but it seemed
> > stale
> > > > and then I 'lost' the tab). Sorry for dumb question.
> > > > St.Ack
> > > >
> > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > Late to the game. A few comments after rereading this thread as a
> > > 'user'.
> > > > >
> > > > > + Before merge, a user-facing feature like this should work (If
> this
> > is
> > > > "higher-bar
> > > > > for new features", bring it on -- smile).
> > > > > + As a user, I tried the branch with tools after reviewing the
> > > > just-posted
> > > > > doc. I had an 'interesting' experience (left comments up on
> issue). I
> > > > think
> > > > > the tooling/doc. important to get right. If it breaks easily or is
> > > > > inconsistent (or lacks 'polish'), operators will judge the whole
> > > > > backup/restore tooling chain as not trustworthy and abandon it.
> Lets
> > > not
> > > > > have this happen to this feature.
> > > > > + Matteo's suggestion (with a helpful starter list) that there
> needs
> > to
> > > > be
> > > > > explicit qualification on what is actually being delivered --
> > > including a
> > > > > listing of limitations (some look serious such as data bleed from
> > other
> > > > > regions in WALs, but maybe I don't care for my use case...) --
> needs
> > to
> > > > > accompany the merge. Lets fold them into the user doc. in the
> > technical
> > > > > overview area as suggested so user expectations are properly
> managed
> > > > > (otherwise, they expect the world and will just give up when we
> fall
> > > > > short). Vladimir did a list of what is in each of the phases above
> > > which
> > > > > would serve as a good start.
> > > > > + Is this feature 'experimental' (Matteo asks above). I'd prefer it
> > is
> > > > > not. If it is, it should be labelled all over that it is so. I see
> > > > current
> > > > > state called out as a '... technical preview feature'. Does this
> mean
> > > > > not-for-users?
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > &

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

That's the correct URL.

Just uploaded patch v21 on HBASE-14123 to review board.

On Wed, Sep 21, 2016 at 10:16 AM, Matteo Bertozzi <theo.berto...@gmail.com>
wrote:

> let me do a "mega patch" review pass.
> Is this the latest? https://reviews.apache.org/r/51823/
>
> Matteo
>
>
> On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Are there more (review) comments ?
> >
> > Thanks
> >
> > On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com>
> > wrote:
> >
> > > Just reviving this thread. Thanks Sean, Stack, Dima, and others for the
> > > thorough reviews and testing. Thanks Ted and Vlad for taking care of
> the
> > > feedback. Are we all good to do the merge now? Rather do sooner than
> > later.
> > > 
> > > From: saint@gmail.com <saint....@gmail.com> on behalf of Stack <
> > > st...@duboce.net>
> > > Sent: Monday, September 12, 2016 1:18 PM
> > > To: HBase Dev List
> > > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
> > >
> > > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> > >
> > > > Mega patch (rev 18) is on HBASE-14123.
> > > >
> > > > Please comment on HBASE-14123 on how you want to review.
> > > >
> > >
> > >
> > > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating
> > it.
> > > RB is pretty good for review. Patch is only 1.5M so should be fine.
> > >
> > > St.Ack
> > >
> > >
> > > >
> > > > Thanks
> > > >
> > > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > On review of the 'patch', do I just compare the branch to master or
> > is
> > > > > there a megapatch posted somewhere (I think I saw one but it seemed
> > > stale
> > > > > and then I 'lost' the tab). Sorry for dumb question.
> > > > > St.Ack
> > > > >
> > > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > > > >
> > > > > > Late to the game. A few comments after rereading this thread as a
> > > > 'user'.
> > > > > >
> > > > > > + Before merge, a user-facing feature like this should work (If
> > this
> > > is
> > > > > "higher-bar
> > > > > > for new features", bring it on -- smile).
> > > > > > + As a user, I tried the branch with tools after reviewing the
> > > > > just-posted
> > > > > > doc. I had an 'interesting' experience (left comments up on
> > issue). I
> > > > > think
> > > > > > the tooling/doc. important to get right. If it breaks easily or
> is
> > > > > > inconsistent (or lacks 'polish'), operators will judge the whole
> > > > > > backup/restore tooling chain as not trustworthy and abandon it.
> > Lets
> > > > not
> > > > > > have this happen to this feature.
> > > > > > + Matteo's suggestion (with a helpful starter list) that there
> > needs
> > > to
> > > > > be
> > > > > > explicit qualification on what is actually being delivered --
> > > > including a
> > > > > > listing of limitations (some look serious such as data bleed from
> > > other
> > > > > > regions in WALs, but maybe I don't care for my use case...) --
> > needs
> > > to
> > > > > > accompany the merge. Lets fold them into the user doc. in the
> > > technical
> > > > > > overview area as suggested so user expectations are properly
> > managed
> > > > > > (otherwise, they expect the world and will just give up when we
> > fall
> > > > > > short). Vladimir did a list of what is in each of the phases
> above
> > > > which
> > > > > > would serve as a good start.
> > > > > > + Is this feature 'experimental' (Matteo asks above). I'd prefer
> it
> > > is
> > > > > > not. If it is, it should be labelled all over that it is so. I
> see
> > > > > current
> > > > > > state called out as a '... technical preview feature'. Does this
> > mean
> > > > > > not-for-users?
> >

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Matteo Bertozzi]

let me do a "mega patch" review pass.
Is this the latest? https://reviews.apache.org/r/51823/

Matteo


On Wed, Sep 21, 2016 at 7:43 AM, Ted Yu <yuzhih...@gmail.com> wrote:

> Are there more (review) comments ?
>
> Thanks
>
> On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com>
> wrote:
>
> > Just reviving this thread. Thanks Sean, Stack, Dima, and others for the
> > thorough reviews and testing. Thanks Ted and Vlad for taking care of the
> > feedback. Are we all good to do the merge now? Rather do sooner than
> later.
> > 
> > From: saint@gmail.com <saint@gmail.com> on behalf of Stack <
> > st...@duboce.net>
> > Sent: Monday, September 12, 2016 1:18 PM
> > To: HBase Dev List
> > Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
> >
> > On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > Mega patch (rev 18) is on HBASE-14123.
> > >
> > > Please comment on HBASE-14123 on how you want to review.
> > >
> >
> >
> > Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating
> it.
> > RB is pretty good for review. Patch is only 1.5M so should be fine.
> >
> > St.Ack
> >
> >
> > >
> > > Thanks
> > >
> > > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > On review of the 'patch', do I just compare the branch to master or
> is
> > > > there a megapatch posted somewhere (I think I saw one but it seemed
> > stale
> > > > and then I 'lost' the tab). Sorry for dumb question.
> > > > St.Ack
> > > >
> > > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > > > Late to the game. A few comments after rereading this thread as a
> > > 'user'.
> > > > >
> > > > > + Before merge, a user-facing feature like this should work (If
> this
> > is
> > > > "higher-bar
> > > > > for new features", bring it on -- smile).
> > > > > + As a user, I tried the branch with tools after reviewing the
> > > > just-posted
> > > > > doc. I had an 'interesting' experience (left comments up on
> issue). I
> > > > think
> > > > > the tooling/doc. important to get right. If it breaks easily or is
> > > > > inconsistent (or lacks 'polish'), operators will judge the whole
> > > > > backup/restore tooling chain as not trustworthy and abandon it.
> Lets
> > > not
> > > > > have this happen to this feature.
> > > > > + Matteo's suggestion (with a helpful starter list) that there
> needs
> > to
> > > > be
> > > > > explicit qualification on what is actually being delivered --
> > > including a
> > > > > listing of limitations (some look serious such as data bleed from
> > other
> > > > > regions in WALs, but maybe I don't care for my use case...) --
> needs
> > to
> > > > > accompany the merge. Lets fold them into the user doc. in the
> > technical
> > > > > overview area as suggested so user expectations are properly
> managed
> > > > > (otherwise, they expect the world and will just give up when we
> fall
> > > > > short). Vladimir did a list of what is in each of the phases above
> > > which
> > > > > would serve as a good start.
> > > > > + Is this feature 'experimental' (Matteo asks above). I'd prefer it
> > is
> > > > > not. If it is, it should be labelled all over that it is so. I see
> > > > current
> > > > > state called out as a '... technical preview feature'. Does this
> mean
> > > > > not-for-users?
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu <yuzhih...@gmail.com>
> wrote:
> > > > >
> > > > >> Sean:
> > > > >> Do you have more comments ?
> > > > >>
> > > > >> Cheers
> > > > >>
> > > > >> On Fri, Sep 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Are there more (review) comments ?

Thanks

On Tue, Sep 20, 2016 at 10:02 AM, Devaraj Das <d...@hortonworks.com> wrote:

> Just reviving this thread. Thanks Sean, Stack, Dima, and others for the
> thorough reviews and testing. Thanks Ted and Vlad for taking care of the
> feedback. Are we all good to do the merge now? Rather do sooner than later.
> 
> From: saint@gmail.com <saint@gmail.com> on behalf of Stack <
> st...@duboce.net>
> Sent: Monday, September 12, 2016 1:18 PM
> To: HBase Dev List
> Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912
>
> On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Mega patch (rev 18) is on HBASE-14123.
> >
> > Please comment on HBASE-14123 on how you want to review.
> >
>
>
> Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating it.
> RB is pretty good for review. Patch is only 1.5M so should be fine.
>
> St.Ack
>
>
> >
> > Thanks
> >
> > On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
> >
> > > On review of the 'patch', do I just compare the branch to master or is
> > > there a megapatch posted somewhere (I think I saw one but it seemed
> stale
> > > and then I 'lost' the tab). Sorry for dumb question.
> > > St.Ack
> > >
> > > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> wrote:
> > >
> > > > Late to the game. A few comments after rereading this thread as a
> > 'user'.
> > > >
> > > > + Before merge, a user-facing feature like this should work (If this
> is
> > > "higher-bar
> > > > for new features", bring it on -- smile).
> > > > + As a user, I tried the branch with tools after reviewing the
> > > just-posted
> > > > doc. I had an 'interesting' experience (left comments up on issue). I
> > > think
> > > > the tooling/doc. important to get right. If it breaks easily or is
> > > > inconsistent (or lacks 'polish'), operators will judge the whole
> > > > backup/restore tooling chain as not trustworthy and abandon it. Lets
> > not
> > > > have this happen to this feature.
> > > > + Matteo's suggestion (with a helpful starter list) that there needs
> to
> > > be
> > > > explicit qualification on what is actually being delivered --
> > including a
> > > > listing of limitations (some look serious such as data bleed from
> other
> > > > regions in WALs, but maybe I don't care for my use case...) -- needs
> to
> > > > accompany the merge. Lets fold them into the user doc. in the
> technical
> > > > overview area as suggested so user expectations are properly managed
> > > > (otherwise, they expect the world and will just give up when we fall
> > > > short). Vladimir did a list of what is in each of the phases above
> > which
> > > > would serve as a good start.
> > > > + Is this feature 'experimental' (Matteo asks above). I'd prefer it
> is
> > > > not. If it is, it should be labelled all over that it is so. I see
> > > current
> > > > state called out as a '... technical preview feature'. Does this mean
> > > > not-for-users?
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > > >
> > > >> Sean:
> > > >> Do you have more comments ?
> > > >>
> > > >> Cheers
> > > >>
> > > >> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov <
> > > vladrodio...@gmail.com
> > > >> >
> > > >> wrote:
> > > >>
> > > >> > Sean,
> > > >> >
> > > >> > Backup/Restore can fail due to various reasons: network outage
> > > (cluster
> > > >> > wide), various time-outs in HBase and HDFS layer, M/R failure due
> to
> > > >> "HDFS
> > > >> > exceeded quota", user error (manual deletion of data) and so on so
> > on.
> > > >> That
> > > >> > is impossible to enumerate all possible types of failures in a
> > > >> distributed

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Devaraj Das]

Just reviving this thread. Thanks Sean, Stack, Dima, and others for the 
thorough reviews and testing. Thanks Ted and Vlad for taking care of the 
feedback. Are we all good to do the merge now? Rather do sooner than later.

From: saint@gmail.com <saint@gmail.com> on behalf of Stack 
<st...@duboce.net>
Sent: Monday, September 12, 2016 1:18 PM
To: HBase Dev List
Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu <yuzhih...@gmail.com> wrote:

> Mega patch (rev 18) is on HBASE-14123.
>
> Please comment on HBASE-14123 on how you want to review.
>


Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating it.
RB is pretty good for review. Patch is only 1.5M so should be fine.

St.Ack


>
> Thanks
>
> On Mon, Sep 12, 2016 at 12:15 PM, Stack <st...@duboce.net> wrote:
>
> > On review of the 'patch', do I just compare the branch to master or is
> > there a megapatch posted somewhere (I think I saw one but it seemed stale
> > and then I 'lost' the tab). Sorry for dumb question.
> > St.Ack
> >
> > On Mon, Sep 12, 2016 at 12:01 PM, Stack <st...@duboce.net> wrote:
> >
> > > Late to the game. A few comments after rereading this thread as a
> 'user'.
> > >
> > > + Before merge, a user-facing feature like this should work (If this is
> > "higher-bar
> > > for new features", bring it on -- smile).
> > > + As a user, I tried the branch with tools after reviewing the
> > just-posted
> > > doc. I had an 'interesting' experience (left comments up on issue). I
> > think
> > > the tooling/doc. important to get right. If it breaks easily or is
> > > inconsistent (or lacks 'polish'), operators will judge the whole
> > > backup/restore tooling chain as not trustworthy and abandon it. Lets
> not
> > > have this happen to this feature.
> > > + Matteo's suggestion (with a helpful starter list) that there needs to
> > be
> > > explicit qualification on what is actually being delivered --
> including a
> > > listing of limitations (some look serious such as data bleed from other
> > > regions in WALs, but maybe I don't care for my use case...) -- needs to
> > > accompany the merge. Lets fold them into the user doc. in the technical
> > > overview area as suggested so user expectations are properly managed
> > > (otherwise, they expect the world and will just give up when we fall
> > > short). Vladimir did a list of what is in each of the phases above
> which
> > > would serve as a good start.
> > > + Is this feature 'experimental' (Matteo asks above). I'd prefer it is
> > > not. If it is, it should be labelled all over that it is so. I see
> > current
> > > state called out as a '... technical preview feature'. Does this mean
> > > not-for-users?
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > >
> > >> Sean:
> > >> Do you have more comments ?
> > >>
> > >> Cheers
> > >>
> > >> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov <
> > vladrodio...@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > Sean,
> > >> >
> > >> > Backup/Restore can fail due to various reasons: network outage
> > (cluster
> > >> > wide), various time-outs in HBase and HDFS layer, M/R failure due to
> > >> "HDFS
> > >> > exceeded quota", user error (manual deletion of data) and so on so
> on.
> > >> That
> > >> > is impossible to enumerate all possible types of failures in a
> > >> distributed
> > >> > system - that is not our goal/task.
> > >> >
> > >> > We focus completely on backup system table consistency in a presence
> > of
> > >> any
> > >> > type of failure. That is what I call "tolerance to failures".
> > >> >
> > >> > On a failure:
> > >> >
> > >> > BACKUP. All backup system information (prior to backup) will be
> > restored
> > >> > and all temporary data, related to a failed session, in HDFS will be
> > >> > deleted
> > >> > RESTORE. We do

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

On Mon, Sep 12, 2016 at 12:19 PM, Ted Yu  wrote:

> Mega patch (rev 18) is on HBASE-14123.
>
> Please comment on HBASE-14123 on how you want to review.
>


Yeah. That was my lost tab. Last rb was 6 months ago. Suggest updating it.
RB is pretty good for review. Patch is only 1.5M so should be fine.

St.Ack


>
> Thanks
>
> On Mon, Sep 12, 2016 at 12:15 PM, Stack  wrote:
>
> > On review of the 'patch', do I just compare the branch to master or is
> > there a megapatch posted somewhere (I think I saw one but it seemed stale
> > and then I 'lost' the tab). Sorry for dumb question.
> > St.Ack
> >
> > On Mon, Sep 12, 2016 at 12:01 PM, Stack  wrote:
> >
> > > Late to the game. A few comments after rereading this thread as a
> 'user'.
> > >
> > > + Before merge, a user-facing feature like this should work (If this is
> > "higher-bar
> > > for new features", bring it on -- smile).
> > > + As a user, I tried the branch with tools after reviewing the
> > just-posted
> > > doc. I had an 'interesting' experience (left comments up on issue). I
> > think
> > > the tooling/doc. important to get right. If it breaks easily or is
> > > inconsistent (or lacks 'polish'), operators will judge the whole
> > > backup/restore tooling chain as not trustworthy and abandon it. Lets
> not
> > > have this happen to this feature.
> > > + Matteo's suggestion (with a helpful starter list) that there needs to
> > be
> > > explicit qualification on what is actually being delivered --
> including a
> > > listing of limitations (some look serious such as data bleed from other
> > > regions in WALs, but maybe I don't care for my use case...) -- needs to
> > > accompany the merge. Lets fold them into the user doc. in the technical
> > > overview area as suggested so user expectations are properly managed
> > > (otherwise, they expect the world and will just give up when we fall
> > > short). Vladimir did a list of what is in each of the phases above
> which
> > > would serve as a good start.
> > > + Is this feature 'experimental' (Matteo asks above). I'd prefer it is
> > > not. If it is, it should be labelled all over that it is so. I see
> > current
> > > state called out as a '... technical preview feature'. Does this mean
> > > not-for-users?
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu  wrote:
> > >
> > >> Sean:
> > >> Do you have more comments ?
> > >>
> > >> Cheers
> > >>
> > >> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov <
> > vladrodio...@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > Sean,
> > >> >
> > >> > Backup/Restore can fail due to various reasons: network outage
> > (cluster
> > >> > wide), various time-outs in HBase and HDFS layer, M/R failure due to
> > >> "HDFS
> > >> > exceeded quota", user error (manual deletion of data) and so on so
> on.
> > >> That
> > >> > is impossible to enumerate all possible types of failures in a
> > >> distributed
> > >> > system - that is not our goal/task.
> > >> >
> > >> > We focus completely on backup system table consistency in a presence
> > of
> > >> any
> > >> > type of failure. That is what I call "tolerance to failures".
> > >> >
> > >> > On a failure:
> > >> >
> > >> > BACKUP. All backup system information (prior to backup) will be
> > restored
> > >> > and all temporary data, related to a failed session, in HDFS will be
> > >> > deleted
> > >> > RESTORE. We do not care about system data, because restore does not
> > >> change
> > >> > it. Temporary data in HDFS will be cleaned up and table will be in a
> > >> state
> > >> > back to where it was before operation started.
> > >> >
> > >> > This is what user should expect in case of a failure.
> > >> >
> > >> > -Vlad
> > >> >
> > >> >
> > >> > -Vlad
> > >> >
> > >> > On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey 
> > wrote:
> > >> >
> > >> > > Failing in a consistent way, with docs that explain the various
> > >> > > expected failures would be sufficient.
> > >> > >
> > >> > > On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov
> > >> > >  wrote:
> > >> > > > Do not worry Sean, doc is coming today as a preview and our
> writer
> > >> > Frank
> > >> > > > will be working on a putting  it into Apache repo. Timeline
> > depends
> > >> on
> > >> > > > Franks schedule but I hope we will get it rather sooner than
> > later.
> > >> > > >
> > >> > > > As for failure testing, we are focusing only on a consistent
> state
> > >> of
> > >> > > > backup system data in a presence of any type of failures, We are
> > not
> > >> > > going
> > >> > > > to implement  anything more "fancy", than that. We allow both:
> > >> backup
> > >> > and
> > >> > > > restore to fail. What we do not allow is to have system data
> > >> corrupted.
> > >> > > > Will it suffice for you? Do you have any other concerns, you
> want
> > >> us to
> > >> > > > address?
> 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

On review of the 'patch', do I just compare the branch to master or is
there a megapatch posted somewhere (I think I saw one but it seemed stale
and then I 'lost' the tab). Sorry for dumb question.
St.Ack

On Mon, Sep 12, 2016 at 12:01 PM, Stack  wrote:

> Late to the game. A few comments after rereading this thread as a 'user'.
>
> + Before merge, a user-facing feature like this should work (If this is 
> "higher-bar
> for new features", bring it on -- smile).
> + As a user, I tried the branch with tools after reviewing the just-posted
> doc. I had an 'interesting' experience (left comments up on issue). I think
> the tooling/doc. important to get right. If it breaks easily or is
> inconsistent (or lacks 'polish'), operators will judge the whole
> backup/restore tooling chain as not trustworthy and abandon it. Lets not
> have this happen to this feature.
> + Matteo's suggestion (with a helpful starter list) that there needs to be
> explicit qualification on what is actually being delivered -- including a
> listing of limitations (some look serious such as data bleed from other
> regions in WALs, but maybe I don't care for my use case...) -- needs to
> accompany the merge. Lets fold them into the user doc. in the technical
> overview area as suggested so user expectations are properly managed
> (otherwise, they expect the world and will just give up when we fall
> short). Vladimir did a list of what is in each of the phases above which
> would serve as a good start.
> + Is this feature 'experimental' (Matteo asks above). I'd prefer it is
> not. If it is, it should be labelled all over that it is so. I see current
> state called out as a '... technical preview feature'. Does this mean
> not-for-users?
>
> St.Ack
>
>
>
>
>
>
>
>
>
>
>
> On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu  wrote:
>
>> Sean:
>> Do you have more comments ?
>>
>> Cheers
>>
>> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov > >
>> wrote:
>>
>> > Sean,
>> >
>> > Backup/Restore can fail due to various reasons: network outage (cluster
>> > wide), various time-outs in HBase and HDFS layer, M/R failure due to
>> "HDFS
>> > exceeded quota", user error (manual deletion of data) and so on so on.
>> That
>> > is impossible to enumerate all possible types of failures in a
>> distributed
>> > system - that is not our goal/task.
>> >
>> > We focus completely on backup system table consistency in a presence of
>> any
>> > type of failure. That is what I call "tolerance to failures".
>> >
>> > On a failure:
>> >
>> > BACKUP. All backup system information (prior to backup) will be restored
>> > and all temporary data, related to a failed session, in HDFS will be
>> > deleted
>> > RESTORE. We do not care about system data, because restore does not
>> change
>> > it. Temporary data in HDFS will be cleaned up and table will be in a
>> state
>> > back to where it was before operation started.
>> >
>> > This is what user should expect in case of a failure.
>> >
>> > -Vlad
>> >
>> >
>> > -Vlad
>> >
>> > On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey  wrote:
>> >
>> > > Failing in a consistent way, with docs that explain the various
>> > > expected failures would be sufficient.
>> > >
>> > > On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov
>> > >  wrote:
>> > > > Do not worry Sean, doc is coming today as a preview and our writer
>> > Frank
>> > > > will be working on a putting  it into Apache repo. Timeline depends
>> on
>> > > > Franks schedule but I hope we will get it rather sooner than later.
>> > > >
>> > > > As for failure testing, we are focusing only on a consistent state
>> of
>> > > > backup system data in a presence of any type of failures, We are not
>> > > going
>> > > > to implement  anything more "fancy", than that. We allow both:
>> backup
>> > and
>> > > > restore to fail. What we do not allow is to have system data
>> corrupted.
>> > > > Will it suffice for you? Do you have any other concerns, you want
>> us to
>> > > > address?
>> > > >
>> > > > -Vlad
>> > > >
>> > > >
>> > > > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey 
>> > wrote:
>> > > >
>> > > >> "docs will come to Apache soon" does not address my concern around
>> > docs
>> > > at
>> > > >> all, unless said docs have already made it into the project repo. I
>> > > don't
>> > > >> want third party resources for using a major and important feature
>> of
>> > > the
>> > > >> project, I want us to provide end users with what they need to get
>> the
>> > > job
>> > > >> done.
>> > > >>
>> > > >> I see some calls for patience on the failure testing, but the
>> appeal
>> > to
>> > > us
>> > > >> having done a bad job of requiring proper tests of previous
>> features
>> > > just
>> > > >> makes me more concerned about not getting them here. I don't want
>> to
>> > set
>> > > >> yet another bad example that will then be pointed to in the future.
>> > > >>
>> > > >> On 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Mega patch (rev 18) is on HBASE-14123.

Please comment on HBASE-14123 on how you want to review.

Thanks

On Mon, Sep 12, 2016 at 12:15 PM, Stack  wrote:

> On review of the 'patch', do I just compare the branch to master or is
> there a megapatch posted somewhere (I think I saw one but it seemed stale
> and then I 'lost' the tab). Sorry for dumb question.
> St.Ack
>
> On Mon, Sep 12, 2016 at 12:01 PM, Stack  wrote:
>
> > Late to the game. A few comments after rereading this thread as a 'user'.
> >
> > + Before merge, a user-facing feature like this should work (If this is
> "higher-bar
> > for new features", bring it on -- smile).
> > + As a user, I tried the branch with tools after reviewing the
> just-posted
> > doc. I had an 'interesting' experience (left comments up on issue). I
> think
> > the tooling/doc. important to get right. If it breaks easily or is
> > inconsistent (or lacks 'polish'), operators will judge the whole
> > backup/restore tooling chain as not trustworthy and abandon it. Lets not
> > have this happen to this feature.
> > + Matteo's suggestion (with a helpful starter list) that there needs to
> be
> > explicit qualification on what is actually being delivered -- including a
> > listing of limitations (some look serious such as data bleed from other
> > regions in WALs, but maybe I don't care for my use case...) -- needs to
> > accompany the merge. Lets fold them into the user doc. in the technical
> > overview area as suggested so user expectations are properly managed
> > (otherwise, they expect the world and will just give up when we fall
> > short). Vladimir did a list of what is in each of the phases above which
> > would serve as a good start.
> > + Is this feature 'experimental' (Matteo asks above). I'd prefer it is
> > not. If it is, it should be labelled all over that it is so. I see
> current
> > state called out as a '... technical preview feature'. Does this mean
> > not-for-users?
> >
> > St.Ack
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu  wrote:
> >
> >> Sean:
> >> Do you have more comments ?
> >>
> >> Cheers
> >>
> >> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov <
> vladrodio...@gmail.com
> >> >
> >> wrote:
> >>
> >> > Sean,
> >> >
> >> > Backup/Restore can fail due to various reasons: network outage
> (cluster
> >> > wide), various time-outs in HBase and HDFS layer, M/R failure due to
> >> "HDFS
> >> > exceeded quota", user error (manual deletion of data) and so on so on.
> >> That
> >> > is impossible to enumerate all possible types of failures in a
> >> distributed
> >> > system - that is not our goal/task.
> >> >
> >> > We focus completely on backup system table consistency in a presence
> of
> >> any
> >> > type of failure. That is what I call "tolerance to failures".
> >> >
> >> > On a failure:
> >> >
> >> > BACKUP. All backup system information (prior to backup) will be
> restored
> >> > and all temporary data, related to a failed session, in HDFS will be
> >> > deleted
> >> > RESTORE. We do not care about system data, because restore does not
> >> change
> >> > it. Temporary data in HDFS will be cleaned up and table will be in a
> >> state
> >> > back to where it was before operation started.
> >> >
> >> > This is what user should expect in case of a failure.
> >> >
> >> > -Vlad
> >> >
> >> >
> >> > -Vlad
> >> >
> >> > On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey 
> wrote:
> >> >
> >> > > Failing in a consistent way, with docs that explain the various
> >> > > expected failures would be sufficient.
> >> > >
> >> > > On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov
> >> > >  wrote:
> >> > > > Do not worry Sean, doc is coming today as a preview and our writer
> >> > Frank
> >> > > > will be working on a putting  it into Apache repo. Timeline
> depends
> >> on
> >> > > > Franks schedule but I hope we will get it rather sooner than
> later.
> >> > > >
> >> > > > As for failure testing, we are focusing only on a consistent state
> >> of
> >> > > > backup system data in a presence of any type of failures, We are
> not
> >> > > going
> >> > > > to implement  anything more "fancy", than that. We allow both:
> >> backup
> >> > and
> >> > > > restore to fail. What we do not allow is to have system data
> >> corrupted.
> >> > > > Will it suffice for you? Do you have any other concerns, you want
> >> us to
> >> > > > address?
> >> > > >
> >> > > > -Vlad
> >> > > >
> >> > > >
> >> > > > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey 
> >> > wrote:
> >> > > >
> >> > > >> "docs will come to Apache soon" does not address my concern
> around
> >> > docs
> >> > > at
> >> > > >> all, unless said docs have already made it into the project
> repo. I
> >> > > don't
> >> > > >> want third party resources for using a major and important
> feature
> >> of
> >> > > the
> >> > > >> project, I want us to provide end users 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Stack]

Late to the game. A few comments after rereading this thread as a 'user'.

+ Before merge, a user-facing feature like this should work (If this
is "higher-bar
for new features", bring it on -- smile).
+ As a user, I tried the branch with tools after reviewing the just-posted
doc. I had an 'interesting' experience (left comments up on issue). I think
the tooling/doc. important to get right. If it breaks easily or is
inconsistent (or lacks 'polish'), operators will judge the whole
backup/restore tooling chain as not trustworthy and abandon it. Lets not
have this happen to this feature.
+ Matteo's suggestion (with a helpful starter list) that there needs to be
explicit qualification on what is actually being delivered -- including a
listing of limitations (some look serious such as data bleed from other
regions in WALs, but maybe I don't care for my use case...) -- needs to
accompany the merge. Lets fold them into the user doc. in the technical
overview area as suggested so user expectations are properly managed
(otherwise, they expect the world and will just give up when we fall
short). Vladimir did a list of what is in each of the phases above which
would serve as a good start.
+ Is this feature 'experimental' (Matteo asks above). I'd prefer it is not.
If it is, it should be labelled all over that it is so. I see current state
called out as a '... technical preview feature'. Does this mean
not-for-users?

St.Ack











On Mon, Sep 12, 2016 at 8:03 AM, Ted Yu  wrote:

> Sean:
> Do you have more comments ?
>
> Cheers
>
> On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov 
> wrote:
>
> > Sean,
> >
> > Backup/Restore can fail due to various reasons: network outage (cluster
> > wide), various time-outs in HBase and HDFS layer, M/R failure due to
> "HDFS
> > exceeded quota", user error (manual deletion of data) and so on so on.
> That
> > is impossible to enumerate all possible types of failures in a
> distributed
> > system - that is not our goal/task.
> >
> > We focus completely on backup system table consistency in a presence of
> any
> > type of failure. That is what I call "tolerance to failures".
> >
> > On a failure:
> >
> > BACKUP. All backup system information (prior to backup) will be restored
> > and all temporary data, related to a failed session, in HDFS will be
> > deleted
> > RESTORE. We do not care about system data, because restore does not
> change
> > it. Temporary data in HDFS will be cleaned up and table will be in a
> state
> > back to where it was before operation started.
> >
> > This is what user should expect in case of a failure.
> >
> > -Vlad
> >
> >
> > -Vlad
> >
> > On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey  wrote:
> >
> > > Failing in a consistent way, with docs that explain the various
> > > expected failures would be sufficient.
> > >
> > > On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov
> > >  wrote:
> > > > Do not worry Sean, doc is coming today as a preview and our writer
> > Frank
> > > > will be working on a putting  it into Apache repo. Timeline depends
> on
> > > > Franks schedule but I hope we will get it rather sooner than later.
> > > >
> > > > As for failure testing, we are focusing only on a consistent state of
> > > > backup system data in a presence of any type of failures, We are not
> > > going
> > > > to implement  anything more "fancy", than that. We allow both: backup
> > and
> > > > restore to fail. What we do not allow is to have system data
> corrupted.
> > > > Will it suffice for you? Do you have any other concerns, you want us
> to
> > > > address?
> > > >
> > > > -Vlad
> > > >
> > > >
> > > > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey 
> > wrote:
> > > >
> > > >> "docs will come to Apache soon" does not address my concern around
> > docs
> > > at
> > > >> all, unless said docs have already made it into the project repo. I
> > > don't
> > > >> want third party resources for using a major and important feature
> of
> > > the
> > > >> project, I want us to provide end users with what they need to get
> the
> > > job
> > > >> done.
> > > >>
> > > >> I see some calls for patience on the failure testing, but the appeal
> > to
> > > us
> > > >> having done a bad job of requiring proper tests of previous features
> > > just
> > > >> makes me more concerned about not getting them here. I don't want to
> > set
> > > >> yet another bad example that will then be pointed to in the future.
> > > >>
> > > >> On Sep 8, 2016 10:50, "Ted Yu"  wrote:
> > > >>
> > > >> > Is there any concern which is not addressed ?
> > > >> >
> > > >> > Do we need another Vote thread ?
> > > >> >
> > > >> > Thanks
> > > >> >
> > > >> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell <
> apurt...@apache.org
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Vlad,
> > > >> > >
> > > >> > > I apologize for using the term 'half-baked' in a way that could
> > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Sean:
Do you have more comments ?

Cheers

On Fri, Sep 9, 2016 at 1:42 PM, Vladimir Rodionov 
wrote:

> Sean,
>
> Backup/Restore can fail due to various reasons: network outage (cluster
> wide), various time-outs in HBase and HDFS layer, M/R failure due to "HDFS
> exceeded quota", user error (manual deletion of data) and so on so on. That
> is impossible to enumerate all possible types of failures in a distributed
> system - that is not our goal/task.
>
> We focus completely on backup system table consistency in a presence of any
> type of failure. That is what I call "tolerance to failures".
>
> On a failure:
>
> BACKUP. All backup system information (prior to backup) will be restored
> and all temporary data, related to a failed session, in HDFS will be
> deleted
> RESTORE. We do not care about system data, because restore does not change
> it. Temporary data in HDFS will be cleaned up and table will be in a state
> back to where it was before operation started.
>
> This is what user should expect in case of a failure.
>
> -Vlad
>
>
> -Vlad
>
> On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey  wrote:
>
> > Failing in a consistent way, with docs that explain the various
> > expected failures would be sufficient.
> >
> > On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov
> >  wrote:
> > > Do not worry Sean, doc is coming today as a preview and our writer
> Frank
> > > will be working on a putting  it into Apache repo. Timeline depends on
> > > Franks schedule but I hope we will get it rather sooner than later.
> > >
> > > As for failure testing, we are focusing only on a consistent state of
> > > backup system data in a presence of any type of failures, We are not
> > going
> > > to implement  anything more "fancy", than that. We allow both: backup
> and
> > > restore to fail. What we do not allow is to have system data corrupted.
> > > Will it suffice for you? Do you have any other concerns, you want us to
> > > address?
> > >
> > > -Vlad
> > >
> > >
> > > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey 
> wrote:
> > >
> > >> "docs will come to Apache soon" does not address my concern around
> docs
> > at
> > >> all, unless said docs have already made it into the project repo. I
> > don't
> > >> want third party resources for using a major and important feature of
> > the
> > >> project, I want us to provide end users with what they need to get the
> > job
> > >> done.
> > >>
> > >> I see some calls for patience on the failure testing, but the appeal
> to
> > us
> > >> having done a bad job of requiring proper tests of previous features
> > just
> > >> makes me more concerned about not getting them here. I don't want to
> set
> > >> yet another bad example that will then be pointed to in the future.
> > >>
> > >> On Sep 8, 2016 10:50, "Ted Yu"  wrote:
> > >>
> > >> > Is there any concern which is not addressed ?
> > >> >
> > >> > Do we need another Vote thread ?
> > >> >
> > >> > Thanks
> > >> >
> > >> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell  >
> > >> > wrote:
> > >> >
> > >> > > Vlad,
> > >> > >
> > >> > > I apologize for using the term 'half-baked' in a way that could
> > seem a
> > >> > > description of HBASE-7912. I meant that as a general hypothetical.
> > >> > >
> > >> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov <
> > >> > vladrodio...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > >> I'm not sure that "There is already lots of half-baked code
> in
> > the
> > >> > > > branch,
> > >> > > > so what's the harm in adding more?"
> > >> > > >
> > >> > > > I meant - not production - ready yet. This is 2.0 development
> > branch
> > >> > and,
> > >> > > > hence many features are in works,
> > >> > > > not being tested well etc. I do not consider backup as half
> baked
> > >> > > feature -
> > >> > > > it has passed our internal QA and has very good doc, which we
> will
> > >> > > provide
> > >> > > > to Apache shortly.
> > >> > > >
> > >> > > > -Vlad
> > >> > > >
> > >> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell <
> > apurt...@apache.org>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > We shouldn't admit half baked changes that won't be finished.
> > >> However
> > >> > > in
> > >> > > > > this case the crew working on this feature are long timers and
> > less
> > >> > > > likely
> > >> > > > > than just about anyone to leave something in a half baked
> > state. Of
> > >> > > > course
> > >> > > > > there is no guarantee how anything will turn out, but I am
> > willing
> > >> to
> > >> > > > take
> > >> > > > > a little on faith if they feel their best path forward now is
> to
> > >> > merge
> > >> > > to
> > >> > > > > trunk. I only wish I had bandwidth to have done some real
> > kicking
> > >> of
> > >> > > the
> > >> > > > > tires by now. Maybe this week.
> > >> > > > >
> > >> > > > > (Yes, I'm using some of that time for this email :-) but I
> 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

Sean,

Backup/Restore can fail due to various reasons: network outage (cluster
wide), various time-outs in HBase and HDFS layer, M/R failure due to "HDFS
exceeded quota", user error (manual deletion of data) and so on so on. That
is impossible to enumerate all possible types of failures in a distributed
system - that is not our goal/task.

We focus completely on backup system table consistency in a presence of any
type of failure. That is what I call "tolerance to failures".

On a failure:

BACKUP. All backup system information (prior to backup) will be restored
and all temporary data, related to a failed session, in HDFS will be deleted
RESTORE. We do not care about system data, because restore does not change
it. Temporary data in HDFS will be cleaned up and table will be in a state
back to where it was before operation started.

This is what user should expect in case of a failure.

-Vlad


-Vlad

On Fri, Sep 9, 2016 at 12:56 PM, Sean Busbey  wrote:

> Failing in a consistent way, with docs that explain the various
> expected failures would be sufficient.
>
> On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov
>  wrote:
> > Do not worry Sean, doc is coming today as a preview and our writer Frank
> > will be working on a putting  it into Apache repo. Timeline depends on
> > Franks schedule but I hope we will get it rather sooner than later.
> >
> > As for failure testing, we are focusing only on a consistent state of
> > backup system data in a presence of any type of failures, We are not
> going
> > to implement  anything more "fancy", than that. We allow both: backup and
> > restore to fail. What we do not allow is to have system data corrupted.
> > Will it suffice for you? Do you have any other concerns, you want us to
> > address?
> >
> > -Vlad
> >
> >
> > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey  wrote:
> >
> >> "docs will come to Apache soon" does not address my concern around docs
> at
> >> all, unless said docs have already made it into the project repo. I
> don't
> >> want third party resources for using a major and important feature of
> the
> >> project, I want us to provide end users with what they need to get the
> job
> >> done.
> >>
> >> I see some calls for patience on the failure testing, but the appeal to
> us
> >> having done a bad job of requiring proper tests of previous features
> just
> >> makes me more concerned about not getting them here. I don't want to set
> >> yet another bad example that will then be pointed to in the future.
> >>
> >> On Sep 8, 2016 10:50, "Ted Yu"  wrote:
> >>
> >> > Is there any concern which is not addressed ?
> >> >
> >> > Do we need another Vote thread ?
> >> >
> >> > Thanks
> >> >
> >> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell 
> >> > wrote:
> >> >
> >> > > Vlad,
> >> > >
> >> > > I apologize for using the term 'half-baked' in a way that could
> seem a
> >> > > description of HBASE-7912. I meant that as a general hypothetical.
> >> > >
> >> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov <
> >> > vladrodio...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > >> I'm not sure that "There is already lots of half-baked code in
> the
> >> > > > branch,
> >> > > > so what's the harm in adding more?"
> >> > > >
> >> > > > I meant - not production - ready yet. This is 2.0 development
> branch
> >> > and,
> >> > > > hence many features are in works,
> >> > > > not being tested well etc. I do not consider backup as half baked
> >> > > feature -
> >> > > > it has passed our internal QA and has very good doc, which we will
> >> > > provide
> >> > > > to Apache shortly.
> >> > > >
> >> > > > -Vlad
> >> > > >
> >> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell <
> apurt...@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > We shouldn't admit half baked changes that won't be finished.
> >> However
> >> > > in
> >> > > > > this case the crew working on this feature are long timers and
> less
> >> > > > likely
> >> > > > > than just about anyone to leave something in a half baked
> state. Of
> >> > > > course
> >> > > > > there is no guarantee how anything will turn out, but I am
> willing
> >> to
> >> > > > take
> >> > > > > a little on faith if they feel their best path forward now is to
> >> > merge
> >> > > to
> >> > > > > trunk. I only wish I had bandwidth to have done some real
> kicking
> >> of
> >> > > the
> >> > > > > tires by now. Maybe this week.
> >> > > > >
> >> > > > > (Yes, I'm using some of that time for this email :-) but I type
> >> > fast.)
> >> > > > >
> >> > > > > That said, I would like to agitate for making 2.0 more real and
> >> spend
> >> > > > some
> >> > > > > time on it now that I'm winding down with 0.98. I think that
> means
> >> > > > > branching for 2.0 real soon now and even evicting things from
> 2.0
> >> > > branch
> >> > > > > that aren't finished or stable, leaving them only once again in
> the
> >> > > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Matteo Bertozzi]

we should probably have a "current limitations" section in the user guide
(maybe near the technical details),
some of this stuff may be in the final 2.0 since some tasks are marked as
phase3,
but I think is important to mention stuff like:
 - if you write to the table with Durability.SKIP_WALS your data will not
be in the incremental-backup
 - if you bulkload files that data will not be in the incremental backup
(HBASE-14417)
 - the incremental backup will not only contains the data of the table you
specified but also the regions from other tables that are on the same set
of RSs (HBASE-14141) ...maybe a note about security around this topic
 - the incremental backup will not contains just the "latest row" between
backup A and B, but it will also contains all the updates occurred in
between. but the restore does not allow you to restore up to a certain
point in time, the restore will always be up to the "latest backup point".
 - you should limit the number of "incremental" up to N (or maybe SIZE), to
avoid replay time becoming the bottleneck. (HBASE-14135)


On Fri, Sep 9, 2016 at 12:25 PM, Vladimir Rodionov 
wrote:

> User Guide, prepared by our tech writer Frank Welsh, was attached to
> HBASE-7912.
>
> -Vlad
>
> On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov  >
> wrote:
>
> > Do not worry Sean, doc is coming today as a preview and our writer Frank
> > will be working on a putting  it into Apache repo. Timeline depends on
> > Franks schedule but I hope we will get it rather sooner than later.
> >
> > As for failure testing, we are focusing only on a consistent state of
> > backup system data in a presence of any type of failures, We are not
> going
> > to implement  anything more "fancy", than that. We allow both: backup and
> > restore to fail. What we do not allow is to have system data corrupted.
> > Will it suffice for you? Do you have any other concerns, you want us to
> > address?
> >
> > -Vlad
> >
> >
> > On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey  wrote:
> >
> >> "docs will come to Apache soon" does not address my concern around docs
> at
> >> all, unless said docs have already made it into the project repo. I
> don't
> >> want third party resources for using a major and important feature of
> the
> >> project, I want us to provide end users with what they need to get the
> job
> >> done.
> >>
> >> I see some calls for patience on the failure testing, but the appeal to
> us
> >> having done a bad job of requiring proper tests of previous features
> just
> >> makes me more concerned about not getting them here. I don't want to set
> >> yet another bad example that will then be pointed to in the future.
> >>
> >> On Sep 8, 2016 10:50, "Ted Yu"  wrote:
> >>
> >> > Is there any concern which is not addressed ?
> >> >
> >> > Do we need another Vote thread ?
> >> >
> >> > Thanks
> >> >
> >> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell 
> >> > wrote:
> >> >
> >> > > Vlad,
> >> > >
> >> > > I apologize for using the term 'half-baked' in a way that could
> seem a
> >> > > description of HBASE-7912. I meant that as a general hypothetical.
> >> > >
> >> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov <
> >> > vladrodio...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > >> I'm not sure that "There is already lots of half-baked code in
> >> the
> >> > > > branch,
> >> > > > so what's the harm in adding more?"
> >> > > >
> >> > > > I meant - not production - ready yet. This is 2.0 development
> branch
> >> > and,
> >> > > > hence many features are in works,
> >> > > > not being tested well etc. I do not consider backup as half baked
> >> > > feature -
> >> > > > it has passed our internal QA and has very good doc, which we will
> >> > > provide
> >> > > > to Apache shortly.
> >> > > >
> >> > > > -Vlad
> >> > > >
> >> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell <
> apurt...@apache.org
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > We shouldn't admit half baked changes that won't be finished.
> >> However
> >> > > in
> >> > > > > this case the crew working on this feature are long timers and
> >> less
> >> > > > likely
> >> > > > > than just about anyone to leave something in a half baked state.
> >> Of
> >> > > > course
> >> > > > > there is no guarantee how anything will turn out, but I am
> >> willing to
> >> > > > take
> >> > > > > a little on faith if they feel their best path forward now is to
> >> > merge
> >> > > to
> >> > > > > trunk. I only wish I had bandwidth to have done some real
> kicking
> >> of
> >> > > the
> >> > > > > tires by now. Maybe this week.
> >> > > > >
> >> > > > > (Yes, I'm using some of that time for this email :-) but I type
> >> > fast.)
> >> > > > >
> >> > > > > That said, I would like to agitate for making 2.0 more real and
> >> spend
> >> > > > some
> >> > > > > time on it now that I'm winding down with 0.98. I think that
> means
> >> > > > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Sean Busbey]

Failing in a consistent way, with docs that explain the various
expected failures would be sufficient.

On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov
 wrote:
> Do not worry Sean, doc is coming today as a preview and our writer Frank
> will be working on a putting  it into Apache repo. Timeline depends on
> Franks schedule but I hope we will get it rather sooner than later.
>
> As for failure testing, we are focusing only on a consistent state of
> backup system data in a presence of any type of failures, We are not going
> to implement  anything more "fancy", than that. We allow both: backup and
> restore to fail. What we do not allow is to have system data corrupted.
> Will it suffice for you? Do you have any other concerns, you want us to
> address?
>
> -Vlad
>
>
> On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey  wrote:
>
>> "docs will come to Apache soon" does not address my concern around docs at
>> all, unless said docs have already made it into the project repo. I don't
>> want third party resources for using a major and important feature of the
>> project, I want us to provide end users with what they need to get the job
>> done.
>>
>> I see some calls for patience on the failure testing, but the appeal to us
>> having done a bad job of requiring proper tests of previous features just
>> makes me more concerned about not getting them here. I don't want to set
>> yet another bad example that will then be pointed to in the future.
>>
>> On Sep 8, 2016 10:50, "Ted Yu"  wrote:
>>
>> > Is there any concern which is not addressed ?
>> >
>> > Do we need another Vote thread ?
>> >
>> > Thanks
>> >
>> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell 
>> > wrote:
>> >
>> > > Vlad,
>> > >
>> > > I apologize for using the term 'half-baked' in a way that could seem a
>> > > description of HBASE-7912. I meant that as a general hypothetical.
>> > >
>> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov <
>> > vladrodio...@gmail.com>
>> > > wrote:
>> > >
>> > > > >> I'm not sure that "There is already lots of half-baked code in the
>> > > > branch,
>> > > > so what's the harm in adding more?"
>> > > >
>> > > > I meant - not production - ready yet. This is 2.0 development branch
>> > and,
>> > > > hence many features are in works,
>> > > > not being tested well etc. I do not consider backup as half baked
>> > > feature -
>> > > > it has passed our internal QA and has very good doc, which we will
>> > > provide
>> > > > to Apache shortly.
>> > > >
>> > > > -Vlad
>> > > >
>> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell 
>> > > > wrote:
>> > > >
>> > > > > We shouldn't admit half baked changes that won't be finished.
>> However
>> > > in
>> > > > > this case the crew working on this feature are long timers and less
>> > > > likely
>> > > > > than just about anyone to leave something in a half baked state. Of
>> > > > course
>> > > > > there is no guarantee how anything will turn out, but I am willing
>> to
>> > > > take
>> > > > > a little on faith if they feel their best path forward now is to
>> > merge
>> > > to
>> > > > > trunk. I only wish I had bandwidth to have done some real kicking
>> of
>> > > the
>> > > > > tires by now. Maybe this week.
>> > > > >
>> > > > > (Yes, I'm using some of that time for this email :-) but I type
>> > fast.)
>> > > > >
>> > > > > That said, I would like to agitate for making 2.0 more real and
>> spend
>> > > > some
>> > > > > time on it now that I'm winding down with 0.98. I think that means
>> > > > > branching for 2.0 real soon now and even evicting things from 2.0
>> > > branch
>> > > > > that aren't finished or stable, leaving them only once again in the
>> > > > master
>> > > > > branch. Or, maybe just evicting them. Let's take it case by case.
>> > > > >
>> > > > > I think this feature can come in relatively safely. As added
>> > insurance,
>> > > > > let's admit the possibility it could be reverted on the 2.0 branch
>> if
>> > > > folks
>> > > > > working on stabilizing 2.0 decide to evict it because it is
>> > unfinished
>> > > or
>> > > > > unstable, because that certainly can happen. I would expect if talk
>> > > like
>> > > > > that starts, we'd get help finishing or stabilizing what's under
>> > > > discussion
>> > > > > for revert. Or, we'd have a revert. Either way the outcome is
>> > > acceptable.
>> > > > >
>> > > > >
>> > > > > On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak > >
>> > > > wrote:
>> > > > >
>> > > > > > I'm not sure that "There is already lots of half-baked code in
>> the
>> > > > > branch,
>> > > > > > so what's the harm in adding more?" is a good code commit
>> > philosophy
>> > > > for
>> > > > > a
>> > > > > > fault-tolerant distributed data store. ;)
>> > > > > >
>> > > > > > More seriously, a lack of test coverage for existing features
>> > > shouldn't
>> > > > > be
>> > > > > > used as justification for 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

User Guide, prepared by our tech writer Frank Welsh, was attached to
HBASE-7912.

-Vlad

On Fri, Sep 9, 2016 at 12:16 PM, Vladimir Rodionov 
wrote:

> Do not worry Sean, doc is coming today as a preview and our writer Frank
> will be working on a putting  it into Apache repo. Timeline depends on
> Franks schedule but I hope we will get it rather sooner than later.
>
> As for failure testing, we are focusing only on a consistent state of
> backup system data in a presence of any type of failures, We are not going
> to implement  anything more "fancy", than that. We allow both: backup and
> restore to fail. What we do not allow is to have system data corrupted.
> Will it suffice for you? Do you have any other concerns, you want us to
> address?
>
> -Vlad
>
>
> On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey  wrote:
>
>> "docs will come to Apache soon" does not address my concern around docs at
>> all, unless said docs have already made it into the project repo. I don't
>> want third party resources for using a major and important feature of the
>> project, I want us to provide end users with what they need to get the job
>> done.
>>
>> I see some calls for patience on the failure testing, but the appeal to us
>> having done a bad job of requiring proper tests of previous features just
>> makes me more concerned about not getting them here. I don't want to set
>> yet another bad example that will then be pointed to in the future.
>>
>> On Sep 8, 2016 10:50, "Ted Yu"  wrote:
>>
>> > Is there any concern which is not addressed ?
>> >
>> > Do we need another Vote thread ?
>> >
>> > Thanks
>> >
>> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell 
>> > wrote:
>> >
>> > > Vlad,
>> > >
>> > > I apologize for using the term 'half-baked' in a way that could seem a
>> > > description of HBASE-7912. I meant that as a general hypothetical.
>> > >
>> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov <
>> > vladrodio...@gmail.com>
>> > > wrote:
>> > >
>> > > > >> I'm not sure that "There is already lots of half-baked code in
>> the
>> > > > branch,
>> > > > so what's the harm in adding more?"
>> > > >
>> > > > I meant - not production - ready yet. This is 2.0 development branch
>> > and,
>> > > > hence many features are in works,
>> > > > not being tested well etc. I do not consider backup as half baked
>> > > feature -
>> > > > it has passed our internal QA and has very good doc, which we will
>> > > provide
>> > > > to Apache shortly.
>> > > >
>> > > > -Vlad
>> > > >
>> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell > >
>> > > > wrote:
>> > > >
>> > > > > We shouldn't admit half baked changes that won't be finished.
>> However
>> > > in
>> > > > > this case the crew working on this feature are long timers and
>> less
>> > > > likely
>> > > > > than just about anyone to leave something in a half baked state.
>> Of
>> > > > course
>> > > > > there is no guarantee how anything will turn out, but I am
>> willing to
>> > > > take
>> > > > > a little on faith if they feel their best path forward now is to
>> > merge
>> > > to
>> > > > > trunk. I only wish I had bandwidth to have done some real kicking
>> of
>> > > the
>> > > > > tires by now. Maybe this week.
>> > > > >
>> > > > > (Yes, I'm using some of that time for this email :-) but I type
>> > fast.)
>> > > > >
>> > > > > That said, I would like to agitate for making 2.0 more real and
>> spend
>> > > > some
>> > > > > time on it now that I'm winding down with 0.98. I think that means
>> > > > > branching for 2.0 real soon now and even evicting things from 2.0
>> > > branch
>> > > > > that aren't finished or stable, leaving them only once again in
>> the
>> > > > master
>> > > > > branch. Or, maybe just evicting them. Let's take it case by case.
>> > > > >
>> > > > > I think this feature can come in relatively safely. As added
>> > insurance,
>> > > > > let's admit the possibility it could be reverted on the 2.0
>> branch if
>> > > > folks
>> > > > > working on stabilizing 2.0 decide to evict it because it is
>> > unfinished
>> > > or
>> > > > > unstable, because that certainly can happen. I would expect if
>> talk
>> > > like
>> > > > > that starts, we'd get help finishing or stabilizing what's under
>> > > > discussion
>> > > > > for revert. Or, we'd have a revert. Either way the outcome is
>> > > acceptable.
>> > > > >
>> > > > >
>> > > > > On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak <
>> dimaspi...@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > > I'm not sure that "There is already lots of half-baked code in
>> the
>> > > > > branch,
>> > > > > > so what's the harm in adding more?" is a good code commit
>> > philosophy
>> > > > for
>> > > > > a
>> > > > > > fault-tolerant distributed data store. ;)
>> > > > > >
>> > > > > > More seriously, a lack of test coverage for existing features
>> > > shouldn't
>> > > > > be
>> > > > > > used as justification for 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

Do not worry Sean, doc is coming today as a preview and our writer Frank
will be working on a putting  it into Apache repo. Timeline depends on
Franks schedule but I hope we will get it rather sooner than later.

As for failure testing, we are focusing only on a consistent state of
backup system data in a presence of any type of failures, We are not going
to implement  anything more "fancy", than that. We allow both: backup and
restore to fail. What we do not allow is to have system data corrupted.
Will it suffice for you? Do you have any other concerns, you want us to
address?

-Vlad


On Fri, Sep 9, 2016 at 10:56 AM, Sean Busbey  wrote:

> "docs will come to Apache soon" does not address my concern around docs at
> all, unless said docs have already made it into the project repo. I don't
> want third party resources for using a major and important feature of the
> project, I want us to provide end users with what they need to get the job
> done.
>
> I see some calls for patience on the failure testing, but the appeal to us
> having done a bad job of requiring proper tests of previous features just
> makes me more concerned about not getting them here. I don't want to set
> yet another bad example that will then be pointed to in the future.
>
> On Sep 8, 2016 10:50, "Ted Yu"  wrote:
>
> > Is there any concern which is not addressed ?
> >
> > Do we need another Vote thread ?
> >
> > Thanks
> >
> > On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell 
> > wrote:
> >
> > > Vlad,
> > >
> > > I apologize for using the term 'half-baked' in a way that could seem a
> > > description of HBASE-7912. I meant that as a general hypothetical.
> > >
> > > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov <
> > vladrodio...@gmail.com>
> > > wrote:
> > >
> > > > >> I'm not sure that "There is already lots of half-baked code in the
> > > > branch,
> > > > so what's the harm in adding more?"
> > > >
> > > > I meant - not production - ready yet. This is 2.0 development branch
> > and,
> > > > hence many features are in works,
> > > > not being tested well etc. I do not consider backup as half baked
> > > feature -
> > > > it has passed our internal QA and has very good doc, which we will
> > > provide
> > > > to Apache shortly.
> > > >
> > > > -Vlad
> > > >
> > > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell 
> > > > wrote:
> > > >
> > > > > We shouldn't admit half baked changes that won't be finished.
> However
> > > in
> > > > > this case the crew working on this feature are long timers and less
> > > > likely
> > > > > than just about anyone to leave something in a half baked state. Of
> > > > course
> > > > > there is no guarantee how anything will turn out, but I am willing
> to
> > > > take
> > > > > a little on faith if they feel their best path forward now is to
> > merge
> > > to
> > > > > trunk. I only wish I had bandwidth to have done some real kicking
> of
> > > the
> > > > > tires by now. Maybe this week.
> > > > >
> > > > > (Yes, I'm using some of that time for this email :-) but I type
> > fast.)
> > > > >
> > > > > That said, I would like to agitate for making 2.0 more real and
> spend
> > > > some
> > > > > time on it now that I'm winding down with 0.98. I think that means
> > > > > branching for 2.0 real soon now and even evicting things from 2.0
> > > branch
> > > > > that aren't finished or stable, leaving them only once again in the
> > > > master
> > > > > branch. Or, maybe just evicting them. Let's take it case by case.
> > > > >
> > > > > I think this feature can come in relatively safely. As added
> > insurance,
> > > > > let's admit the possibility it could be reverted on the 2.0 branch
> if
> > > > folks
> > > > > working on stabilizing 2.0 decide to evict it because it is
> > unfinished
> > > or
> > > > > unstable, because that certainly can happen. I would expect if talk
> > > like
> > > > > that starts, we'd get help finishing or stabilizing what's under
> > > > discussion
> > > > > for revert. Or, we'd have a revert. Either way the outcome is
> > > acceptable.
> > > > >
> > > > >
> > > > > On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak  >
> > > > wrote:
> > > > >
> > > > > > I'm not sure that "There is already lots of half-baked code in
> the
> > > > > branch,
> > > > > > so what's the harm in adding more?" is a good code commit
> > philosophy
> > > > for
> > > > > a
> > > > > > fault-tolerant distributed data store. ;)
> > > > > >
> > > > > > More seriously, a lack of test coverage for existing features
> > > shouldn't
> > > > > be
> > > > > > used as justification for introducing new features with the same
> > > > > > shortcomings. Ultimately, it's the end user who will feel the
> pain,
> > > so
> > > > > > shouldn't we do everything we can to mitigate that?
> > > > > >
> > > > > > -Dima
> > > > > >
> > > > > > On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov <
> > > > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Sean Busbey]

"docs will come to Apache soon" does not address my concern around docs at
all, unless said docs have already made it into the project repo. I don't
want third party resources for using a major and important feature of the
project, I want us to provide end users with what they need to get the job
done.

I see some calls for patience on the failure testing, but the appeal to us
having done a bad job of requiring proper tests of previous features just
makes me more concerned about not getting them here. I don't want to set
yet another bad example that will then be pointed to in the future.

On Sep 8, 2016 10:50, "Ted Yu"  wrote:

> Is there any concern which is not addressed ?
>
> Do we need another Vote thread ?
>
> Thanks
>
> On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell 
> wrote:
>
> > Vlad,
> >
> > I apologize for using the term 'half-baked' in a way that could seem a
> > description of HBASE-7912. I meant that as a general hypothetical.
> >
> > On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov <
> vladrodio...@gmail.com>
> > wrote:
> >
> > > >> I'm not sure that "There is already lots of half-baked code in the
> > > branch,
> > > so what's the harm in adding more?"
> > >
> > > I meant - not production - ready yet. This is 2.0 development branch
> and,
> > > hence many features are in works,
> > > not being tested well etc. I do not consider backup as half baked
> > feature -
> > > it has passed our internal QA and has very good doc, which we will
> > provide
> > > to Apache shortly.
> > >
> > > -Vlad
> > >
> > > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell 
> > > wrote:
> > >
> > > > We shouldn't admit half baked changes that won't be finished. However
> > in
> > > > this case the crew working on this feature are long timers and less
> > > likely
> > > > than just about anyone to leave something in a half baked state. Of
> > > course
> > > > there is no guarantee how anything will turn out, but I am willing to
> > > take
> > > > a little on faith if they feel their best path forward now is to
> merge
> > to
> > > > trunk. I only wish I had bandwidth to have done some real kicking of
> > the
> > > > tires by now. Maybe this week.
> > > >
> > > > (Yes, I'm using some of that time for this email :-) but I type
> fast.)
> > > >
> > > > That said, I would like to agitate for making 2.0 more real and spend
> > > some
> > > > time on it now that I'm winding down with 0.98. I think that means
> > > > branching for 2.0 real soon now and even evicting things from 2.0
> > branch
> > > > that aren't finished or stable, leaving them only once again in the
> > > master
> > > > branch. Or, maybe just evicting them. Let's take it case by case.
> > > >
> > > > I think this feature can come in relatively safely. As added
> insurance,
> > > > let's admit the possibility it could be reverted on the 2.0 branch if
> > > folks
> > > > working on stabilizing 2.0 decide to evict it because it is
> unfinished
> > or
> > > > unstable, because that certainly can happen. I would expect if talk
> > like
> > > > that starts, we'd get help finishing or stabilizing what's under
> > > discussion
> > > > for revert. Or, we'd have a revert. Either way the outcome is
> > acceptable.
> > > >
> > > >
> > > > On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak 
> > > wrote:
> > > >
> > > > > I'm not sure that "There is already lots of half-baked code in the
> > > > branch,
> > > > > so what's the harm in adding more?" is a good code commit
> philosophy
> > > for
> > > > a
> > > > > fault-tolerant distributed data store. ;)
> > > > >
> > > > > More seriously, a lack of test coverage for existing features
> > shouldn't
> > > > be
> > > > > used as justification for introducing new features with the same
> > > > > shortcomings. Ultimately, it's the end user who will feel the pain,
> > so
> > > > > shouldn't we do everything we can to mitigate that?
> > > > >
> > > > > -Dima
> > > > >
> > > > > On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov <
> > > > vladrodio...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Sean,
> > > > > >
> > > > > > * have docs
> > > > > >
> > > > > > Agree. We have a doc and backup is the most documented feature
> :),
> > we
> > > > > will
> > > > > > release it shortly to Apache.
> > > > > >
> > > > > > * have sunny-day correctness tests
> > > > > >
> > > > > > Feature has  close to 60 test cases, which run for approx 30 min.
> > We
> > > > can
> > > > > > add more, if community do not mind :)
> > > > > >
> > > > > > * have correctness-in-face-of-failure tests
> > > > > >
> > > > > > Any examples of these tests in existing features? In works, we
> > have a
> > > > > clear
> > > > > > understanding of what should be done by the time of 2.0 release.
> > > > > > That is very close goal for us, to verify IT monkey for existing
> > > code.
> > > > > >
> > > > > > * don't rely on things outside of HBase for normal operation
> (okay
> > > for

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Is there any concern which is not addressed ?

Do we need another Vote thread ?

Thanks

On Thu, Sep 8, 2016 at 9:21 AM, Andrew Purtell  wrote:

> Vlad,
>
> I apologize for using the term 'half-baked' in a way that could seem a
> description of HBASE-7912. I meant that as a general hypothetical.
>
> On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov 
> wrote:
>
> > >> I'm not sure that "There is already lots of half-baked code in the
> > branch,
> > so what's the harm in adding more?"
> >
> > I meant - not production - ready yet. This is 2.0 development branch and,
> > hence many features are in works,
> > not being tested well etc. I do not consider backup as half baked
> feature -
> > it has passed our internal QA and has very good doc, which we will
> provide
> > to Apache shortly.
> >
> > -Vlad
> >
> > On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell 
> > wrote:
> >
> > > We shouldn't admit half baked changes that won't be finished. However
> in
> > > this case the crew working on this feature are long timers and less
> > likely
> > > than just about anyone to leave something in a half baked state. Of
> > course
> > > there is no guarantee how anything will turn out, but I am willing to
> > take
> > > a little on faith if they feel their best path forward now is to merge
> to
> > > trunk. I only wish I had bandwidth to have done some real kicking of
> the
> > > tires by now. Maybe this week.
> > >
> > > (Yes, I'm using some of that time for this email :-) but I type fast.)
> > >
> > > That said, I would like to agitate for making 2.0 more real and spend
> > some
> > > time on it now that I'm winding down with 0.98. I think that means
> > > branching for 2.0 real soon now and even evicting things from 2.0
> branch
> > > that aren't finished or stable, leaving them only once again in the
> > master
> > > branch. Or, maybe just evicting them. Let's take it case by case.
> > >
> > > I think this feature can come in relatively safely. As added insurance,
> > > let's admit the possibility it could be reverted on the 2.0 branch if
> > folks
> > > working on stabilizing 2.0 decide to evict it because it is unfinished
> or
> > > unstable, because that certainly can happen. I would expect if talk
> like
> > > that starts, we'd get help finishing or stabilizing what's under
> > discussion
> > > for revert. Or, we'd have a revert. Either way the outcome is
> acceptable.
> > >
> > >
> > > On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak 
> > wrote:
> > >
> > > > I'm not sure that "There is already lots of half-baked code in the
> > > branch,
> > > > so what's the harm in adding more?" is a good code commit philosophy
> > for
> > > a
> > > > fault-tolerant distributed data store. ;)
> > > >
> > > > More seriously, a lack of test coverage for existing features
> shouldn't
> > > be
> > > > used as justification for introducing new features with the same
> > > > shortcomings. Ultimately, it's the end user who will feel the pain,
> so
> > > > shouldn't we do everything we can to mitigate that?
> > > >
> > > > -Dima
> > > >
> > > > On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov <
> > > vladrodio...@gmail.com>
> > > > wrote:
> > > >
> > > > > Sean,
> > > > >
> > > > > * have docs
> > > > >
> > > > > Agree. We have a doc and backup is the most documented feature :),
> we
> > > > will
> > > > > release it shortly to Apache.
> > > > >
> > > > > * have sunny-day correctness tests
> > > > >
> > > > > Feature has  close to 60 test cases, which run for approx 30 min.
> We
> > > can
> > > > > add more, if community do not mind :)
> > > > >
> > > > > * have correctness-in-face-of-failure tests
> > > > >
> > > > > Any examples of these tests in existing features? In works, we
> have a
> > > > clear
> > > > > understanding of what should be done by the time of 2.0 release.
> > > > > That is very close goal for us, to verify IT monkey for existing
> > code.
> > > > >
> > > > > * don't rely on things outside of HBase for normal operation (okay
> > for
> > > > > advanced operation)
> > > > >
> > > > > We do not.
> > > > >
> > > > > Enormous time has been spent already on the development and testing
> > the
> > > > > feature, it has passed our internal tests and many rounds of code
> > > reviews
> > > > > by HBase committers. We do not mind if someone from HBase community
> > > > > (outside of HW) will review the code, but it will probably takes
> > > forever
> > > > to
> > > > > wait for volunteer?, the feature is quite large (1MB+ cumulative
> > patch)
> > > > >
> > > > > 2.0 branch is full of half baked features, most of them are in
> active
> > > > > development, therefore I am not following you here, Sean? Why
> > > HBASE-7912
> > > > is
> > > > > not good enough yet to be integrated into 2.0 branch?
> > > > >
> > > > > -Vlad
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Andrew Purtell]

Vlad,

I apologize for using the term 'half-baked' in a way that could seem a
description of HBASE-7912. I meant that as a general hypothetical.

On Wed, Sep 7, 2016 at 9:36 AM, Vladimir Rodionov 
wrote:

> >> I'm not sure that "There is already lots of half-baked code in the
> branch,
> so what's the harm in adding more?"
>
> I meant - not production - ready yet. This is 2.0 development branch and,
> hence many features are in works,
> not being tested well etc. I do not consider backup as half baked feature -
> it has passed our internal QA and has very good doc, which we will provide
> to Apache shortly.
>
> -Vlad
>
> On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell 
> wrote:
>
> > We shouldn't admit half baked changes that won't be finished. However in
> > this case the crew working on this feature are long timers and less
> likely
> > than just about anyone to leave something in a half baked state. Of
> course
> > there is no guarantee how anything will turn out, but I am willing to
> take
> > a little on faith if they feel their best path forward now is to merge to
> > trunk. I only wish I had bandwidth to have done some real kicking of the
> > tires by now. Maybe this week.
> >
> > (Yes, I'm using some of that time for this email :-) but I type fast.)
> >
> > That said, I would like to agitate for making 2.0 more real and spend
> some
> > time on it now that I'm winding down with 0.98. I think that means
> > branching for 2.0 real soon now and even evicting things from 2.0 branch
> > that aren't finished or stable, leaving them only once again in the
> master
> > branch. Or, maybe just evicting them. Let's take it case by case.
> >
> > I think this feature can come in relatively safely. As added insurance,
> > let's admit the possibility it could be reverted on the 2.0 branch if
> folks
> > working on stabilizing 2.0 decide to evict it because it is unfinished or
> > unstable, because that certainly can happen. I would expect if talk like
> > that starts, we'd get help finishing or stabilizing what's under
> discussion
> > for revert. Or, we'd have a revert. Either way the outcome is acceptable.
> >
> >
> > On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak 
> wrote:
> >
> > > I'm not sure that "There is already lots of half-baked code in the
> > branch,
> > > so what's the harm in adding more?" is a good code commit philosophy
> for
> > a
> > > fault-tolerant distributed data store. ;)
> > >
> > > More seriously, a lack of test coverage for existing features shouldn't
> > be
> > > used as justification for introducing new features with the same
> > > shortcomings. Ultimately, it's the end user who will feel the pain, so
> > > shouldn't we do everything we can to mitigate that?
> > >
> > > -Dima
> > >
> > > On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov <
> > vladrodio...@gmail.com>
> > > wrote:
> > >
> > > > Sean,
> > > >
> > > > * have docs
> > > >
> > > > Agree. We have a doc and backup is the most documented feature :), we
> > > will
> > > > release it shortly to Apache.
> > > >
> > > > * have sunny-day correctness tests
> > > >
> > > > Feature has  close to 60 test cases, which run for approx 30 min. We
> > can
> > > > add more, if community do not mind :)
> > > >
> > > > * have correctness-in-face-of-failure tests
> > > >
> > > > Any examples of these tests in existing features? In works, we have a
> > > clear
> > > > understanding of what should be done by the time of 2.0 release.
> > > > That is very close goal for us, to verify IT monkey for existing
> code.
> > > >
> > > > * don't rely on things outside of HBase for normal operation (okay
> for
> > > > advanced operation)
> > > >
> > > > We do not.
> > > >
> > > > Enormous time has been spent already on the development and testing
> the
> > > > feature, it has passed our internal tests and many rounds of code
> > reviews
> > > > by HBase committers. We do not mind if someone from HBase community
> > > > (outside of HW) will review the code, but it will probably takes
> > forever
> > > to
> > > > wait for volunteer?, the feature is quite large (1MB+ cumulative
> patch)
> > > >
> > > > 2.0 branch is full of half baked features, most of them are in active
> > > > development, therefore I am not following you here, Sean? Why
> > HBASE-7912
> > > is
> > > > not good enough yet to be integrated into 2.0 branch?
> > > >
> > > > -Vlad
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey 
> wrote:
> > > >
> > > > > On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser 
> > > > wrote:
> > > > > > So, the answer to Sean's original question is "as robust as
> > snapshots
> > > > > > presently are"? (independence of backup/restore failure tolerance
> > > from
> > > > > > snapshot failure tolerance)
> > > > > >
> > > > > > Is this just a question WRT context of the change, or is it means
> > > for a
> > > > > veto
> > > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Enis Söztutar]

On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell  wrote:

> We shouldn't admit half baked changes that won't be finished. However in
> this case the crew working on this feature are long timers and less likely
> than just about anyone to leave something in a half baked state. Of course
> there is no guarantee how anything will turn out, but I am willing to take
> a little on faith if they feel their best path forward now is to merge to
> trunk.


I think this is a very good point that Andrew raised. The developers have
no
intention to stop working on improvements after the merge. Likely that the
stabilization and working on remaining phase 3/4 items will continue for
the next months to come. If we can get a commitment that any remaining
issues will be addressed by end-of-year timeframe, then making a call for
the merge and inclusion in hbase-2.0 will become much easier.



> I only wish I had bandwidth to have done some real kicking of the
> tires by now. Maybe this week.
>
> (Yes, I'm using some of that time for this email :-) but I type fast.)
>
> That said, I would like to agitate for making 2.0 more real and spend some
> time on it now that I'm winding down with 0.98. I think that means
> branching for 2.0 real soon now and even evicting things from 2.0 branch
> that aren't finished or stable, leaving them only once again in the master
> branch. Or, maybe just evicting them. Let's take it case by case.
>
> I think this feature can come in relatively safely. As added insurance,
> let's admit the possibility it could be reverted on the 2.0 branch if folks
> working on stabilizing 2.0 decide to evict it because it is unfinished or
> unstable, because that certainly can happen. I would expect if talk like
> that starts, we'd get help finishing or stabilizing what's under discussion
> for revert. Or, we'd have a revert. Either way the outcome is acceptable.
>
>
> On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak  wrote:
>
> > I'm not sure that "There is already lots of half-baked code in the
> branch,
> > so what's the harm in adding more?" is a good code commit philosophy for
> a
> > fault-tolerant distributed data store. ;)
> >
> > More seriously, a lack of test coverage for existing features shouldn't
> be
> > used as justification for introducing new features with the same
> > shortcomings. Ultimately, it's the end user who will feel the pain, so
> > shouldn't we do everything we can to mitigate that?
> >
> > -Dima
> >
> > On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov <
> vladrodio...@gmail.com>
> > wrote:
> >
> > > Sean,
> > >
> > > * have docs
> > >
> > > Agree. We have a doc and backup is the most documented feature :), we
> > will
> > > release it shortly to Apache.
> > >
> > > * have sunny-day correctness tests
> > >
> > > Feature has  close to 60 test cases, which run for approx 30 min. We
> can
> > > add more, if community do not mind :)
> > >
> > > * have correctness-in-face-of-failure tests
> > >
> > > Any examples of these tests in existing features? In works, we have a
> > clear
> > > understanding of what should be done by the time of 2.0 release.
> > > That is very close goal for us, to verify IT monkey for existing code.
> > >
> > > * don't rely on things outside of HBase for normal operation (okay for
> > > advanced operation)
> > >
> > > We do not.
> > >
> > > Enormous time has been spent already on the development and testing the
> > > feature, it has passed our internal tests and many rounds of code
> reviews
> > > by HBase committers. We do not mind if someone from HBase community
> > > (outside of HW) will review the code, but it will probably takes
> forever
> > to
> > > wait for volunteer?, the feature is quite large (1MB+ cumulative patch)
> > >
> > > 2.0 branch is full of half baked features, most of them are in active
> > > development, therefore I am not following you here, Sean? Why
> HBASE-7912
> > is
> > > not good enough yet to be integrated into 2.0 branch?
> > >
> > > -Vlad
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey  wrote:
> > >
> > > > On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser 
> > > wrote:
> > > > > So, the answer to Sean's original question is "as robust as
> snapshots
> > > > > presently are"? (independence of backup/restore failure tolerance
> > from
> > > > > snapshot failure tolerance)
> > > > >
> > > > > Is this just a question WRT context of the change, or is it means
> > for a
> > > > veto
> > > > > from you, Sean? Just trying to make sure I'm following along
> > > adequately.
> > > > >
> > > > >
> > > >
> > > > I'd say ATM I'm -0, bordering on -1 but not for reasons I can
> > articulate
> > > > well.
> > > >
> > > > Here's an attempt.
> > > >
> > > > We've been trying to move, as a community, towards minimizing risk to
> > > > downstream folks by getting "complete enough for use" gates in place
> > > > before we introduce new features. This 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> I'm not sure that "There is already lots of half-baked code in the
branch,
so what's the harm in adding more?"

I meant - not production - ready yet. This is 2.0 development branch and,
hence many features are in works,
not being tested well etc. I do not consider backup as half baked feature -
it has passed our internal QA and has very good doc, which we will provide
to Apache shortly.

-Vlad

On Wed, Sep 7, 2016 at 9:13 AM, Andrew Purtell  wrote:

> We shouldn't admit half baked changes that won't be finished. However in
> this case the crew working on this feature are long timers and less likely
> than just about anyone to leave something in a half baked state. Of course
> there is no guarantee how anything will turn out, but I am willing to take
> a little on faith if they feel their best path forward now is to merge to
> trunk. I only wish I had bandwidth to have done some real kicking of the
> tires by now. Maybe this week.
>
> (Yes, I'm using some of that time for this email :-) but I type fast.)
>
> That said, I would like to agitate for making 2.0 more real and spend some
> time on it now that I'm winding down with 0.98. I think that means
> branching for 2.0 real soon now and even evicting things from 2.0 branch
> that aren't finished or stable, leaving them only once again in the master
> branch. Or, maybe just evicting them. Let's take it case by case.
>
> I think this feature can come in relatively safely. As added insurance,
> let's admit the possibility it could be reverted on the 2.0 branch if folks
> working on stabilizing 2.0 decide to evict it because it is unfinished or
> unstable, because that certainly can happen. I would expect if talk like
> that starts, we'd get help finishing or stabilizing what's under discussion
> for revert. Or, we'd have a revert. Either way the outcome is acceptable.
>
>
> On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak  wrote:
>
> > I'm not sure that "There is already lots of half-baked code in the
> branch,
> > so what's the harm in adding more?" is a good code commit philosophy for
> a
> > fault-tolerant distributed data store. ;)
> >
> > More seriously, a lack of test coverage for existing features shouldn't
> be
> > used as justification for introducing new features with the same
> > shortcomings. Ultimately, it's the end user who will feel the pain, so
> > shouldn't we do everything we can to mitigate that?
> >
> > -Dima
> >
> > On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov <
> vladrodio...@gmail.com>
> > wrote:
> >
> > > Sean,
> > >
> > > * have docs
> > >
> > > Agree. We have a doc and backup is the most documented feature :), we
> > will
> > > release it shortly to Apache.
> > >
> > > * have sunny-day correctness tests
> > >
> > > Feature has  close to 60 test cases, which run for approx 30 min. We
> can
> > > add more, if community do not mind :)
> > >
> > > * have correctness-in-face-of-failure tests
> > >
> > > Any examples of these tests in existing features? In works, we have a
> > clear
> > > understanding of what should be done by the time of 2.0 release.
> > > That is very close goal for us, to verify IT monkey for existing code.
> > >
> > > * don't rely on things outside of HBase for normal operation (okay for
> > > advanced operation)
> > >
> > > We do not.
> > >
> > > Enormous time has been spent already on the development and testing the
> > > feature, it has passed our internal tests and many rounds of code
> reviews
> > > by HBase committers. We do not mind if someone from HBase community
> > > (outside of HW) will review the code, but it will probably takes
> forever
> > to
> > > wait for volunteer?, the feature is quite large (1MB+ cumulative patch)
> > >
> > > 2.0 branch is full of half baked features, most of them are in active
> > > development, therefore I am not following you here, Sean? Why
> HBASE-7912
> > is
> > > not good enough yet to be integrated into 2.0 branch?
> > >
> > > -Vlad
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey  wrote:
> > >
> > > > On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser 
> > > wrote:
> > > > > So, the answer to Sean's original question is "as robust as
> snapshots
> > > > > presently are"? (independence of backup/restore failure tolerance
> > from
> > > > > snapshot failure tolerance)
> > > > >
> > > > > Is this just a question WRT context of the change, or is it means
> > for a
> > > > veto
> > > > > from you, Sean? Just trying to make sure I'm following along
> > > adequately.
> > > > >
> > > > >
> > > >
> > > > I'd say ATM I'm -0, bordering on -1 but not for reasons I can
> > articulate
> > > > well.
> > > >
> > > > Here's an attempt.
> > > >
> > > > We've been trying to move, as a community, towards minimizing risk to
> > > > downstream folks by getting "complete enough for use" gates in place
> > > > before we introduce new features. This was spurred by a some features
> > > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Josh Elser]

So, given your stated concerns, Sean, the correctness-in-face-of-failure 
tests is the only WIP thing remaining that you'd like to see, right? Are 
there any other issues that you can think of now that would prevent you 
from +1'ing?


Regarding the move to a "higher-bar for new features" by the community, 
I'll stand back because I know that I have not been following closely 
enough to comment here. If this is a community decision that everyone is 
on board with, great. I honestly have not followed discussions closely 
enough to understand whether this is a "we'd like to do this" or "we are 
doing this".


This leads me to asking how can we address the concern on the amorphous 
state of what "HBase-2.0" and how to avoid this blocking contributions 
to 2.0. It seems like there's a chicken-and-egg problem with being 
unable to determine the expected level of testing while there's no 
concrete roadmap for 2.0. To me, it seems premature to raise the bar 
higher in this case (because there is no firm date where continued work 
must be completed by to avoid blocking the release), but, again, I type 
this with great hesitance (as I have not be diligent with my inbox).


Vladimir Rodionov wrote:

Sean,

* have docs

Agree. We have a doc and backup is the most documented feature :), we will
release it shortly to Apache.

* have sunny-day correctness tests

Feature has  close to 60 test cases, which run for approx 30 min. We can
add more, if community do not mind :)

* have correctness-in-face-of-failure tests

Any examples of these tests in existing features? In works, we have a clear
understanding of what should be done by the time of 2.0 release.
That is very close goal for us, to verify IT monkey for existing code.

* don't rely on things outside of HBase for normal operation (okay for
advanced operation)

We do not.

Enormous time has been spent already on the development and testing the
feature, it has passed our internal tests and many rounds of code reviews
by HBase committers. We do not mind if someone from HBase community
(outside of HW) will review the code, but it will probably takes forever to
wait for volunteer?, the feature is quite large (1MB+ cumulative patch)

2.0 branch is full of half baked features, most of them are in active
development, therefore I am not following you here, Sean? Why HBASE-7912 is
not good enough yet to be integrated into 2.0 branch?

-Vlad

On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey  wrote:


On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser  wrote:

So, the answer to Sean's original question is "as robust as snapshots
presently are"? (independence of backup/restore failure tolerance from
snapshot failure tolerance)

Is this just a question WRT context of the change, or is it means for a

veto

from you, Sean? Just trying to make sure I'm following along adequately.



I'd say ATM I'm -0, bordering on -1 but not for reasons I can articulate
well.

Here's an attempt.

We've been trying to move, as a community, towards minimizing risk to
downstream folks by getting "complete enough for use" gates in place
before we introduce new features. This was spurred by a some features
getting in half-baked and never making it to "can really use" status
(I'm thinking of distributed log replay and the zk-less assignment
stuff, I don't recall if there was more).

The gates, generally, included things like:

* have docs
* have sunny-day correctness tests
* have correctness-in-face-of-failure tests
* don't rely on things outside of HBase for normal operation (okay for
advanced operation)

As an example, we kept the MOB work off in a branch and out of master
until it could pass these criteria. The big exemption we've had to
this was the hbase-spark integration, where we all agreed it could
land in master because it was very well isolated (the slide away from
including docs as a first-class part of building up that integration
has led me to doubt the wisdom of this decision).

We've also been treating inclusion in a "probably will be released to
downstream" branches as a higher bar, requiring

* don't moderately impact performance when the feature isn't in use
* don't severely impact performance when the feature is in use
* either default-to-on or show enough demand to believe a non-trivial
number of folks will turn the feature on

The above has kept MOB and hbase-spark integration out of branch-1,
presumably while they've "gotten more stable" in master from the odd
vendor inclusion.

Are we going to have a 2.0 release before the end of the year? We're
coming up on 1.5 years since the release of version 1.0; seems like
it's about time, though I haven't seen any concrete plans this year.
Presuming we are going to have one by the end of the year, it seems a
bit close to still be adding in "features that need maturing" on the
branch.

The lack of a concrete plan for 2.0 keeps me from considering these
things blocker at the moment. But I know first hand how 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Adding more tests around backup / restore is in the works.
e.g. I will start with HBASE-16497 this week.

I agree that developers of the feature should try out typical scenarios so
that user experience is better.

30 minute of unit tests runtime for backup / restore is for machine with
SSDs. For machine without SSD, it would be longer. We would continue to add
new tests and prune redundant test cases.

Thanks


On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak  wrote:

> I'm not sure that "There is already lots of half-baked code in the branch,
> so what's the harm in adding more?" is a good code commit philosophy for a
> fault-tolerant distributed data store. ;)
>
> More seriously, a lack of test coverage for existing features shouldn't be
> used as justification for introducing new features with the same
> shortcomings. Ultimately, it's the end user who will feel the pain, so
> shouldn't we do everything we can to mitigate that?
>
> -Dima
>
> On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov 
> wrote:
>
> > Sean,
> >
> > * have docs
> >
> > Agree. We have a doc and backup is the most documented feature :), we
> will
> > release it shortly to Apache.
> >
> > * have sunny-day correctness tests
> >
> > Feature has  close to 60 test cases, which run for approx 30 min. We can
> > add more, if community do not mind :)
> >
> > * have correctness-in-face-of-failure tests
> >
> > Any examples of these tests in existing features? In works, we have a
> clear
> > understanding of what should be done by the time of 2.0 release.
> > That is very close goal for us, to verify IT monkey for existing code.
> >
> > * don't rely on things outside of HBase for normal operation (okay for
> > advanced operation)
> >
> > We do not.
> >
> > Enormous time has been spent already on the development and testing the
> > feature, it has passed our internal tests and many rounds of code reviews
> > by HBase committers. We do not mind if someone from HBase community
> > (outside of HW) will review the code, but it will probably takes forever
> to
> > wait for volunteer?, the feature is quite large (1MB+ cumulative patch)
> >
> > 2.0 branch is full of half baked features, most of them are in active
> > development, therefore I am not following you here, Sean? Why HBASE-7912
> is
> > not good enough yet to be integrated into 2.0 branch?
> >
> > -Vlad
> >
> >
> >
> >
> >
> > On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey  wrote:
> >
> > > On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser 
> > wrote:
> > > > So, the answer to Sean's original question is "as robust as snapshots
> > > > presently are"? (independence of backup/restore failure tolerance
> from
> > > > snapshot failure tolerance)
> > > >
> > > > Is this just a question WRT context of the change, or is it means
> for a
> > > veto
> > > > from you, Sean? Just trying to make sure I'm following along
> > adequately.
> > > >
> > > >
> > >
> > > I'd say ATM I'm -0, bordering on -1 but not for reasons I can
> articulate
> > > well.
> > >
> > > Here's an attempt.
> > >
> > > We've been trying to move, as a community, towards minimizing risk to
> > > downstream folks by getting "complete enough for use" gates in place
> > > before we introduce new features. This was spurred by a some features
> > > getting in half-baked and never making it to "can really use" status
> > > (I'm thinking of distributed log replay and the zk-less assignment
> > > stuff, I don't recall if there was more).
> > >
> > > The gates, generally, included things like:
> > >
> > > * have docs
> > > * have sunny-day correctness tests
> > > * have correctness-in-face-of-failure tests
> > > * don't rely on things outside of HBase for normal operation (okay for
> > > advanced operation)
> > >
> > > As an example, we kept the MOB work off in a branch and out of master
> > > until it could pass these criteria. The big exemption we've had to
> > > this was the hbase-spark integration, where we all agreed it could
> > > land in master because it was very well isolated (the slide away from
> > > including docs as a first-class part of building up that integration
> > > has led me to doubt the wisdom of this decision).
> > >
> > > We've also been treating inclusion in a "probably will be released to
> > > downstream" branches as a higher bar, requiring
> > >
> > > * don't moderately impact performance when the feature isn't in use
> > > * don't severely impact performance when the feature is in use
> > > * either default-to-on or show enough demand to believe a non-trivial
> > > number of folks will turn the feature on
> > >
> > > The above has kept MOB and hbase-spark integration out of branch-1,
> > > presumably while they've "gotten more stable" in master from the odd
> > > vendor inclusion.
> > >
> > > Are we going to have a 2.0 release before the end of the year? We're
> > > coming up on 1.5 years since the release of version 1.0; seems like
> > > 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Andrew Purtell]

We shouldn't admit half baked changes that won't be finished. However in
this case the crew working on this feature are long timers and less likely
than just about anyone to leave something in a half baked state. Of course
there is no guarantee how anything will turn out, but I am willing to take
a little on faith if they feel their best path forward now is to merge to
trunk. I only wish I had bandwidth to have done some real kicking of the
tires by now. Maybe this week.

(Yes, I'm using some of that time for this email :-) but I type fast.)

That said, I would like to agitate for making 2.0 more real and spend some
time on it now that I'm winding down with 0.98. I think that means
branching for 2.0 real soon now and even evicting things from 2.0 branch
that aren't finished or stable, leaving them only once again in the master
branch. Or, maybe just evicting them. Let's take it case by case.

I think this feature can come in relatively safely. As added insurance,
let's admit the possibility it could be reverted on the 2.0 branch if folks
working on stabilizing 2.0 decide to evict it because it is unfinished or
unstable, because that certainly can happen. I would expect if talk like
that starts, we'd get help finishing or stabilizing what's under discussion
for revert. Or, we'd have a revert. Either way the outcome is acceptable.


On Wed, Sep 7, 2016 at 8:56 AM, Dima Spivak  wrote:

> I'm not sure that "There is already lots of half-baked code in the branch,
> so what's the harm in adding more?" is a good code commit philosophy for a
> fault-tolerant distributed data store. ;)
>
> More seriously, a lack of test coverage for existing features shouldn't be
> used as justification for introducing new features with the same
> shortcomings. Ultimately, it's the end user who will feel the pain, so
> shouldn't we do everything we can to mitigate that?
>
> -Dima
>
> On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov 
> wrote:
>
> > Sean,
> >
> > * have docs
> >
> > Agree. We have a doc and backup is the most documented feature :), we
> will
> > release it shortly to Apache.
> >
> > * have sunny-day correctness tests
> >
> > Feature has  close to 60 test cases, which run for approx 30 min. We can
> > add more, if community do not mind :)
> >
> > * have correctness-in-face-of-failure tests
> >
> > Any examples of these tests in existing features? In works, we have a
> clear
> > understanding of what should be done by the time of 2.0 release.
> > That is very close goal for us, to verify IT monkey for existing code.
> >
> > * don't rely on things outside of HBase for normal operation (okay for
> > advanced operation)
> >
> > We do not.
> >
> > Enormous time has been spent already on the development and testing the
> > feature, it has passed our internal tests and many rounds of code reviews
> > by HBase committers. We do not mind if someone from HBase community
> > (outside of HW) will review the code, but it will probably takes forever
> to
> > wait for volunteer?, the feature is quite large (1MB+ cumulative patch)
> >
> > 2.0 branch is full of half baked features, most of them are in active
> > development, therefore I am not following you here, Sean? Why HBASE-7912
> is
> > not good enough yet to be integrated into 2.0 branch?
> >
> > -Vlad
> >
> >
> >
> >
> >
> > On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey  wrote:
> >
> > > On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser 
> > wrote:
> > > > So, the answer to Sean's original question is "as robust as snapshots
> > > > presently are"? (independence of backup/restore failure tolerance
> from
> > > > snapshot failure tolerance)
> > > >
> > > > Is this just a question WRT context of the change, or is it means
> for a
> > > veto
> > > > from you, Sean? Just trying to make sure I'm following along
> > adequately.
> > > >
> > > >
> > >
> > > I'd say ATM I'm -0, bordering on -1 but not for reasons I can
> articulate
> > > well.
> > >
> > > Here's an attempt.
> > >
> > > We've been trying to move, as a community, towards minimizing risk to
> > > downstream folks by getting "complete enough for use" gates in place
> > > before we introduce new features. This was spurred by a some features
> > > getting in half-baked and never making it to "can really use" status
> > > (I'm thinking of distributed log replay and the zk-less assignment
> > > stuff, I don't recall if there was more).
> > >
> > > The gates, generally, included things like:
> > >
> > > * have docs
> > > * have sunny-day correctness tests
> > > * have correctness-in-face-of-failure tests
> > > * don't rely on things outside of HBase for normal operation (okay for
> > > advanced operation)
> > >
> > > As an example, we kept the MOB work off in a branch and out of master
> > > until it could pass these criteria. The big exemption we've had to
> > > this was the hbase-spark integration, where we all agreed it could
> > > land in 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Dima Spivak]

I'm not sure that "There is already lots of half-baked code in the branch,
so what's the harm in adding more?" is a good code commit philosophy for a
fault-tolerant distributed data store. ;)

More seriously, a lack of test coverage for existing features shouldn't be
used as justification for introducing new features with the same
shortcomings. Ultimately, it's the end user who will feel the pain, so
shouldn't we do everything we can to mitigate that?

-Dima

On Wed, Sep 7, 2016 at 8:46 AM, Vladimir Rodionov 
wrote:

> Sean,
>
> * have docs
>
> Agree. We have a doc and backup is the most documented feature :), we will
> release it shortly to Apache.
>
> * have sunny-day correctness tests
>
> Feature has  close to 60 test cases, which run for approx 30 min. We can
> add more, if community do not mind :)
>
> * have correctness-in-face-of-failure tests
>
> Any examples of these tests in existing features? In works, we have a clear
> understanding of what should be done by the time of 2.0 release.
> That is very close goal for us, to verify IT monkey for existing code.
>
> * don't rely on things outside of HBase for normal operation (okay for
> advanced operation)
>
> We do not.
>
> Enormous time has been spent already on the development and testing the
> feature, it has passed our internal tests and many rounds of code reviews
> by HBase committers. We do not mind if someone from HBase community
> (outside of HW) will review the code, but it will probably takes forever to
> wait for volunteer?, the feature is quite large (1MB+ cumulative patch)
>
> 2.0 branch is full of half baked features, most of them are in active
> development, therefore I am not following you here, Sean? Why HBASE-7912 is
> not good enough yet to be integrated into 2.0 branch?
>
> -Vlad
>
>
>
>
>
> On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey  wrote:
>
> > On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser 
> wrote:
> > > So, the answer to Sean's original question is "as robust as snapshots
> > > presently are"? (independence of backup/restore failure tolerance from
> > > snapshot failure tolerance)
> > >
> > > Is this just a question WRT context of the change, or is it means for a
> > veto
> > > from you, Sean? Just trying to make sure I'm following along
> adequately.
> > >
> > >
> >
> > I'd say ATM I'm -0, bordering on -1 but not for reasons I can articulate
> > well.
> >
> > Here's an attempt.
> >
> > We've been trying to move, as a community, towards minimizing risk to
> > downstream folks by getting "complete enough for use" gates in place
> > before we introduce new features. This was spurred by a some features
> > getting in half-baked and never making it to "can really use" status
> > (I'm thinking of distributed log replay and the zk-less assignment
> > stuff, I don't recall if there was more).
> >
> > The gates, generally, included things like:
> >
> > * have docs
> > * have sunny-day correctness tests
> > * have correctness-in-face-of-failure tests
> > * don't rely on things outside of HBase for normal operation (okay for
> > advanced operation)
> >
> > As an example, we kept the MOB work off in a branch and out of master
> > until it could pass these criteria. The big exemption we've had to
> > this was the hbase-spark integration, where we all agreed it could
> > land in master because it was very well isolated (the slide away from
> > including docs as a first-class part of building up that integration
> > has led me to doubt the wisdom of this decision).
> >
> > We've also been treating inclusion in a "probably will be released to
> > downstream" branches as a higher bar, requiring
> >
> > * don't moderately impact performance when the feature isn't in use
> > * don't severely impact performance when the feature is in use
> > * either default-to-on or show enough demand to believe a non-trivial
> > number of folks will turn the feature on
> >
> > The above has kept MOB and hbase-spark integration out of branch-1,
> > presumably while they've "gotten more stable" in master from the odd
> > vendor inclusion.
> >
> > Are we going to have a 2.0 release before the end of the year? We're
> > coming up on 1.5 years since the release of version 1.0; seems like
> > it's about time, though I haven't seen any concrete plans this year.
> > Presuming we are going to have one by the end of the year, it seems a
> > bit close to still be adding in "features that need maturing" on the
> > branch.
> >
> > The lack of a concrete plan for 2.0 keeps me from considering these
> > things blocker at the moment. But I know first hand how much trouble
> > folks have had with other features that have gone into downstream
> > facing releases without robustness checks (i.e. replication), and I'm
> > concerned about what we're setting up if 2.0 goes out with this
> > feature in its current state.
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

Sean,

* have docs

Agree. We have a doc and backup is the most documented feature :), we will
release it shortly to Apache.

* have sunny-day correctness tests

Feature has  close to 60 test cases, which run for approx 30 min. We can
add more, if community do not mind :)

* have correctness-in-face-of-failure tests

Any examples of these tests in existing features? In works, we have a clear
understanding of what should be done by the time of 2.0 release.
That is very close goal for us, to verify IT monkey for existing code.

* don't rely on things outside of HBase for normal operation (okay for
advanced operation)

We do not.

Enormous time has been spent already on the development and testing the
feature, it has passed our internal tests and many rounds of code reviews
by HBase committers. We do not mind if someone from HBase community
(outside of HW) will review the code, but it will probably takes forever to
wait for volunteer?, the feature is quite large (1MB+ cumulative patch)

2.0 branch is full of half baked features, most of them are in active
development, therefore I am not following you here, Sean? Why HBASE-7912 is
not good enough yet to be integrated into 2.0 branch?

-Vlad





On Wed, Sep 7, 2016 at 8:23 AM, Sean Busbey  wrote:

> On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser  wrote:
> > So, the answer to Sean's original question is "as robust as snapshots
> > presently are"? (independence of backup/restore failure tolerance from
> > snapshot failure tolerance)
> >
> > Is this just a question WRT context of the change, or is it means for a
> veto
> > from you, Sean? Just trying to make sure I'm following along adequately.
> >
> >
>
> I'd say ATM I'm -0, bordering on -1 but not for reasons I can articulate
> well.
>
> Here's an attempt.
>
> We've been trying to move, as a community, towards minimizing risk to
> downstream folks by getting "complete enough for use" gates in place
> before we introduce new features. This was spurred by a some features
> getting in half-baked and never making it to "can really use" status
> (I'm thinking of distributed log replay and the zk-less assignment
> stuff, I don't recall if there was more).
>
> The gates, generally, included things like:
>
> * have docs
> * have sunny-day correctness tests
> * have correctness-in-face-of-failure tests
> * don't rely on things outside of HBase for normal operation (okay for
> advanced operation)
>
> As an example, we kept the MOB work off in a branch and out of master
> until it could pass these criteria. The big exemption we've had to
> this was the hbase-spark integration, where we all agreed it could
> land in master because it was very well isolated (the slide away from
> including docs as a first-class part of building up that integration
> has led me to doubt the wisdom of this decision).
>
> We've also been treating inclusion in a "probably will be released to
> downstream" branches as a higher bar, requiring
>
> * don't moderately impact performance when the feature isn't in use
> * don't severely impact performance when the feature is in use
> * either default-to-on or show enough demand to believe a non-trivial
> number of folks will turn the feature on
>
> The above has kept MOB and hbase-spark integration out of branch-1,
> presumably while they've "gotten more stable" in master from the odd
> vendor inclusion.
>
> Are we going to have a 2.0 release before the end of the year? We're
> coming up on 1.5 years since the release of version 1.0; seems like
> it's about time, though I haven't seen any concrete plans this year.
> Presuming we are going to have one by the end of the year, it seems a
> bit close to still be adding in "features that need maturing" on the
> branch.
>
> The lack of a concrete plan for 2.0 keeps me from considering these
> things blocker at the moment. But I know first hand how much trouble
> folks have had with other features that have gone into downstream
> facing releases without robustness checks (i.e. replication), and I'm
> concerned about what we're setting up if 2.0 goes out with this
> feature in its current state.
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Sean Busbey]

On Tue, Sep 6, 2016 at 10:36 PM, Josh Elser  wrote:
> So, the answer to Sean's original question is "as robust as snapshots
> presently are"? (independence of backup/restore failure tolerance from
> snapshot failure tolerance)
>
> Is this just a question WRT context of the change, or is it means for a veto
> from you, Sean? Just trying to make sure I'm following along adequately.
>
>

I'd say ATM I'm -0, bordering on -1 but not for reasons I can articulate well.

Here's an attempt.

We've been trying to move, as a community, towards minimizing risk to
downstream folks by getting "complete enough for use" gates in place
before we introduce new features. This was spurred by a some features
getting in half-baked and never making it to "can really use" status
(I'm thinking of distributed log replay and the zk-less assignment
stuff, I don't recall if there was more).

The gates, generally, included things like:

* have docs
* have sunny-day correctness tests
* have correctness-in-face-of-failure tests
* don't rely on things outside of HBase for normal operation (okay for
advanced operation)

As an example, we kept the MOB work off in a branch and out of master
until it could pass these criteria. The big exemption we've had to
this was the hbase-spark integration, where we all agreed it could
land in master because it was very well isolated (the slide away from
including docs as a first-class part of building up that integration
has led me to doubt the wisdom of this decision).

We've also been treating inclusion in a "probably will be released to
downstream" branches as a higher bar, requiring

* don't moderately impact performance when the feature isn't in use
* don't severely impact performance when the feature is in use
* either default-to-on or show enough demand to believe a non-trivial
number of folks will turn the feature on

The above has kept MOB and hbase-spark integration out of branch-1,
presumably while they've "gotten more stable" in master from the odd
vendor inclusion.

Are we going to have a 2.0 release before the end of the year? We're
coming up on 1.5 years since the release of version 1.0; seems like
it's about time, though I haven't seen any concrete plans this year.
Presuming we are going to have one by the end of the year, it seems a
bit close to still be adding in "features that need maturing" on the
branch.

The lack of a concrete plan for 2.0 keeps me from considering these
things blocker at the moment. But I know first hand how much trouble
folks have had with other features that have gone into downstream
facing releases without robustness checks (i.e. replication), and I'm
concerned about what we're setting up if 2.0 goes out with this
feature in its current state.

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Andrew Purtell]

Fine I'll cast a vote as -0. 

If I find time to test that could easily change to +1. Perhaps my vote won't be 
needed. I don't wish to block you. 

> On Sep 6, 2016, at 6:55 PM, Ted Yu  wrote:
> 
> Andrew:
> Do you think you would have some time this week ?
> 
> Thanks
> 
> On Thu, Sep 1, 2016 at 8:47 AM, Andrew Purtell 
> wrote:
> 
>> Busy at work, aiming for next week.
>> 
>>> On Sep 1, 2016, at 8:44 AM, Ted Yu  wrote:
>>> 
>>> Andrew:
>>> HBASE-16255 has been resolved.
>>> 
>>> Kindly provide your feedback.
>>> 
>>> Thanks
>>> 
>>> On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell <
>> andrew.purt...@gmail.com>
>>> wrote:
>>> 
 I plan to spin up a test cluster with clusterdock and try running the IT
 under a number of different scenarios. I understand snapshots have to
 function so baseline would be the calm monkey.
 
 Unless you have some other automated way for me to run the new
 functionality repeatedly, the IT is it.
 
>> On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov <
>> vladrodio...@gmail.com>
> wrote:
> 
> Not sure what do you mean, Andrew by "trying out the branch via the
>> IT",
> but we do not recommend running this with monkey enabled.
> It has not been tested in a such scenario yet and frankly speaking it
>> is
> not supposed to work (snapshots will fail anyway and we depends on
> snapshots)
> 
> -Vladimir
> 
> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell <
 andrew.purt...@gmail.com>
> wrote:
> 
>> Let's commit the IT to the branch, if you think the v5 patch is ready
 for
>> commit Ted.
>> 
>> I will be able to spend some time next week trying out the branch via
 the
>> IT, and poking around with the new tools. After that I feel like I'll
>> be
>> informed enough to vote on a branch merge vote.
>> 
>>> On Aug 19, 2016, at 12:38 PM, Ted Yu  wrote:
>>> 
>>> IT test is provided on HBASE-16255.
>>> 
>>> Any other comment ?
>>> 
>>> Thanks
>>> 
 On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak 
>> wrote:
 
 Any chance for an IT test being added to the branch first? I'd love
>> to
>> put
 it through the paces with clusterdock to make sure it behaves well
 with
 fault injection and the like.
 
 -Dima
 
> On Tuesday, August 2, 2016, Ted Yu  wrote:
> 
> Any more comments from the community on whether the merge can be
 conducted
> ?
> 
> Thanks
> 
> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov <
 vladrodio...@gmail.com
> >
> wrote:
> 
>> Carter Shanklin posted a blog article about the feature:
>> Some use cases and examples of a command line interface usage.
> https://hortonworks.com/blog/coming-hdp-2-5-incremental-
 backup-restore-apache-hbase-apache-phoenix/
>> 
>> -Vlad
>> 
>> On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov <
> vladrodio...@gmail.com 
>> wrote:
>> 
>>> Ok, got it.
>>> 
>>> -Vlad
>>> 
>>> On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar  > wrote:
>>> 
 We keep the WALs which can accumulate a lot if the use case is
>> to
 only
>> do
 backups infrequently. This will definitely cause issues since
>> HDFS
> space
 will get filled up. That is why we may need an option for having
 incremental backups not used, and WAL references being deleted.
 
 Enis
 
 On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov <
 vladrodio...@gmail.com >
 wrote:
 
> Why anyone will ever need disabling incremental backups? If you
 do
> not
 need
> it - just run only full backups.
> 
> -Vlad
> 
> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar <
>> e...@apache.org
> >
>> wrote:
> 
>> Thanks Matteo for chiming in.
>> 
>> On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi <
> theo.berto...@gmail.com >
>> wrote:
>> 
>>> I did some review in the early beginning, but then lost track
 of
>> the
>>> changes.
>>> but I'd like to give a quick review to the full code once
 people
 here
> are
>>> ok with getting this feature in master (2.0).
>>> (let say we put a 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Josh Elser]

So, the answer to Sean's original question is "as robust as snapshots 
presently are"? (independence of backup/restore failure tolerance from 
snapshot failure tolerance)


Is this just a question WRT context of the change, or is it means for a 
veto from you, Sean? Just trying to make sure I'm following along 
adequately.


Vladimir Rodionov wrote:

Snapshot robustness is better now with introduction of region splits/merges
on/off feature. Region splits during snapshots was the major problem.

-Vlad

On Fri, Sep 2, 2016 at 8:12 AM, Vladimir Rodionov
wrote:


Are they independent enough that we can get backup/restore tolerant to
failures prior to merge to master? Prior to backport to branch-1?

As we stated already, snapshots are not part of the feature, snapshots has
been merged into the master long time ago
and as far as I understood - without requiring them to be 100% robust and
fault tolerant and they are widely used in many production systems
nevertheless. https://issues.apache.org/jira/browse/HBASE-14415 relies on
Snapshots v2 but we can reconsider it, there are some thoughts how to make
backups snapshotless.

Backups are fault tolerant to some extent - in case of failure (and
failures can happen) we clean everything up and do not leave system table
in inconsistent state. Would it be enough, Sean Busbey?

-Vlad

On Fri, Sep 2, 2016 at 7:38 AM, Ted Yu  wrote:


We're continuing to make backup / restore more robust.
Work in progress (both are close to being integrated):

HBASE-15565 Rewrite restore with Procedure V2
HBASE-15449 Support physical table layout change

Since snapshot is dependency in the full backup, backup / restore wouldn't
be more robust than snapshot is.

On Fri, Sep 2, 2016 at 7:03 AM, Sean Busbey  wrote:


right, they're separate features but when asked about "robust
backup/restore" (which is what I care about for this feature getting
merged) things were pawned off on snapshots.

Are they independent enough that we can get backup/restore tolerant to
failures prior to merge to master? Prior to backport to branch-1?

On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell
wrote:

I agree these are separate features FWIW

On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov<

vladrodio...@gmail.com>

wrote:


Do we have JIRA issue(s) covering making snapshots robust in the

face

of monkeys?

I would like to mention that "robust snapshots" and "table

backup/restore"

are totally separate features, but we have separate JIRA for fault
tolerance (HBASE-14413).

-Vlad

On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:


Sean:
Please see HBASE-14413 for the last question.

FYI

On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey

wrote:

On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
  wrote:

Not sure what do you mean, Andrew by "trying out the branch via

the

IT",

but we do not recommend running this with monkey enabled.
It has not been tested in a such scenario yet and frankly

speaking it

is

not supposed to work (snapshots will fail anyway and we

depends on

snapshots)



Also won't have time to test out the branch this week, but if

we're

not going to handle failures do we have tools or guidance on
recovering in the case of things falling over?

Do we have JIRA issue(s) covering making snapshots robust in the

face

of monkeys?

--
busbey




--
Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back. - Piet

Hein

(via Tom White)



--
busbey

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Andrew:
Do you think you would have some time this week ?

Thanks

On Thu, Sep 1, 2016 at 8:47 AM, Andrew Purtell 
wrote:

> Busy at work, aiming for next week.
>
> > On Sep 1, 2016, at 8:44 AM, Ted Yu  wrote:
> >
> > Andrew:
> > HBASE-16255 has been resolved.
> >
> > Kindly provide your feedback.
> >
> > Thanks
> >
> > On Sat, Aug 20, 2016 at 11:06 AM, Andrew Purtell <
> andrew.purt...@gmail.com>
> > wrote:
> >
> >> I plan to spin up a test cluster with clusterdock and try running the IT
> >> under a number of different scenarios. I understand snapshots have to
> >> function so baseline would be the calm monkey.
> >>
> >> Unless you have some other automated way for me to run the new
> >> functionality repeatedly, the IT is it.
> >>
>  On Aug 20, 2016, at 10:59 AM, Vladimir Rodionov <
> vladrodio...@gmail.com>
> >>> wrote:
> >>>
> >>> Not sure what do you mean, Andrew by "trying out the branch via the
> IT",
> >>> but we do not recommend running this with monkey enabled.
> >>> It has not been tested in a such scenario yet and frankly speaking it
> is
> >>> not supposed to work (snapshots will fail anyway and we depends on
> >>> snapshots)
> >>>
> >>> -Vladimir
> >>>
> >>> On Sat, Aug 20, 2016 at 10:29 AM, Andrew Purtell <
> >> andrew.purt...@gmail.com>
> >>> wrote:
> >>>
>  Let's commit the IT to the branch, if you think the v5 patch is ready
> >> for
>  commit Ted.
> 
>  I will be able to spend some time next week trying out the branch via
> >> the
>  IT, and poking around with the new tools. After that I feel like I'll
> be
>  informed enough to vote on a branch merge vote.
> 
> > On Aug 19, 2016, at 12:38 PM, Ted Yu  wrote:
> >
> > IT test is provided on HBASE-16255.
> >
> > Any other comment ?
> >
> > Thanks
> >
> >> On Tue, Aug 2, 2016 at 9:09 PM, Dima Spivak 
>  wrote:
> >>
> >> Any chance for an IT test being added to the branch first? I'd love
> to
>  put
> >> it through the paces with clusterdock to make sure it behaves well
> >> with
> >> fault injection and the like.
> >>
> >> -Dima
> >>
> >>> On Tuesday, August 2, 2016, Ted Yu  wrote:
> >>>
> >>> Any more comments from the community on whether the merge can be
> >> conducted
> >>> ?
> >>>
> >>> Thanks
> >>>
> >>> On Mon, Aug 1, 2016 at 12:03 PM, Vladimir Rodionov <
> >> vladrodio...@gmail.com
> >>> >
> >>> wrote:
> >>>
>  Carter Shanklin posted a blog article about the feature:
>  Some use cases and examples of a command line interface usage.
> >>> https://hortonworks.com/blog/coming-hdp-2-5-incremental-
> >> backup-restore-apache-hbase-apache-phoenix/
> 
>  -Vlad
> 
>  On Wed, Jul 20, 2016 at 1:25 PM, Vladimir Rodionov <
> >>> vladrodio...@gmail.com 
>  wrote:
> 
> > Ok, got it.
> >
> > -Vlad
> >
> > On Wed, Jul 20, 2016 at 12:15 PM, Enis Söztutar  >>> > wrote:
> >
> >> We keep the WALs which can accumulate a lot if the use case is
> to
> >> only
>  do
> >> backups infrequently. This will definitely cause issues since
> HDFS
> >>> space
> >> will get filled up. That is why we may need an option for having
> >> incremental backups not used, and WAL references being deleted.
> >>
> >> Enis
> >>
> >> On Tue, Jul 19, 2016 at 6:33 PM, Vladimir Rodionov <
> >> vladrodio...@gmail.com >
> >> wrote:
> >>
> >>> Why anyone will ever need disabling incremental backups? If you
> >> do
> >>> not
> >> need
> >>> it - just run only full backups.
> >>>
> >>> -Vlad
> >>>
> >>> On Tue, Jul 19, 2016 at 6:21 PM, Enis Söztutar <
> e...@apache.org
> >>> >
>  wrote:
> >>>
>  Thanks Matteo for chiming in.
> 
>  On Tue, Jul 19, 2016 at 5:02 PM, Matteo Bertozzi <
> >>> theo.berto...@gmail.com >
>  wrote:
> 
> > I did some review in the early beginning, but then lost track
> >> of
>  the
> > changes.
> > but I'd like to give a quick review to the full code once
> >> people
> >> here
> >>> are
> > ok with getting this feature in master (2.0).
> > (let say we put a deadline for reviews, like 1 week for
> >>> reviewing
> >> the
>  full
> > stuff after everyone agrees to get this in. just to avoid
> >>> holding
> >> this
>  for
> > too long, but still enough time to have people 

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Devaraj Das]

+1 for merge of HBASE-7912 to master. I think this feature can go in in its 
present form and bake for sometime in the master. Work is continuing to harden 
it.


From: Vladimir Rodionov <vladrodio...@gmail.com>
Sent: Friday, September 02, 2016 8:51 AM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

Snapshot robustness is better now with introduction of region splits/merges
on/off feature. Region splits during snapshots was the major problem.

-Vlad

On Fri, Sep 2, 2016 at 8:12 AM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> >>Are they independent enough that we can get backup/restore tolerant to
> >>failures prior to merge to master? Prior to backport to branch-1?
>
> As we stated already, snapshots are not part of the feature, snapshots has
> been merged into the master long time ago
> and as far as I understood - without requiring them to be 100% robust and
> fault tolerant and they are widely used in many production systems
> nevertheless. https://issues.apache.org/jira/browse/HBASE-14415 relies on
> Snapshots v2 but we can reconsider it, there are some thoughts how to make
> backups snapshotless.
>
> Backups are fault tolerant to some extent - in case of failure (and
> failures can happen) we clean everything up and do not leave system table
> in inconsistent state. Would it be enough, Sean Busbey?
>
> -Vlad
>
> On Fri, Sep 2, 2016 at 7:38 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> We're continuing to make backup / restore more robust.
>> Work in progress (both are close to being integrated):
>>
>> HBASE-15565 Rewrite restore with Procedure V2
>> HBASE-15449 Support physical table layout change
>>
>> Since snapshot is dependency in the full backup, backup / restore wouldn't
>> be more robust than snapshot is.
>>
>> On Fri, Sep 2, 2016 at 7:03 AM, Sean Busbey <bus...@cloudera.com> wrote:
>>
>> > right, they're separate features but when asked about "robust
>> > backup/restore" (which is what I care about for this feature getting
>> > merged) things were pawned off on snapshots.
>> >
>> > Are they independent enough that we can get backup/restore tolerant to
>> > failures prior to merge to master? Prior to backport to branch-1?
>> >
>> > On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell <apurt...@apache.org>
>> > wrote:
>> > > I agree these are separate features FWIW
>> > >
>> > > On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov <
>> > vladrodio...@gmail.com>
>> > > wrote:
>> > >
>> > >> >> Do we have JIRA issue(s) covering making snapshots robust in the
>> face
>> > >> >> of monkeys?
>> > >>
>> > >> I would like to mention that "robust snapshots" and "table
>> > backup/restore"
>> > >> are totally separate features, but we have separate JIRA for fault
>> > >> tolerance (HBASE-14413).
>> > >>
>> > >> -Vlad
>> > >>
>> > >> On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>> > >>
>> > >> > Sean:
>> > >> > Please see HBASE-14413 for the last question.
>> > >> >
>> > >> > FYI
>> > >> >
>> > >> > On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey <bus...@cloudera.com>
>> > wrote:
>> > >> >
>> > >> > > On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
>> > >> > > <vladrodio...@gmail.com> wrote:
>> > >> > > > Not sure what do you mean, Andrew by "trying out the branch via
>> > the
>> > >> > IT",
>> > >> > > > but we do not recommend running this with monkey enabled.
>> > >> > > > It has not been tested in a such scenario yet and frankly
>> > speaking it
>> > >> > is
>> > >> > > > not supposed to work (snapshots will fail anyway and we
>> depends on
>> > >> > > > snapshots)
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> > > Also won't have time to test out the branch this week, but if
>> we're
>> > >> > > not going to handle failures do we have tools or guidance on
>> > >> > > recovering in the case of things falling over?
>> > >> > >
>> > >> > > Do we have JIRA issue(s) covering making snapshots robust in the
>> > face
>> > >> > > of monkeys?
>> > >> > >
>> > >> > > --
>> > >> > > busbey
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > >
>> > >- Andy
>> > >
>> > > Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> > > (via Tom White)
>> >
>> >
>> >
>> > --
>> > busbey
>> >
>>
>
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

Snapshot robustness is better now with introduction of region splits/merges
on/off feature. Region splits during snapshots was the major problem.

-Vlad

On Fri, Sep 2, 2016 at 8:12 AM, Vladimir Rodionov 
wrote:

> >>Are they independent enough that we can get backup/restore tolerant to
> >>failures prior to merge to master? Prior to backport to branch-1?
>
> As we stated already, snapshots are not part of the feature, snapshots has
> been merged into the master long time ago
> and as far as I understood - without requiring them to be 100% robust and
> fault tolerant and they are widely used in many production systems
> nevertheless. https://issues.apache.org/jira/browse/HBASE-14415 relies on
> Snapshots v2 but we can reconsider it, there are some thoughts how to make
> backups snapshotless.
>
> Backups are fault tolerant to some extent - in case of failure (and
> failures can happen) we clean everything up and do not leave system table
> in inconsistent state. Would it be enough, Sean Busbey?
>
> -Vlad
>
> On Fri, Sep 2, 2016 at 7:38 AM, Ted Yu  wrote:
>
>> We're continuing to make backup / restore more robust.
>> Work in progress (both are close to being integrated):
>>
>> HBASE-15565 Rewrite restore with Procedure V2
>> HBASE-15449 Support physical table layout change
>>
>> Since snapshot is dependency in the full backup, backup / restore wouldn't
>> be more robust than snapshot is.
>>
>> On Fri, Sep 2, 2016 at 7:03 AM, Sean Busbey  wrote:
>>
>> > right, they're separate features but when asked about "robust
>> > backup/restore" (which is what I care about for this feature getting
>> > merged) things were pawned off on snapshots.
>> >
>> > Are they independent enough that we can get backup/restore tolerant to
>> > failures prior to merge to master? Prior to backport to branch-1?
>> >
>> > On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell 
>> > wrote:
>> > > I agree these are separate features FWIW
>> > >
>> > > On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov <
>> > vladrodio...@gmail.com>
>> > > wrote:
>> > >
>> > >> >> Do we have JIRA issue(s) covering making snapshots robust in the
>> face
>> > >> >> of monkeys?
>> > >>
>> > >> I would like to mention that "robust snapshots" and "table
>> > backup/restore"
>> > >> are totally separate features, but we have separate JIRA for fault
>> > >> tolerance (HBASE-14413).
>> > >>
>> > >> -Vlad
>> > >>
>> > >> On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:
>> > >>
>> > >> > Sean:
>> > >> > Please see HBASE-14413 for the last question.
>> > >> >
>> > >> > FYI
>> > >> >
>> > >> > On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey 
>> > wrote:
>> > >> >
>> > >> > > On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
>> > >> > >  wrote:
>> > >> > > > Not sure what do you mean, Andrew by "trying out the branch via
>> > the
>> > >> > IT",
>> > >> > > > but we do not recommend running this with monkey enabled.
>> > >> > > > It has not been tested in a such scenario yet and frankly
>> > speaking it
>> > >> > is
>> > >> > > > not supposed to work (snapshots will fail anyway and we
>> depends on
>> > >> > > > snapshots)
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> > > Also won't have time to test out the branch this week, but if
>> we're
>> > >> > > not going to handle failures do we have tools or guidance on
>> > >> > > recovering in the case of things falling over?
>> > >> > >
>> > >> > > Do we have JIRA issue(s) covering making snapshots robust in the
>> > face
>> > >> > > of monkeys?
>> > >> > >
>> > >> > > --
>> > >> > > busbey
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > >
>> > >- Andy
>> > >
>> > > Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> > > (via Tom White)
>> >
>> >
>> >
>> > --
>> > busbey
>> >
>>
>
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>>Are they independent enough that we can get backup/restore tolerant to
>>failures prior to merge to master? Prior to backport to branch-1?

As we stated already, snapshots are not part of the feature, snapshots has
been merged into the master long time ago
and as far as I understood - without requiring them to be 100% robust and
fault tolerant and they are widely used in many production systems
nevertheless. https://issues.apache.org/jira/browse/HBASE-14415 relies on
Snapshots v2 but we can reconsider it, there are some thoughts how to make
backups snapshotless.

Backups are fault tolerant to some extent - in case of failure (and
failures can happen) we clean everything up and do not leave system table
in inconsistent state. Would it be enough, Sean Busbey?

-Vlad

On Fri, Sep 2, 2016 at 7:38 AM, Ted Yu  wrote:

> We're continuing to make backup / restore more robust.
> Work in progress (both are close to being integrated):
>
> HBASE-15565 Rewrite restore with Procedure V2
> HBASE-15449 Support physical table layout change
>
> Since snapshot is dependency in the full backup, backup / restore wouldn't
> be more robust than snapshot is.
>
> On Fri, Sep 2, 2016 at 7:03 AM, Sean Busbey  wrote:
>
> > right, they're separate features but when asked about "robust
> > backup/restore" (which is what I care about for this feature getting
> > merged) things were pawned off on snapshots.
> >
> > Are they independent enough that we can get backup/restore tolerant to
> > failures prior to merge to master? Prior to backport to branch-1?
> >
> > On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell 
> > wrote:
> > > I agree these are separate features FWIW
> > >
> > > On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov <
> > vladrodio...@gmail.com>
> > > wrote:
> > >
> > >> >> Do we have JIRA issue(s) covering making snapshots robust in the
> face
> > >> >> of monkeys?
> > >>
> > >> I would like to mention that "robust snapshots" and "table
> > backup/restore"
> > >> are totally separate features, but we have separate JIRA for fault
> > >> tolerance (HBASE-14413).
> > >>
> > >> -Vlad
> > >>
> > >> On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:
> > >>
> > >> > Sean:
> > >> > Please see HBASE-14413 for the last question.
> > >> >
> > >> > FYI
> > >> >
> > >> > On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey 
> > wrote:
> > >> >
> > >> > > On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
> > >> > >  wrote:
> > >> > > > Not sure what do you mean, Andrew by "trying out the branch via
> > the
> > >> > IT",
> > >> > > > but we do not recommend running this with monkey enabled.
> > >> > > > It has not been tested in a such scenario yet and frankly
> > speaking it
> > >> > is
> > >> > > > not supposed to work (snapshots will fail anyway and we depends
> on
> > >> > > > snapshots)
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > Also won't have time to test out the branch this week, but if
> we're
> > >> > > not going to handle failures do we have tools or guidance on
> > >> > > recovering in the case of things falling over?
> > >> > >
> > >> > > Do we have JIRA issue(s) covering making snapshots robust in the
> > face
> > >> > > of monkeys?
> > >> > >
> > >> > > --
> > >> > > busbey
> > >> > >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> >
> >
> >
> > --
> > busbey
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

We're continuing to make backup / restore more robust.
Work in progress (both are close to being integrated):

HBASE-15565 Rewrite restore with Procedure V2
HBASE-15449 Support physical table layout change

Since snapshot is dependency in the full backup, backup / restore wouldn't
be more robust than snapshot is.

On Fri, Sep 2, 2016 at 7:03 AM, Sean Busbey  wrote:

> right, they're separate features but when asked about "robust
> backup/restore" (which is what I care about for this feature getting
> merged) things were pawned off on snapshots.
>
> Are they independent enough that we can get backup/restore tolerant to
> failures prior to merge to master? Prior to backport to branch-1?
>
> On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell 
> wrote:
> > I agree these are separate features FWIW
> >
> > On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov <
> vladrodio...@gmail.com>
> > wrote:
> >
> >> >> Do we have JIRA issue(s) covering making snapshots robust in the face
> >> >> of monkeys?
> >>
> >> I would like to mention that "robust snapshots" and "table
> backup/restore"
> >> are totally separate features, but we have separate JIRA for fault
> >> tolerance (HBASE-14413).
> >>
> >> -Vlad
> >>
> >> On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:
> >>
> >> > Sean:
> >> > Please see HBASE-14413 for the last question.
> >> >
> >> > FYI
> >> >
> >> > On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey 
> wrote:
> >> >
> >> > > On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
> >> > >  wrote:
> >> > > > Not sure what do you mean, Andrew by "trying out the branch via
> the
> >> > IT",
> >> > > > but we do not recommend running this with monkey enabled.
> >> > > > It has not been tested in a such scenario yet and frankly
> speaking it
> >> > is
> >> > > > not supposed to work (snapshots will fail anyway and we depends on
> >> > > > snapshots)
> >> > > >
> >> > > >
> >> > >
> >> > > Also won't have time to test out the branch this week, but if we're
> >> > > not going to handle failures do we have tools or guidance on
> >> > > recovering in the case of things falling over?
> >> > >
> >> > > Do we have JIRA issue(s) covering making snapshots robust in the
> face
> >> > > of monkeys?
> >> > >
> >> > > --
> >> > > busbey
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>
>
>
> --
> busbey
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Sean Busbey]

right, they're separate features but when asked about "robust
backup/restore" (which is what I care about for this feature getting
merged) things were pawned off on snapshots.

Are they independent enough that we can get backup/restore tolerant to
failures prior to merge to master? Prior to backport to branch-1?

On Thu, Sep 1, 2016 at 1:11 PM, Andrew Purtell  wrote:
> I agree these are separate features FWIW
>
> On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov 
> wrote:
>
>> >> Do we have JIRA issue(s) covering making snapshots robust in the face
>> >> of monkeys?
>>
>> I would like to mention that "robust snapshots" and "table backup/restore"
>> are totally separate features, but we have separate JIRA for fault
>> tolerance (HBASE-14413).
>>
>> -Vlad
>>
>> On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:
>>
>> > Sean:
>> > Please see HBASE-14413 for the last question.
>> >
>> > FYI
>> >
>> > On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey  wrote:
>> >
>> > > On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
>> > >  wrote:
>> > > > Not sure what do you mean, Andrew by "trying out the branch via the
>> > IT",
>> > > > but we do not recommend running this with monkey enabled.
>> > > > It has not been tested in a such scenario yet and frankly speaking it
>> > is
>> > > > not supposed to work (snapshots will fail anyway and we depends on
>> > > > snapshots)
>> > > >
>> > > >
>> > >
>> > > Also won't have time to test out the branch this week, but if we're
>> > > not going to handle failures do we have tools or guidance on
>> > > recovering in the case of things falling over?
>> > >
>> > > Do we have JIRA issue(s) covering making snapshots robust in the face
>> > > of monkeys?
>> > >
>> > > --
>> > > busbey
>> > >
>> >
>>
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)



-- 
busbey

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Andrew Purtell]

I agree these are separate features FWIW

On Thu, Sep 1, 2016 at 11:09 AM, Vladimir Rodionov 
wrote:

> >> Do we have JIRA issue(s) covering making snapshots robust in the face
> >> of monkeys?
>
> I would like to mention that "robust snapshots" and "table backup/restore"
> are totally separate features, but we have separate JIRA for fault
> tolerance (HBASE-14413).
>
> -Vlad
>
> On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:
>
> > Sean:
> > Please see HBASE-14413 for the last question.
> >
> > FYI
> >
> > On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey  wrote:
> >
> > > On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
> > >  wrote:
> > > > Not sure what do you mean, Andrew by "trying out the branch via the
> > IT",
> > > > but we do not recommend running this with monkey enabled.
> > > > It has not been tested in a such scenario yet and frankly speaking it
> > is
> > > > not supposed to work (snapshots will fail anyway and we depends on
> > > > snapshots)
> > > >
> > > >
> > >
> > > Also won't have time to test out the branch this week, but if we're
> > > not going to handle failures do we have tools or guidance on
> > > recovering in the case of things falling over?
> > >
> > > Do we have JIRA issue(s) covering making snapshots robust in the face
> > > of monkeys?
> > >
> > > --
> > > busbey
> > >
> >
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Vladimir Rodionov]

>> Do we have JIRA issue(s) covering making snapshots robust in the face
>> of monkeys?

I would like to mention that "robust snapshots" and "table backup/restore"
are totally separate features, but we have separate JIRA for fault
tolerance (HBASE-14413).

-Vlad

On Thu, Sep 1, 2016 at 9:28 AM, Ted Yu  wrote:

> Sean:
> Please see HBASE-14413 for the last question.
>
> FYI
>
> On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey  wrote:
>
> > On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
> >  wrote:
> > > Not sure what do you mean, Andrew by "trying out the branch via the
> IT",
> > > but we do not recommend running this with monkey enabled.
> > > It has not been tested in a such scenario yet and frankly speaking it
> is
> > > not supposed to work (snapshots will fail anyway and we depends on
> > > snapshots)
> > >
> > >
> >
> > Also won't have time to test out the branch this week, but if we're
> > not going to handle failures do we have tools or guidance on
> > recovering in the case of things falling over?
> >
> > Do we have JIRA issue(s) covering making snapshots robust in the face
> > of monkeys?
> >
> > --
> > busbey
> >
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Ted Yu]

Sean:
Please see HBASE-14413 for the last question.

FYI

On Thu, Sep 1, 2016 at 9:24 AM, Sean Busbey  wrote:

> On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
>  wrote:
> > Not sure what do you mean, Andrew by "trying out the branch via the IT",
> > but we do not recommend running this with monkey enabled.
> > It has not been tested in a such scenario yet and frankly speaking it is
> > not supposed to work (snapshots will fail anyway and we depends on
> > snapshots)
> >
> >
>
> Also won't have time to test out the branch this week, but if we're
> not going to handle failures do we have tools or guidance on
> recovering in the case of things falling over?
>
> Do we have JIRA issue(s) covering making snapshots robust in the face
> of monkeys?
>
> --
> busbey
>

Re: [DISCUSSION] Merge Backup / Restore - Branch HBASE-7912

[Sean Busbey]

On Sat, Aug 20, 2016 at 12:59 PM, Vladimir Rodionov
 wrote:
> Not sure what do you mean, Andrew by "trying out the branch via the IT",
> but we do not recommend running this with monkey enabled.
> It has not been tested in a such scenario yet and frankly speaking it is
> not supposed to work (snapshots will fail anyway and we depends on
> snapshots)
>
>

Also won't have time to test out the branch this week, but if we're
not going to handle failures do we have tools or guidance on
recovering in the case of things falling over?

Do we have JIRA issue(s) covering making snapshots robust in the face
of monkeys?

-- 
busbey