Re: DISCUSS: Remove hbase-backup from master?
Sure. I have assigned it to myself. Will look into it. Last time I checked, I did not find any failed tests and it was not hadoop3 --- Mallikarjun On Fri, May 14, 2021 at 2:34 AM Nick Dimiduk wrote: > Hi Mallikarjun, > > I just saw a bunch of backup tests fail on an unrelated PR build. I filed > HBASE-25888 and uploaded some logs. I have the full test-logs.zip file, but > it's too big to upload to jira. I linked it from the Jira, but the archive > will disappear when the PR is eventually closed. I would ping you from > Jira, but I didn't find any Jira user that seemed likely to be your > account. Would you mind taking a look? > > Thanks, > Nick > > On Wed, Dec 23, 2020 at 8:20 PM Mallikarjun > wrote: > > > Yea. I have noticed that. > > > > I have some patches ready and have to add unit tests. Will start raising > in > > a couple of weeks time. > > --- > > Mallikarjun > > > > > > On Thu, Dec 24, 2020 at 7:48 AM 张铎(Duo Zhang) > > wrote: > > > > > The UTs in the backup module are easy to fail with NPE, I've seen this > in > > > several pre commit results. > > > > > > Any progress here? > > > > > > [email protected] 于2020年12月3日周四 > > > 上午9:58写道： > > > > > > > On Tue, Dec 1, 2020 at 11:26 PM Stack wrote: > > > > > > > > > On Tue, Dec 1, 2020 at 6:38 AM [email protected] < > > > > > [email protected]> wrote: > > > > > > > > > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey > > > wrote: > > > > > > > > > > > > > One reason for moving it out of core is that at some point we > > will > > > > have > > > > > > > hbase 3 releases. Right now hbase 3 targeted things are in the > > > master > > > > > > > branch. If the feature is not sufficiently ready to be in a > > release > > > > > then > > > > > > > when the time for HBase 3 releases comes we'll have to do work > to > > > > pull > > > > > it > > > > > > > out of the release branch. If the feature needs to stay in the > > core > > > > > repo > > > > > > > that will necessitate that hbase 3 have a branch distinct from > > the > > > > > master > > > > > > > branch (which may or may not happen anyways). At that point we > > risk > > > > > > having > > > > > > > to do the work to remove the feature from a release branch > again > > > come > > > > > > hbase > > > > > > > 4. > > > > > > > > > > > > > > > > > > > > > I think a lot of this is moot if you'd like to start providing > > > > patches > > > > > > for > > > > > > > the things you've needed to use it. If there are gaps that you > > > think > > > > > can > > > > > > > trip folks up maybe we could label it "experimental" to provide > > > > better > > > > > > > context for others. > > > > > > > > > > > > > > > > > > > I will start putting effort in maintaining this feature. > > > > > > > > > > > > > > > > > FYI, seems like backup has a bunch of flakey tests. Might be worth > > > > looking > > > > > at. > > > > > > > > > > > > > Any reference I can get. They seem fine when I run tests. > > > > > > > > > > > > > Thanks, > > > > > S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 07:48 [email protected] < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack > > wrote: > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > Inline > > > > > > > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > We are allowed to debate if this should be in the tree > or > > > > not. > > > > > > > Given > > > > > > > > > the > > > > > > > > > > > lack of interest, as evidenced by incomplete state, > lack > > of > > > > > > > release, > > > > > > > > > and > > > > > > > > > > > lack of contribution, it is more than fair to discuss > > > > removal. > > > > > > > > > > > > > > > > > > > > > > Here is my take: First of all, it is not released. > There > > is > > > > no > > > > > > > > implied > > > > > > > > > > > roadmap or support. Second, there do not seem to be any > > > > active > > > > > > > > > > maintainers > > > > > > > > > > > or volunteers as such. Third, unless someone shows up > > with > > > > more > > > > > > > > patches > > > > > > > > > > for > > > > > > > > > > > it there will be no polish or maturing, there can be no > > > > > > > expectations > > > > > > > > of > > > > > > > > > > > further improvement. > > > > > > > > > > > > > > > > > > > > > > That said, this is open source. New code contribution > > will > > > > > change > > > > > > > the > > > > > > > > > > > facts as they stand. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If there is an agreement upon the direction I am willing > to > > > > > provide > > > > > > > > > patches > > > > > > > > > > to take this feature forward. > > > > >
Re: DISCUSS: Remove hbase-backup from master?
Hi Mallikarjun, I just saw a bunch of backup tests fail on an unrelated PR build. I filed HBASE-25888 and uploaded some logs. I have the full test-logs.zip file, but it's too big to upload to jira. I linked it from the Jira, but the archive will disappear when the PR is eventually closed. I would ping you from Jira, but I didn't find any Jira user that seemed likely to be your account. Would you mind taking a look? Thanks, Nick On Wed, Dec 23, 2020 at 8:20 PM Mallikarjun wrote: > Yea. I have noticed that. > > I have some patches ready and have to add unit tests. Will start raising in > a couple of weeks time. > --- > Mallikarjun > > > On Thu, Dec 24, 2020 at 7:48 AM 张铎(Duo Zhang) > wrote: > > > The UTs in the backup module are easy to fail with NPE, I've seen this in > > several pre commit results. > > > > Any progress here? > > > > [email protected] 于2020年12月3日周四 > > 上午9:58写道： > > > > > On Tue, Dec 1, 2020 at 11:26 PM Stack wrote: > > > > > > > On Tue, Dec 1, 2020 at 6:38 AM [email protected] < > > > > [email protected]> wrote: > > > > > > > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey > > wrote: > > > > > > > > > > > One reason for moving it out of core is that at some point we > will > > > have > > > > > > hbase 3 releases. Right now hbase 3 targeted things are in the > > master > > > > > > branch. If the feature is not sufficiently ready to be in a > release > > > > then > > > > > > when the time for HBase 3 releases comes we'll have to do work to > > > pull > > > > it > > > > > > out of the release branch. If the feature needs to stay in the > core > > > > repo > > > > > > that will necessitate that hbase 3 have a branch distinct from > the > > > > master > > > > > > branch (which may or may not happen anyways). At that point we > risk > > > > > having > > > > > > to do the work to remove the feature from a release branch again > > come > > > > > hbase > > > > > > 4. > > > > > > > > > > > > > > > > > > I think a lot of this is moot if you'd like to start providing > > > patches > > > > > for > > > > > > the things you've needed to use it. If there are gaps that you > > think > > > > can > > > > > > trip folks up maybe we could label it "experimental" to provide > > > better > > > > > > context for others. > > > > > > > > > > > > > > > > I will start putting effort in maintaining this feature. > > > > > > > > > > > > > > FYI, seems like backup has a bunch of flakey tests. Might be worth > > > looking > > > > at. > > > > > > > > > > Any reference I can get. They seem fine when I run tests. > > > > > > > > > > Thanks, > > > > S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 07:48 [email protected] < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack > wrote: > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > Inline > > > > > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > We are allowed to debate if this should be in the tree or > > > not. > > > > > > Given > > > > > > > > the > > > > > > > > > > lack of interest, as evidenced by incomplete state, lack > of > > > > > > release, > > > > > > > > and > > > > > > > > > > lack of contribution, it is more than fair to discuss > > > removal. > > > > > > > > > > > > > > > > > > > > Here is my take: First of all, it is not released. There > is > > > no > > > > > > > implied > > > > > > > > > > roadmap or support. Second, there do not seem to be any > > > active > > > > > > > > > maintainers > > > > > > > > > > or volunteers as such. Third, unless someone shows up > with > > > more > > > > > > > patches > > > > > > > > > for > > > > > > > > > > it there will be no polish or maturing, there can be no > > > > > > expectations > > > > > > > of > > > > > > > > > > further improvement. > > > > > > > > > > > > > > > > > > > > That said, this is open source. New code contribution > will > > > > change > > > > > > the > > > > > > > > > > facts as they stand. > > > > > > > > > > > > > > > > > > > > > > > > > > > If there is an agreement upon the direction I am willing to > > > > provide > > > > > > > > patches > > > > > > > > > to take this feature forward. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What about the changes you made to make it work? > > > > > > > > > > > > > > > > > > > > > > I'd be interested in patches that take this feature out of > > core. > > > > How > > > > > > hard > > > > > > > > do you think that would be? See the items Duo list above for > a > > > > start > > > > > on > > > > > > > > obstacles to tackle? > > > > > > > > > > > > > > > > > > > > > > I am yet to be very familiar with the code to comment on the > > effort > > > > > > >
Re: DISCUSS: Remove hbase-backup from master?
Yea. I have noticed that. I have some patches ready and have to add unit tests. Will start raising in a couple of weeks time. --- Mallikarjun On Thu, Dec 24, 2020 at 7:48 AM 张铎(Duo Zhang) wrote: > The UTs in the backup module are easy to fail with NPE, I've seen this in > several pre commit results. > > Any progress here? > > [email protected] 于2020年12月3日周四 > 上午9:58写道： > > > On Tue, Dec 1, 2020 at 11:26 PM Stack wrote: > > > > > On Tue, Dec 1, 2020 at 6:38 AM [email protected] < > > > [email protected]> wrote: > > > > > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey > wrote: > > > > > > > > > One reason for moving it out of core is that at some point we will > > have > > > > > hbase 3 releases. Right now hbase 3 targeted things are in the > master > > > > > branch. If the feature is not sufficiently ready to be in a release > > > then > > > > > when the time for HBase 3 releases comes we'll have to do work to > > pull > > > it > > > > > out of the release branch. If the feature needs to stay in the core > > > repo > > > > > that will necessitate that hbase 3 have a branch distinct from the > > > master > > > > > branch (which may or may not happen anyways). At that point we risk > > > > having > > > > > to do the work to remove the feature from a release branch again > come > > > > hbase > > > > > 4. > > > > > > > > > > > > > > > I think a lot of this is moot if you'd like to start providing > > patches > > > > for > > > > > the things you've needed to use it. If there are gaps that you > think > > > can > > > > > trip folks up maybe we could label it "experimental" to provide > > better > > > > > context for others. > > > > > > > > > > > > > I will start putting effort in maintaining this feature. > > > > > > > > > > > FYI, seems like backup has a bunch of flakey tests. Might be worth > > looking > > > at. > > > > > > > Any reference I can get. They seem fine when I run tests. > > > > > > > Thanks, > > > S > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 07:48 [email protected] < > > > > > [email protected]> wrote: > > > > > > > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack wrote: > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Inline > > > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > We are allowed to debate if this should be in the tree or > > not. > > > > > Given > > > > > > > the > > > > > > > > > lack of interest, as evidenced by incomplete state, lack of > > > > > release, > > > > > > > and > > > > > > > > > lack of contribution, it is more than fair to discuss > > removal. > > > > > > > > > > > > > > > > > > Here is my take: First of all, it is not released. There is > > no > > > > > > implied > > > > > > > > > roadmap or support. Second, there do not seem to be any > > active > > > > > > > > maintainers > > > > > > > > > or volunteers as such. Third, unless someone shows up with > > more > > > > > > patches > > > > > > > > for > > > > > > > > > it there will be no polish or maturing, there can be no > > > > > expectations > > > > > > of > > > > > > > > > further improvement. > > > > > > > > > > > > > > > > > > That said, this is open source. New code contribution will > > > change > > > > > the > > > > > > > > > facts as they stand. > > > > > > > > > > > > > > > > > > > > > > > > If there is an agreement upon the direction I am willing to > > > provide > > > > > > > patches > > > > > > > > to take this feature forward. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What about the changes you made to make it work? > > > > > > > > > > > > > > > > > > > I'd be interested in patches that take this feature out of > core. > > > How > > > > > hard > > > > > > > do you think that would be? See the items Duo list above for a > > > start > > > > on > > > > > > > obstacles to tackle? > > > > > > > > > > > > > > > > > > > I am yet to be very familiar with the code to comment on the > effort > > > > > > required to take it out of the core. Backport and bug fixes > > required > > > > very > > > > > > shallow understanding of the code and only some parts of it. > > > > > > > > > > > > I am not very clear on why some strong opinions on moving backup > > out > > > of > > > > > the > > > > > > core? Is there something defined on what should be operator > tools? > > > > > > > > > > > > Quoting what Duo Zhang said > > > > > > > > > > > > You can see the code under the > > org.apache.hadoop.hbase.backup.master > > > > > > > package, we need to set up log cleaner at master side, and > also, > > > > > > > the LogRollMasterProcedureManager class needs > MasterServices(not > > > > > > > MasterRpcServices), which means it must be used in the same > > process > > > > > with > > > > > > > HMaster. > >
Re: DISCUSS: Remove hbase-backup from master?
The UTs in the backup module are easy to fail with NPE, I've seen this in several pre commit results. Any progress here? [email protected] 于2020年12月3日周四 上午9:58写道： > On Tue, Dec 1, 2020 at 11:26 PM Stack wrote: > > > On Tue, Dec 1, 2020 at 6:38 AM [email protected] < > > [email protected]> wrote: > > > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey wrote: > > > > > > > One reason for moving it out of core is that at some point we will > have > > > > hbase 3 releases. Right now hbase 3 targeted things are in the master > > > > branch. If the feature is not sufficiently ready to be in a release > > then > > > > when the time for HBase 3 releases comes we'll have to do work to > pull > > it > > > > out of the release branch. If the feature needs to stay in the core > > repo > > > > that will necessitate that hbase 3 have a branch distinct from the > > master > > > > branch (which may or may not happen anyways). At that point we risk > > > having > > > > to do the work to remove the feature from a release branch again come > > > hbase > > > > 4. > > > > > > > > > > > > I think a lot of this is moot if you'd like to start providing > patches > > > for > > > > the things you've needed to use it. If there are gaps that you think > > can > > > > trip folks up maybe we could label it "experimental" to provide > better > > > > context for others. > > > > > > > > > > I will start putting effort in maintaining this feature. > > > > > > > > FYI, seems like backup has a bunch of flakey tests. Might be worth > looking > > at. > > > > Any reference I can get. They seem fine when I run tests. > > > > Thanks, > > S > > > > > > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 07:48 [email protected] < > > > > [email protected]> wrote: > > > > > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack wrote: > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Inline > > > > > > > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > We are allowed to debate if this should be in the tree or > not. > > > > Given > > > > > > the > > > > > > > > lack of interest, as evidenced by incomplete state, lack of > > > > release, > > > > > > and > > > > > > > > lack of contribution, it is more than fair to discuss > removal. > > > > > > > > > > > > > > > > Here is my take: First of all, it is not released. There is > no > > > > > implied > > > > > > > > roadmap or support. Second, there do not seem to be any > active > > > > > > > maintainers > > > > > > > > or volunteers as such. Third, unless someone shows up with > more > > > > > patches > > > > > > > for > > > > > > > > it there will be no polish or maturing, there can be no > > > > expectations > > > > > of > > > > > > > > further improvement. > > > > > > > > > > > > > > > > That said, this is open source. New code contribution will > > change > > > > the > > > > > > > > facts as they stand. > > > > > > > > > > > > > > > > > > > > > If there is an agreement upon the direction I am willing to > > provide > > > > > > patches > > > > > > > to take this feature forward. > > > > > > > > > > > > > > > > > > > > > > > > > > What about the changes you made to make it work? > > > > > > > > > > > > > > > > I'd be interested in patches that take this feature out of core. > > How > > > > hard > > > > > > do you think that would be? See the items Duo list above for a > > start > > > on > > > > > > obstacles to tackle? > > > > > > > > > > > > > > > > I am yet to be very familiar with the code to comment on the effort > > > > > required to take it out of the core. Backport and bug fixes > required > > > very > > > > > shallow understanding of the code and only some parts of it. > > > > > > > > > > I am not very clear on why some strong opinions on moving backup > out > > of > > > > the > > > > > core? Is there something defined on what should be operator tools? > > > > > > > > > > Quoting what Duo Zhang said > > > > > > > > > > You can see the code under the > org.apache.hadoop.hbase.backup.master > > > > > > package, we need to set up log cleaner at master side, and also, > > > > > > the LogRollMasterProcedureManager class needs MasterServices(not > > > > > > MasterRpcServices), which means it must be used in the same > process > > > > with > > > > > > HMaster. > > > > > > > > > > . > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 30, 2020, at 5:52 PM, [email protected] > wrote: > > > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack > > > wrote: > > > > > > > > > > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] > < > > > > > > > > >>> [email protected]> wrote: > > > > > > > > >>> > > > > > > > > >>> Inl
Re: DISCUSS: Remove hbase-backup from master?
On Tue, Dec 1, 2020 at 11:26 PM Stack wrote: > On Tue, Dec 1, 2020 at 6:38 AM [email protected] < > [email protected]> wrote: > > > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey wrote: > > > > > One reason for moving it out of core is that at some point we will have > > > hbase 3 releases. Right now hbase 3 targeted things are in the master > > > branch. If the feature is not sufficiently ready to be in a release > then > > > when the time for HBase 3 releases comes we'll have to do work to pull > it > > > out of the release branch. If the feature needs to stay in the core > repo > > > that will necessitate that hbase 3 have a branch distinct from the > master > > > branch (which may or may not happen anyways). At that point we risk > > having > > > to do the work to remove the feature from a release branch again come > > hbase > > > 4. > > > > > > > > > I think a lot of this is moot if you'd like to start providing patches > > for > > > the things you've needed to use it. If there are gaps that you think > can > > > trip folks up maybe we could label it "experimental" to provide better > > > context for others. > > > > > > > I will start putting effort in maintaining this feature. > > > > > FYI, seems like backup has a bunch of flakey tests. Might be worth looking > at. > Any reference I can get. They seem fine when I run tests. > Thanks, > S > > > > > > > > > > > > On Tue, Dec 1, 2020, 07:48 [email protected] < > > > [email protected]> wrote: > > > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack wrote: > > > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > > > > [email protected]> wrote: > > > > > > > > > > > Inline > > > > > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > We are allowed to debate if this should be in the tree or not. > > > Given > > > > > the > > > > > > > lack of interest, as evidenced by incomplete state, lack of > > > release, > > > > > and > > > > > > > lack of contribution, it is more than fair to discuss removal. > > > > > > > > > > > > > > Here is my take: First of all, it is not released. There is no > > > > implied > > > > > > > roadmap or support. Second, there do not seem to be any active > > > > > > maintainers > > > > > > > or volunteers as such. Third, unless someone shows up with more > > > > patches > > > > > > for > > > > > > > it there will be no polish or maturing, there can be no > > > expectations > > > > of > > > > > > > further improvement. > > > > > > > > > > > > > > That said, this is open source. New code contribution will > change > > > the > > > > > > > facts as they stand. > > > > > > > > > > > > > > > > > > If there is an agreement upon the direction I am willing to > provide > > > > > patches > > > > > > to take this feature forward. > > > > > > > > > > > > > > > > > > > > > > What about the changes you made to make it work? > > > > > > > > > > > > > I'd be interested in patches that take this feature out of core. > How > > > hard > > > > > do you think that would be? See the items Duo list above for a > start > > on > > > > > obstacles to tackle? > > > > > > > > > > > > > I am yet to be very familiar with the code to comment on the effort > > > > required to take it out of the core. Backport and bug fixes required > > very > > > > shallow understanding of the code and only some parts of it. > > > > > > > > I am not very clear on why some strong opinions on moving backup out > of > > > the > > > > core? Is there something defined on what should be operator tools? > > > > > > > > Quoting what Duo Zhang said > > > > > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > > > > package, we need to set up log cleaner at master side, and also, > > > > > the LogRollMasterProcedureManager class needs MasterServices(not > > > > > MasterRpcServices), which means it must be used in the same process > > > with > > > > > HMaster. > > > > > > > > . > > > > > > > > > > > > > > > > > > Thanks, > > > > > S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack > > wrote: > > > > > > > > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > > > > > > > >>> [email protected]> wrote: > > > > > > > >>> > > > > > > > >>> Inline. > > > > > > > >>> > > > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) < > > > > [email protected] > > > > > > > > > > > > > >> wrote: > > > > > > > >>> > > > > > > > I'm afraid it is not easy to be moved to > > hbase-operator-tools. > > > > > > > > > > > > > > You can see the code under the > > > > > org.apache.hadoop.hbase.backup.master > > > > > > > package, we need to set up log cleaner at master side, and > > > also, > > > > > > > the LogRollMaster
Re: DISCUSS: Remove hbase-backup from master?
On Tue, Dec 1, 2020 at 6:38 AM [email protected] < [email protected]> wrote: > On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey wrote: > > > One reason for moving it out of core is that at some point we will have > > hbase 3 releases. Right now hbase 3 targeted things are in the master > > branch. If the feature is not sufficiently ready to be in a release then > > when the time for HBase 3 releases comes we'll have to do work to pull it > > out of the release branch. If the feature needs to stay in the core repo > > that will necessitate that hbase 3 have a branch distinct from the master > > branch (which may or may not happen anyways). At that point we risk > having > > to do the work to remove the feature from a release branch again come > hbase > > 4. > > > > > > I think a lot of this is moot if you'd like to start providing patches > for > > the things you've needed to use it. If there are gaps that you think can > > trip folks up maybe we could label it "experimental" to provide better > > context for others. > > > > I will start putting effort in maintaining this feature. > > FYI, seems like backup has a bunch of flakey tests. Might be worth looking at. Thanks, S > > > > > On Tue, Dec 1, 2020, 07:48 [email protected] < > > [email protected]> wrote: > > > > > On Tue, Dec 1, 2020 at 12:14 PM Stack wrote: > > > > > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > > > [email protected]> wrote: > > > > > > > > > Inline > > > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > We are allowed to debate if this should be in the tree or not. > > Given > > > > the > > > > > > lack of interest, as evidenced by incomplete state, lack of > > release, > > > > and > > > > > > lack of contribution, it is more than fair to discuss removal. > > > > > > > > > > > > Here is my take: First of all, it is not released. There is no > > > implied > > > > > > roadmap or support. Second, there do not seem to be any active > > > > > maintainers > > > > > > or volunteers as such. Third, unless someone shows up with more > > > patches > > > > > for > > > > > > it there will be no polish or maturing, there can be no > > expectations > > > of > > > > > > further improvement. > > > > > > > > > > > > That said, this is open source. New code contribution will change > > the > > > > > > facts as they stand. > > > > > > > > > > > > > > > If there is an agreement upon the direction I am willing to provide > > > > patches > > > > > to take this feature forward. > > > > > > > > > > > > > > > > > > What about the changes you made to make it work? > > > > > > > > > > I'd be interested in patches that take this feature out of core. How > > hard > > > > do you think that would be? See the items Duo list above for a start > on > > > > obstacles to tackle? > > > > > > > > > > I am yet to be very familiar with the code to comment on the effort > > > required to take it out of the core. Backport and bug fixes required > very > > > shallow understanding of the code and only some parts of it. > > > > > > I am not very clear on why some strong opinions on moving backup out of > > the > > > core? Is there something defined on what should be operator tools? > > > > > > Quoting what Duo Zhang said > > > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > > > package, we need to set up log cleaner at master side, and also, > > > > the LogRollMasterProcedureManager class needs MasterServices(not > > > > MasterRpcServices), which means it must be used in the same process > > with > > > > HMaster. > > > > > > . > > > > > > > > > > > > > > Thanks, > > > > S > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > > > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack > wrote: > > > > > > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > > > > > > >>> [email protected]> wrote: > > > > > > >>> > > > > > > >>> Inline. > > > > > > >>> > > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) < > > > [email protected] > > > > > > > > > > > >> wrote: > > > > > > >>> > > > > > > I'm afraid it is not easy to be moved to > hbase-operator-tools. > > > > > > > > > > > > You can see the code under the > > > > org.apache.hadoop.hbase.backup.master > > > > > > package, we need to set up log cleaner at master side, and > > also, > > > > > > the LogRollMasterProcedureManager class needs > > MasterServices(not > > > > > > MasterRpcServices), which means it must be used in the same > > > > process > > > > > > >> with > > > > > > HMaster. > > > > > > > > > > > > And I'm OK with purging this feature, especially if there is > > no > > > > > > >> developer > > > > > > who wants to maintain it. > > > > > > > > > > > > > > > > > > >>> Is the
Re: DISCUSS: Remove hbase-backup from master?
On Tue, Dec 1, 2020 at 7:34 PM Sean Busbey wrote: > One reason for moving it out of core is that at some point we will have > hbase 3 releases. Right now hbase 3 targeted things are in the master > branch. If the feature is not sufficiently ready to be in a release then > when the time for HBase 3 releases comes we'll have to do work to pull it > out of the release branch. If the feature needs to stay in the core repo > that will necessitate that hbase 3 have a branch distinct from the master > branch (which may or may not happen anyways). At that point we risk having > to do the work to remove the feature from a release branch again come hbase > 4. > > > I think a lot of this is moot if you'd like to start providing patches for > the things you've needed to use it. If there are gaps that you think can > trip folks up maybe we could label it "experimental" to provide better > context for others. > I will start putting effort in maintaining this feature. > > On Tue, Dec 1, 2020, 07:48 [email protected] < > [email protected]> wrote: > > > On Tue, Dec 1, 2020 at 12:14 PM Stack wrote: > > > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > > [email protected]> wrote: > > > > > > > Inline > > > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell < > [email protected] > > > > > > > wrote: > > > > > > > > > We are allowed to debate if this should be in the tree or not. > Given > > > the > > > > > lack of interest, as evidenced by incomplete state, lack of > release, > > > and > > > > > lack of contribution, it is more than fair to discuss removal. > > > > > > > > > > Here is my take: First of all, it is not released. There is no > > implied > > > > > roadmap or support. Second, there do not seem to be any active > > > > maintainers > > > > > or volunteers as such. Third, unless someone shows up with more > > patches > > > > for > > > > > it there will be no polish or maturing, there can be no > expectations > > of > > > > > further improvement. > > > > > > > > > > That said, this is open source. New code contribution will change > the > > > > > facts as they stand. > > > > > > > > > > > > If there is an agreement upon the direction I am willing to provide > > > patches > > > > to take this feature forward. > > > > > > > > > > > > > > What about the changes you made to make it work? > > > > > > > I'd be interested in patches that take this feature out of core. How > hard > > > do you think that would be? See the items Duo list above for a start on > > > obstacles to tackle? > > > > > > > I am yet to be very familiar with the code to comment on the effort > > required to take it out of the core. Backport and bug fixes required very > > shallow understanding of the code and only some parts of it. > > > > I am not very clear on why some strong opinions on moving backup out of > the > > core? Is there something defined on what should be operator tools? > > > > Quoting what Duo Zhang said > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > > package, we need to set up log cleaner at master side, and also, > > > the LogRollMasterProcedureManager class needs MasterServices(not > > > MasterRpcServices), which means it must be used in the same process > with > > > HMaster. > > > > . > > > > > > > > > > Thanks, > > > S > > > > > > > > > > > > > > > > > > > > > > > > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > > > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack wrote: > > > > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > > > > > >>> [email protected]> wrote: > > > > > >>> > > > > > >>> Inline. > > > > > >>> > > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) < > > [email protected] > > > > > > > > > >> wrote: > > > > > >>> > > > > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > > > > > > > You can see the code under the > > > org.apache.hadoop.hbase.backup.master > > > > > package, we need to set up log cleaner at master side, and > also, > > > > > the LogRollMasterProcedureManager class needs > MasterServices(not > > > > > MasterRpcServices), which means it must be used in the same > > > process > > > > > >> with > > > > > HMaster. > > > > > > > > > > And I'm OK with purging this feature, especially if there is > no > > > > > >> developer > > > > > who wants to maintain it. > > > > > > > > > > > > > > > >>> Is the sole reason to purge this feature is no developer > > > volunteering > > > > > to > > > > > >>> maintain this? > > > > > >>> > > > > > >>> > > > > > >> I'd suggest this feature does not belong in core at all. > > > > > >> > > > > > >> And some of us have spent time reviewing the feature in the past > > and > > > > > back > > > > > >> then determined it needed lots of finishing work, test, and > polish > > > -- > > > > > none > > > > > >> of which it seems to have gotten s
Re: DISCUSS: Remove hbase-backup from master?
One reason for moving it out of core is that at some point we will have hbase 3 releases. Right now hbase 3 targeted things are in the master branch. If the feature is not sufficiently ready to be in a release then when the time for HBase 3 releases comes we'll have to do work to pull it out of the release branch. If the feature needs to stay in the core repo that will necessitate that hbase 3 have a branch distinct from the master branch (which may or may not happen anyways). At that point we risk having to do the work to remove the feature from a release branch again come hbase 4. I think a lot of this is moot if you'd like to start providing patches for the things you've needed to use it. If there are gaps that you think can trip folks up maybe we could label it "experimental" to provide better context for others. On Tue, Dec 1, 2020, 07:48 [email protected] < [email protected]> wrote: > On Tue, Dec 1, 2020 at 12:14 PM Stack wrote: > > > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > > [email protected]> wrote: > > > > > Inline > > > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell > > > > wrote: > > > > > > > We are allowed to debate if this should be in the tree or not. Given > > the > > > > lack of interest, as evidenced by incomplete state, lack of release, > > and > > > > lack of contribution, it is more than fair to discuss removal. > > > > > > > > Here is my take: First of all, it is not released. There is no > implied > > > > roadmap or support. Second, there do not seem to be any active > > > maintainers > > > > or volunteers as such. Third, unless someone shows up with more > patches > > > for > > > > it there will be no polish or maturing, there can be no expectations > of > > > > further improvement. > > > > > > > > That said, this is open source. New code contribution will change the > > > > facts as they stand. > > > > > > > > > If there is an agreement upon the direction I am willing to provide > > patches > > > to take this feature forward. > > > > > > > > > > What about the changes you made to make it work? > > > > I'd be interested in patches that take this feature out of core. How hard > > do you think that would be? See the items Duo list above for a start on > > obstacles to tackle? > > > > I am yet to be very familiar with the code to comment on the effort > required to take it out of the core. Backport and bug fixes required very > shallow understanding of the code and only some parts of it. > > I am not very clear on why some strong opinions on moving backup out of the > core? Is there something defined on what should be operator tools? > > Quoting what Duo Zhang said > > You can see the code under the org.apache.hadoop.hbase.backup.master > > package, we need to set up log cleaner at master side, and also, > > the LogRollMasterProcedureManager class needs MasterServices(not > > MasterRpcServices), which means it must be used in the same process with > > HMaster. > > . > > > > > > Thanks, > > S > > > > > > > > > > > > > > > > > > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack wrote: > > > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > > > > >>> [email protected]> wrote: > > > > >>> > > > > >>> Inline. > > > > >>> > > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) < > [email protected] > > > > > > > >> wrote: > > > > >>> > > > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > > > > > You can see the code under the > > org.apache.hadoop.hbase.backup.master > > > > package, we need to set up log cleaner at master side, and also, > > > > the LogRollMasterProcedureManager class needs MasterServices(not > > > > MasterRpcServices), which means it must be used in the same > > process > > > > >> with > > > > HMaster. > > > > > > > > And I'm OK with purging this feature, especially if there is no > > > > >> developer > > > > who wants to maintain it. > > > > > > > > > > > > >>> Is the sole reason to purge this feature is no developer > > volunteering > > > > to > > > > >>> maintain this? > > > > >>> > > > > >>> > > > > >> I'd suggest this feature does not belong in core at all. > > > > >> > > > > >> And some of us have spent time reviewing the feature in the past > and > > > > back > > > > >> then determined it needed lots of finishing work, test, and polish > > -- > > > > none > > > > >> of which it seems to have gotten since commit. > > > > >> > > > > >> You are using it Mallik? Tell us more please. How is it working > for > > > you? > > > > >> > > > > >> > > > > > It has been a decent v1 release for us. As pointed out by others, > > > feature > > > > > should be polished and matured without debating if backup should be > > > part > > > > of > > > > > core or not. Some of notable points are > > > > > > > > > > - Limitations like serial backup s
Re: DISCUSS: Remove hbase-backup from master?
On Tue, Dec 1, 2020 at 12:14 PM Stack wrote: > On Mon, Nov 30, 2020 at 9:30 PM [email protected] < > [email protected]> wrote: > > > Inline > > > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell > > wrote: > > > > > We are allowed to debate if this should be in the tree or not. Given > the > > > lack of interest, as evidenced by incomplete state, lack of release, > and > > > lack of contribution, it is more than fair to discuss removal. > > > > > > Here is my take: First of all, it is not released. There is no implied > > > roadmap or support. Second, there do not seem to be any active > > maintainers > > > or volunteers as such. Third, unless someone shows up with more patches > > for > > > it there will be no polish or maturing, there can be no expectations of > > > further improvement. > > > > > > That said, this is open source. New code contribution will change the > > > facts as they stand. > > > > > > If there is an agreement upon the direction I am willing to provide > patches > > to take this feature forward. > > > > > > What about the changes you made to make it work? > I'd be interested in patches that take this feature out of core. How hard > do you think that would be? See the items Duo list above for a start on > obstacles to tackle? > I am yet to be very familiar with the code to comment on the effort required to take it out of the core. Backport and bug fixes required very shallow understanding of the code and only some parts of it. I am not very clear on why some strong opinions on moving backup out of the core? Is there something defined on what should be operator tools? Quoting what Duo Zhang said You can see the code under the org.apache.hadoop.hbase.backup.master > package, we need to set up log cleaner at master side, and also, > the LogRollMasterProcedureManager class needs MasterServices(not > MasterRpcServices), which means it must be used in the same process with > HMaster. . > > Thanks, > S > > > > > > > > > > > > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack wrote: > > > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > > > >>> [email protected]> wrote: > > > >>> > > > >>> Inline. > > > >>> > > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) > > > > >> wrote: > > > >>> > > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > > > You can see the code under the > org.apache.hadoop.hbase.backup.master > > > package, we need to set up log cleaner at master side, and also, > > > the LogRollMasterProcedureManager class needs MasterServices(not > > > MasterRpcServices), which means it must be used in the same > process > > > >> with > > > HMaster. > > > > > > And I'm OK with purging this feature, especially if there is no > > > >> developer > > > who wants to maintain it. > > > > > > > > > >>> Is the sole reason to purge this feature is no developer > volunteering > > > to > > > >>> maintain this? > > > >>> > > > >>> > > > >> I'd suggest this feature does not belong in core at all. > > > >> > > > >> And some of us have spent time reviewing the feature in the past and > > > back > > > >> then determined it needed lots of finishing work, test, and polish > -- > > > none > > > >> of which it seems to have gotten since commit. > > > >> > > > >> You are using it Mallik? Tell us more please. How is it working for > > you? > > > >> > > > >> > > > > It has been a decent v1 release for us. As pointed out by others, > > feature > > > > should be polished and matured without debating if backup should be > > part > > > of > > > > core or not. Some of notable points are > > > > > > > > - Limitations like serial backup set runs per deployment bothered > us. > > > > - Minor logging enhancements for debugging, some NPE's, etc > > > > - Bandwidth limits, incremental backup did not work outright. > > > > > > > > Since it is only a couple of months, we might not have seen a lot of > > > corner > > > > cases. > > > > > > > > > > > >> S > > > >> > > > >> > > > >>> > > > For me, I suspect that the backup feature could be done more > > > separately > > > with the main cluster. We could use replication to backup the > WALs, > > > and > > > >>> use > > > Snapshot and ExportSnapshot to backup the HFiles. The feature > could > > be > > > >>> done > > > as a separated project. > > > > > > Thanks. > > > > > > Stack 于2020年11月20日周五 上午10:18写道： > > > > > > > It strikes me as work that has been abandoned with no supporting > > > developer. > > > > It has had no improvement and few commits other than adjustment > > > >>> because a > > > > backing dependency has changed since original contribution. It > has > > > >> not > > > been > > > > included in a release so has no users as yet. Does anyone use it > or > > > >>> want > > > > it? > >
Re: DISCUSS: Remove hbase-backup from master?
On Mon, Nov 30, 2020 at 9:30 PM [email protected] < [email protected]> wrote: > Inline > > On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell > wrote: > > > We are allowed to debate if this should be in the tree or not. Given the > > lack of interest, as evidenced by incomplete state, lack of release, and > > lack of contribution, it is more than fair to discuss removal. > > > > Here is my take: First of all, it is not released. There is no implied > > roadmap or support. Second, there do not seem to be any active > maintainers > > or volunteers as such. Third, unless someone shows up with more patches > for > > it there will be no polish or maturing, there can be no expectations of > > further improvement. > > > > That said, this is open source. New code contribution will change the > > facts as they stand. > > > If there is an agreement upon the direction I am willing to provide patches > to take this feature forward. > > What about the changes you made to make it work? I'd be interested in patches that take this feature out of core. How hard do you think that would be? See the items Duo list above for a start on obstacles to tackle? Thanks, S > > > > > > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack wrote: > > > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > > >>> [email protected]> wrote: > > >>> > > >>> Inline. > > >>> > > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) > > >> wrote: > > >>> > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > package, we need to set up log cleaner at master side, and also, > > the LogRollMasterProcedureManager class needs MasterServices(not > > MasterRpcServices), which means it must be used in the same process > > >> with > > HMaster. > > > > And I'm OK with purging this feature, especially if there is no > > >> developer > > who wants to maintain it. > > > > > > >>> Is the sole reason to purge this feature is no developer volunteering > > to > > >>> maintain this? > > >>> > > >>> > > >> I'd suggest this feature does not belong in core at all. > > >> > > >> And some of us have spent time reviewing the feature in the past and > > back > > >> then determined it needed lots of finishing work, test, and polish -- > > none > > >> of which it seems to have gotten since commit. > > >> > > >> You are using it Mallik? Tell us more please. How is it working for > you? > > >> > > >> > > > It has been a decent v1 release for us. As pointed out by others, > feature > > > should be polished and matured without debating if backup should be > part > > of > > > core or not. Some of notable points are > > > > > > - Limitations like serial backup set runs per deployment bothered us. > > > - Minor logging enhancements for debugging, some NPE's, etc > > > - Bandwidth limits, incremental backup did not work outright. > > > > > > Since it is only a couple of months, we might not have seen a lot of > > corner > > > cases. > > > > > > > > >> S > > >> > > >> > > >>> > > For me, I suspect that the backup feature could be done more > > separately > > with the main cluster. We could use replication to backup the WALs, > > and > > >>> use > > Snapshot and ExportSnapshot to backup the HFiles. The feature could > be > > >>> done > > as a separated project. > > > > Thanks. > > > > Stack 于2020年11月20日周五 上午10:18写道： > > > > > It strikes me as work that has been abandoned with no supporting > > developer. > > > It has had no improvement and few commits other than adjustment > > >>> because a > > > backing dependency has changed since original contribution. It has > > >> not > > been > > > included in a release so has no users as yet. Does anyone use it or > > >>> want > > > it? > > >>> > > >>> > > >>> We have back ported this feature to 2.1.x and has been using for last > > few > > >>> months in few of our deployments. > > >>> > > >>> > > If not, I suggest we remove it. I could file an issue for it to be > > >>> > > added to hbase-operator-tools for some gallant dev to pick up if > they > > > wanted to use this backup work? (I could help w/ the migration). > > > > > > What do others think? > > > > > > S > > > > > > > >>> > > >> > > >
Re: DISCUSS: Remove hbase-backup from master?
Inline On Tue, Dec 1, 2020, 10:10 AM Andrew Purtell wrote: > We are allowed to debate if this should be in the tree or not. Given the > lack of interest, as evidenced by incomplete state, lack of release, and > lack of contribution, it is more than fair to discuss removal. > > Here is my take: First of all, it is not released. There is no implied > roadmap or support. Second, there do not seem to be any active maintainers > or volunteers as such. Third, unless someone shows up with more patches for > it there will be no polish or maturing, there can be no expectations of > further improvement. > > That said, this is open source. New code contribution will change the > facts as they stand. If there is an agreement upon the direction I am willing to provide patches to take this feature forward. > > > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > > > On Mon, Nov 30, 2020 at 9:59 PM Stack wrote: > > > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > >>> [email protected]> wrote: > >>> > >>> Inline. > >>> > >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) > >> wrote: > >>> > I'm afraid it is not easy to be moved to hbase-operator-tools. > > You can see the code under the org.apache.hadoop.hbase.backup.master > package, we need to set up log cleaner at master side, and also, > the LogRollMasterProcedureManager class needs MasterServices(not > MasterRpcServices), which means it must be used in the same process > >> with > HMaster. > > And I'm OK with purging this feature, especially if there is no > >> developer > who wants to maintain it. > > > >>> Is the sole reason to purge this feature is no developer volunteering > to > >>> maintain this? > >>> > >>> > >> I'd suggest this feature does not belong in core at all. > >> > >> And some of us have spent time reviewing the feature in the past and > back > >> then determined it needed lots of finishing work, test, and polish -- > none > >> of which it seems to have gotten since commit. > >> > >> You are using it Mallik? Tell us more please. How is it working for you? > >> > >> > > It has been a decent v1 release for us. As pointed out by others, feature > > should be polished and matured without debating if backup should be part > of > > core or not. Some of notable points are > > > > - Limitations like serial backup set runs per deployment bothered us. > > - Minor logging enhancements for debugging, some NPE's, etc > > - Bandwidth limits, incremental backup did not work outright. > > > > Since it is only a couple of months, we might not have seen a lot of > corner > > cases. > > > > > >> S > >> > >> > >>> > For me, I suspect that the backup feature could be done more > separately > with the main cluster. We could use replication to backup the WALs, > and > >>> use > Snapshot and ExportSnapshot to backup the HFiles. The feature could be > >>> done > as a separated project. > > Thanks. > > Stack 于2020年11月20日周五 上午10:18写道： > > > It strikes me as work that has been abandoned with no supporting > developer. > > It has had no improvement and few commits other than adjustment > >>> because a > > backing dependency has changed since original contribution. It has > >> not > been > > included in a release so has no users as yet. Does anyone use it or > >>> want > > it? > >>> > >>> > >>> We have back ported this feature to 2.1.x and has been using for last > few > >>> months in few of our deployments. > >>> > >>> > If not, I suggest we remove it. I could file an issue for it to be > >>> > added to hbase-operator-tools for some gallant dev to pick up if they > > wanted to use this backup work? (I could help w/ the migration). > > > > What do others think? > > > > S > > > > >>> > >> >
Re: DISCUSS: Remove hbase-backup from master?
Inline On Mon, Nov 30, 2020, 9:56 PM Andrew Purtell wrote: > If we have unmaintained and unreleased, and unreleasable, code sitting in > the tree it should be moved out or deleted. > > > We could use replication to backup the WALs, and use Snapshot and > ExportSnapshot to backup the HFiles. > > This is essentially how we have implemented backup at the $dayjob. > > However, while this will move the data, verifying the integrity of the > backup remains an exercise for the user. (I don’t know the hbase-backup > solution well but imagine if being in tree it does some trick to support > verification?) For verification in our own solution we enable some schema > options and manage compactions with a coprocessor such that within a > sliding window of time (our backup SLA) it is possible to do digest based > comparisons of each site-pair’s data, including deleted data and > tombstones. Digest site A, then digest site B, only the data in the window, > then compare, and if good we have proven equivalence and can move the far > (older) edge of the window leftward, so normal compaction activity can > resume beyond the new right side boundary. This is site and somewhat > application specific, and the age of the design is showing for various > reasons that would make this email long if discussed in depth. I only > wanted to talk about this briefly to illustrate that verification is a > harder problem. Data movement is only the first aspect of a complete backup > solution. > > That said, for years we have been considering a move away from snapshot > based backup to a WAL based backup model, or more recently one based on > publication and consumption of change data capture. HBase replication has > some rough edges if used as a source of change capture data. I have some > thoughts on this: essentially, what could be a design doc for the causal > replication JIRA. If we had an officially supported change capture device > (could be in operator tools) then backup and restore could be implemented > as tooling built on a foundation of retention and replay of change stream > data (also in operator tools). > With change data capture backing backup will enforce certain requirements which might be unnecessary * There is a need for online system to capture the change always unlike point in time based backup solutions * There is a continuous data outflow from hbase cluster which reduces the flexibility with respect to scaling the cluster and backup systems * Not every deployments / tenant with require very low recover point objectives and off peak backups can't be configured Current solution seems elegant to me. Where wals are retained until backup is performed and wal references are cleared with respect to backup. > > On Nov 28, 2020, at 7:48 PM, 张铎 wrote: > > > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > package, we need to set up log cleaner at master side, and also, > > the LogRollMasterProcedureManager class needs MasterServices(not > > MasterRpcServices), which means it must be used in the same process with > > HMaster. > > > > And I'm OK with purging this feature, especially if there is no developer > > who wants to maintain it. > > > > For me, I suspect that the backup feature could be done more separately > > with the main cluster. We could use replication to backup the WALs, and > use > > Snapshot and ExportSnapshot to backup the HFiles. The feature could be > done > > as a separated project. > > > > Thanks. > > > > Stack 于2020年11月20日周五 上午10:18写道： > > > >> It strikes me as work that has been abandoned with no supporting > developer. > >> It has had no improvement and few commits other than adjustment because > a > >> backing dependency has changed since original contribution. It has not > been > >> included in a release so has no users as yet. Does anyone use it or want > >> it? If not, I suggest we remove it. I could file an issue for it to be > >> added to hbase-operator-tools for some gallant dev to pick up if they > >> wanted to use this backup work? (I could help w/ the migration). > >> > >> What do others think? > >> > >> S > >> >
Re: DISCUSS: Remove hbase-backup from master?
We are allowed to debate if this should be in the tree or not. Given the lack of interest, as evidenced by incomplete state, lack of release, and lack of contribution, it is more than fair to discuss removal. Here is my take: First of all, it is not released. There is no implied roadmap or support. Second, there do not seem to be any active maintainers or volunteers as such. Third, unless someone shows up with more patches for it there will be no polish or maturing, there can be no expectations of further improvement. That said, this is open source. New code contribution will change the facts as they stand. > On Nov 30, 2020, at 5:52 PM, [email protected] wrote: > > On Mon, Nov 30, 2020 at 9:59 PM Stack wrote: > >>> On Mon, Nov 30, 2020 at 5:55 AM [email protected] < >>> [email protected]> wrote: >>> >>> Inline. >>> >>> On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) >> wrote: >>> I'm afraid it is not easy to be moved to hbase-operator-tools. You can see the code under the org.apache.hadoop.hbase.backup.master package, we need to set up log cleaner at master side, and also, the LogRollMasterProcedureManager class needs MasterServices(not MasterRpcServices), which means it must be used in the same process >> with HMaster. And I'm OK with purging this feature, especially if there is no >> developer who wants to maintain it. >>> Is the sole reason to purge this feature is no developer volunteering to >>> maintain this? >>> >>> >> I'd suggest this feature does not belong in core at all. >> >> And some of us have spent time reviewing the feature in the past and back >> then determined it needed lots of finishing work, test, and polish -- none >> of which it seems to have gotten since commit. >> >> You are using it Mallik? Tell us more please. How is it working for you? >> >> > It has been a decent v1 release for us. As pointed out by others, feature > should be polished and matured without debating if backup should be part of > core or not. Some of notable points are > > - Limitations like serial backup set runs per deployment bothered us. > - Minor logging enhancements for debugging, some NPE's, etc > - Bandwidth limits, incremental backup did not work outright. > > Since it is only a couple of months, we might not have seen a lot of corner > cases. > > >> S >> >> >>> For me, I suspect that the backup feature could be done more separately with the main cluster. We could use replication to backup the WALs, and >>> use Snapshot and ExportSnapshot to backup the HFiles. The feature could be >>> done as a separated project. Thanks. Stack 于2020年11月20日周五 上午10:18写道： > It strikes me as work that has been abandoned with no supporting developer. > It has had no improvement and few commits other than adjustment >>> because a > backing dependency has changed since original contribution. It has >> not been > included in a release so has no users as yet. Does anyone use it or >>> want > it? >>> >>> >>> We have back ported this feature to 2.1.x and has been using for last few >>> months in few of our deployments. >>> >>> If not, I suggest we remove it. I could file an issue for it to be >>> added to hbase-operator-tools for some gallant dev to pick up if they > wanted to use this backup work? (I could help w/ the migration). > > What do others think? > > S > >>> >>
Re: DISCUSS: Remove hbase-backup from master?
On Mon, Nov 30, 2020 at 9:59 PM Stack wrote: > On Mon, Nov 30, 2020 at 5:55 AM [email protected] < > [email protected]> wrote: > > > Inline. > > > > On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) > wrote: > > > > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > > package, we need to set up log cleaner at master side, and also, > > > the LogRollMasterProcedureManager class needs MasterServices(not > > > MasterRpcServices), which means it must be used in the same process > with > > > HMaster. > > > > > > And I'm OK with purging this feature, especially if there is no > developer > > > who wants to maintain it. > > > > > > > > Is the sole reason to purge this feature is no developer volunteering to > > maintain this? > > > > > I'd suggest this feature does not belong in core at all. > > And some of us have spent time reviewing the feature in the past and back > then determined it needed lots of finishing work, test, and polish -- none > of which it seems to have gotten since commit. > > You are using it Mallik? Tell us more please. How is it working for you? > > It has been a decent v1 release for us. As pointed out by others, feature should be polished and matured without debating if backup should be part of core or not. Some of notable points are - Limitations like serial backup set runs per deployment bothered us. - Minor logging enhancements for debugging, some NPE's, etc - Bandwidth limits, incremental backup did not work outright. Since it is only a couple of months, we might not have seen a lot of corner cases. > S > > > > > > > For me, I suspect that the backup feature could be done more separately > > > with the main cluster. We could use replication to backup the WALs, and > > use > > > Snapshot and ExportSnapshot to backup the HFiles. The feature could be > > done > > > as a separated project. > > > > > > Thanks. > > > > > > Stack 于2020年11月20日周五 上午10:18写道： > > > > > > > It strikes me as work that has been abandoned with no supporting > > > developer. > > > > It has had no improvement and few commits other than adjustment > > because a > > > > backing dependency has changed since original contribution. It has > not > > > been > > > > included in a release so has no users as yet. Does anyone use it or > > want > > > > it? > > > > > > We have back ported this feature to 2.1.x and has been using for last few > > months in few of our deployments. > > > > > > > If not, I suggest we remove it. I could file an issue for it to be > > > > > added to hbase-operator-tools for some gallant dev to pick up if they > > > > wanted to use this backup work? (I could help w/ the migration). > > > > > > > > What do others think? > > > > > > > > S > > > > > > > > > >
Re: DISCUSS: Remove hbase-backup from master?
On Mon, Nov 30, 2020 at 5:55 AM [email protected] < [email protected]> wrote: > Inline. > > On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) wrote: > > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > package, we need to set up log cleaner at master side, and also, > > the LogRollMasterProcedureManager class needs MasterServices(not > > MasterRpcServices), which means it must be used in the same process with > > HMaster. > > > > And I'm OK with purging this feature, especially if there is no developer > > who wants to maintain it. > > > > > Is the sole reason to purge this feature is no developer volunteering to > maintain this? > > I'd suggest this feature does not belong in core at all. And some of us have spent time reviewing the feature in the past and back then determined it needed lots of finishing work, test, and polish -- none of which it seems to have gotten since commit. You are using it Mallik? Tell us more please. How is it working for you? S > > > For me, I suspect that the backup feature could be done more separately > > with the main cluster. We could use replication to backup the WALs, and > use > > Snapshot and ExportSnapshot to backup the HFiles. The feature could be > done > > as a separated project. > > > > Thanks. > > > > Stack 于2020年11月20日周五 上午10:18写道： > > > > > It strikes me as work that has been abandoned with no supporting > > developer. > > > It has had no improvement and few commits other than adjustment > because a > > > backing dependency has changed since original contribution. It has not > > been > > > included in a release so has no users as yet. Does anyone use it or > want > > > it? > > > We have back ported this feature to 2.1.x and has been using for last few > months in few of our deployments. > > > > If not, I suggest we remove it. I could file an issue for it to be > > > added to hbase-operator-tools for some gallant dev to pick up if they > > > wanted to use this backup work? (I could help w/ the migration). > > > > > > What do others think? > > > > > > S > > > > > >
Re: DISCUSS: Remove hbase-backup from master?
If we have unmaintained and unreleased, and unreleasable, code sitting in the tree it should be moved out or deleted. > We could use replication to backup the WALs, and use Snapshot and > ExportSnapshot to backup the HFiles. This is essentially how we have implemented backup at the $dayjob. However, while this will move the data, verifying the integrity of the backup remains an exercise for the user. (I don’t know the hbase-backup solution well but imagine if being in tree it does some trick to support verification?) For verification in our own solution we enable some schema options and manage compactions with a coprocessor such that within a sliding window of time (our backup SLA) it is possible to do digest based comparisons of each site-pair’s data, including deleted data and tombstones. Digest site A, then digest site B, only the data in the window, then compare, and if good we have proven equivalence and can move the far (older) edge of the window leftward, so normal compaction activity can resume beyond the new right side boundary. This is site and somewhat application specific, and the age of the design is showing for various reasons that would make this email long if discussed in depth. I only wanted to talk about this briefly to illustrate that verification is a harder problem. Data movement is only the first aspect of a complete backup solution. That said, for years we have been considering a move away from snapshot based backup to a WAL based backup model, or more recently one based on publication and consumption of change data capture. HBase replication has some rough edges if used as a source of change capture data. I have some thoughts on this: essentially, what could be a design doc for the causal replication JIRA. If we had an officially supported change capture device (could be in operator tools) then backup and restore could be implemented as tooling built on a foundation of retention and replay of change stream data (also in operator tools). > On Nov 28, 2020, at 7:48 PM, 张铎 wrote: > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > You can see the code under the org.apache.hadoop.hbase.backup.master > package, we need to set up log cleaner at master side, and also, > the LogRollMasterProcedureManager class needs MasterServices(not > MasterRpcServices), which means it must be used in the same process with > HMaster. > > And I'm OK with purging this feature, especially if there is no developer > who wants to maintain it. > > For me, I suspect that the backup feature could be done more separately > with the main cluster. We could use replication to backup the WALs, and use > Snapshot and ExportSnapshot to backup the HFiles. The feature could be done > as a separated project. > > Thanks. > > Stack 于2020年11月20日周五 上午10:18写道： > >> It strikes me as work that has been abandoned with no supporting developer. >> It has had no improvement and few commits other than adjustment because a >> backing dependency has changed since original contribution. It has not been >> included in a release so has no users as yet. Does anyone use it or want >> it? If not, I suggest we remove it. I could file an issue for it to be >> added to hbase-operator-tools for some gallant dev to pick up if they >> wanted to use this backup work? (I could help w/ the migration). >> >> What do others think? >> >> S >>
Re: DISCUSS: Remove hbase-backup from master?
[email protected] 于2020年11月30日周一 下午9:55写道： > Inline. > > On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) wrote: > > > I'm afraid it is not easy to be moved to hbase-operator-tools. > > > > You can see the code under the org.apache.hadoop.hbase.backup.master > > package, we need to set up log cleaner at master side, and also, > > the LogRollMasterProcedureManager class needs MasterServices(not > > MasterRpcServices), which means it must be used in the same process with > > HMaster. > > > > And I'm OK with purging this feature, especially if there is no developer > > who wants to maintain it. > > > > > Is the sole reason to purge this feature is no developer volunteering to > maintain this? > Yes, exactly. The recent problem is that, after HBASE-18070 had been merged, we found that the UTs from the backup module are flaky, but no one knows this feature well so it is not easy to fix the UTs. And since this feature has not been officially released yet, so if no one wants to maintain it, we will see if it is possible to just purge this feature from the code base, or move it to other sub repos. > > > > For me, I suspect that the backup feature could be done more separately > > with the main cluster. We could use replication to backup the WALs, and > use > > Snapshot and ExportSnapshot to backup the HFiles. The feature could be > done > > as a separated project. > > > > Thanks. > > > > Stack 于2020年11月20日周五 上午10:18写道： > > > > > It strikes me as work that has been abandoned with no supporting > > developer. > > > It has had no improvement and few commits other than adjustment > because a > > > backing dependency has changed since original contribution. It has not > > been > > > included in a release so has no users as yet. Does anyone use it or > want > > > it? > > > We have back ported this feature to 2.1.x and has been using for last few > months in few of our deployments. > > > > If not, I suggest we remove it. I could file an issue for it to be > > > added to hbase-operator-tools for some gallant dev to pick up if they > > > wanted to use this backup work? (I could help w/ the migration). > > > > > > What do others think? > > > > > > S > > > > > >
Re: DISCUSS: Remove hbase-backup from master?
Inline. On Sun, Nov 29, 2020, 9:18 AM 张铎(Duo Zhang) wrote: > I'm afraid it is not easy to be moved to hbase-operator-tools. > > You can see the code under the org.apache.hadoop.hbase.backup.master > package, we need to set up log cleaner at master side, and also, > the LogRollMasterProcedureManager class needs MasterServices(not > MasterRpcServices), which means it must be used in the same process with > HMaster. > > And I'm OK with purging this feature, especially if there is no developer > who wants to maintain it. > > Is the sole reason to purge this feature is no developer volunteering to maintain this? > For me, I suspect that the backup feature could be done more separately > with the main cluster. We could use replication to backup the WALs, and use > Snapshot and ExportSnapshot to backup the HFiles. The feature could be done > as a separated project. > > Thanks. > > Stack 于2020年11月20日周五 上午10:18写道： > > > It strikes me as work that has been abandoned with no supporting > developer. > > It has had no improvement and few commits other than adjustment because a > > backing dependency has changed since original contribution. It has not > been > > included in a release so has no users as yet. Does anyone use it or want > > it? We have back ported this feature to 2.1.x and has been using for last few months in few of our deployments. > If not, I suggest we remove it. I could file an issue for it to be > added to hbase-operator-tools for some gallant dev to pick up if they > > wanted to use this backup work? (I could help w/ the migration). > > > > What do others think? > > > > S > > >
Re: DISCUSS: Remove hbase-backup from master?
I'm afraid it is not easy to be moved to hbase-operator-tools. You can see the code under the org.apache.hadoop.hbase.backup.master package, we need to set up log cleaner at master side, and also, the LogRollMasterProcedureManager class needs MasterServices(not MasterRpcServices), which means it must be used in the same process with HMaster. And I'm OK with purging this feature, especially if there is no developer who wants to maintain it. For me, I suspect that the backup feature could be done more separately with the main cluster. We could use replication to backup the WALs, and use Snapshot and ExportSnapshot to backup the HFiles. The feature could be done as a separated project. Thanks. Stack 于2020年11月20日周五 上午10:18写道： > It strikes me as work that has been abandoned with no supporting developer. > It has had no improvement and few commits other than adjustment because a > backing dependency has changed since original contribution. It has not been > included in a release so has no users as yet. Does anyone use it or want > it? If not, I suggest we remove it. I could file an issue for it to be > added to hbase-operator-tools for some gallant dev to pick up if they > wanted to use this backup work? (I could help w/ the migration). > > What do others think? > > S >
