[jira] [Created] (HBASE-21739) Move grant/revoke from regionserver to master

[Yi Mei (JIRA)]

Yi Mei created HBASE-21739:
--

 Summary: Move grant/revoke from regionserver to master
 Key: HBASE-21739
 URL: https://issues.apache.org/jira/browse/HBASE-21739
 Project: HBase
  Issue Type: Sub-task
Reporter: Yi Mei
Assignee: Yi Mei


Create a sub-task to move grant/revoke from regionserver to master. Other 
access control operations(getUserPermissions/ checkPermissions/ hasPermission) 
will be moved in another sub-task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

RE: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.

[Pankaj kr]

Yeah, HBCK2/ OfflineMetaRepair tools are really required to migrate old version 
data to HBase-2. We have use cases where we are using these tools to rebuild 
the meta for further region assignment.
Similar discussion is going on HBASE-21665, after fixing the NPE and rebuilding 
the meta, master don't assign the regions as we skip the empty regions while 
loading meta during master startup.

A big +1 from my side on this... 

Regards,
Pankaj

-Original Message-
From: 张铎(Duo Zhang) [mailto:palomino...@gmail.com] 
Sent: 18 January 2019 11:55
To: HBase Dev List 
Subject: Re: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' 
pointer.

So the first priority is to make progress on HBCK2? If we all agree, let's 
start to work.

Andrew Purtell  于2019年1月18日周五 下午12:31写道：

> Sorry, let me add... Check all the boxes on that list and I'm +1 for 
> moving the stable pointer (modulo some time to pound on the candidate 
> to really put it through its paces, like two weeks of chaos...)
>
> On Thu, Jan 17, 2019 at 8:28 PM Andrew Purtell 
> wrote:
>
> > I do not believe we should move the stable pointer to any 2.x until 
> > HBCK2 is feature complete. We can discuss what that milestone should look 
> > like.
> > At a minimum, I think we need:
> >
> >- Rebuild meta from region metadata in the filesystem, aka offline
> >meta rebuild.
> >- Fix assignment errors (undeployed regions, double assignments (yes,
> >should not be possible), etc)
> >- Fix region holes, overlaps, and other errors in the region chain
> >- Fix failed split and merge transactions that have failed to roll
> >back due to some bug (related to previous)
> >- Enumerate store files to determine file level corruption and
> >sideline corrupt files
> >- Fix hfile link problems (dangling / broken)
> >
> > This is a list of the real problems I have had to fix in production 
> > at least once (in the past 10 years...).
> >
> > On Thu, Jan 17, 2019 at 8:19 PM 张铎(Duo Zhang) 
> > 
> > wrote:
> >
> >> There are still lots of small new features which we want to 
> >> integrate
> into
> >> branch-2 so I'm -1 on making release directly from branch-2. 
> >> Backporting at once before release is a pain I'd say, I've tried 
> >> this many times recently, as we have to follow up the community 
> >> version...Let's make a branch-2.2 when we want to release 2.2.0, 
> >> and maybe also retire the branch-2.0?
> >>
> >> For the stable pointer, I think 2.1.x maybe a good candidate? 
> >> Though we know that we may still have some bugs for the AMv2, but 
> >> actually we all know that the AMv1 for all the branch-1.x also has 
> >> lots of bugs, that's why hbck is very important.
> >>
> >> And also +! on making progress on HBCK2, we need to port he useful 
> >> features of HBCK1 to HBCK2. There is no software can guarantee that 
> >> there is no bug, so FWIW we should have a way to fix broken 
> >> clusters.
> >>
> >> Sean Busbey  于2019年1月18日周五 上午11:47写道：
> >>
> >> > There are a few related topics I'd like to discuss and I figured 
> >> > this subject line is the most likely to get a bit of attention. 
> >> > :)
> >> >
> >> > First, I'd like us all to get on the same page wrt the current 
> >> > state of branch-2. Personally, I don't think it can be released 
> >> > as-is with a 2.y version because folks can't rolling upgrade from 
> >> > 2.0 or 2.1 to it due to the current implementation of 
> >> > HBASE-20881. As Duo has mentioned a couple of times, folks have 
> >> > to ensure there are no region transitions around during the 
> >> > upgrade. I think that will be prohibitive for folks looking to upgrade. 
> >> > What do other folks think?
> >> >
> >> > Second, I think our recent discussions around the need for 
> >> > shifting to more minor releases for HBase 1.y also applies to the 2.y 
> >> > branches.
> >> > branch-2 hasn't had a release since 2.1.0 came out in July 2018.
> >> > That's a scary long amount of time. I think it contributes to us 
> >> > ending up with changes like the above since it's easy to think 
> >> > about the branch as something that has a lot of time before the 
> >> > next release.
> >> >
> >> > Personally, I'd like to see us skip making minor-release specific 
> >> > branches for a bit unless a CVE fix or something comes up. 
> >> > Ideally, that would mean we work towards a 2.2.0 release directly 
> >> > from branch-2 and then 2.2.1, etc. When we have a feature that's 
> >> > ready to backport from the master branch for a release we then 
> >> > update branch-2's version to be 2.3.0.
> >> >
> >> > Or maybe we try set a regular cadence to feature releases by 
> >> > having
> >> > branch-2 release a new minor, two months of new maintenance 
> >> > releases, followed by a new minor. That would mean after the last 
> >> > of the maintenance releases we'd have a window of a few weeks 
> >> > where we can all decide which features in master are mature 
> >> > enough to backport for the new minor 

Re: Requesting access to hbase slack channel

[Abhishek Gupta]

Hi,

I would like an invitation too abhila...@gmail.com

Thanks

On Fri, Jan 18, 2019 at 11:38 AM Manjeet Singh 
wrote:

> Done
>
> On Fri, 18 Jan 2019, 11:24 Buchi Reddy Busi Reddy  wrote:
>
> > Can you also invite mailtobu...@gmail.com please?
> >
> > On Thu, Jan 17, 2019 at 8:34 PM Manjeet Singh <
> manjeet.chand...@gmail.com>
> > wrote:
> >
> > > Seems someone else already did it
> > >
> > > Manjeet
> > >
> > > On Wed, 16 Jan 2019, 17:33 Nihal Jain  > >
> > > > Hi
> > > >
> > > > Could you please invite me: nihaljain...@gmail.com <
> > > nihaljain...@gmail.com
> > > > >?
> > > >
> > > > Regards,
> > > > Nihal
> > > >
> > > > On Wed, 16 Jan, 2019, 2:54 PM Peter Somogyi  > wrote:
> > > >
> > > > > Sent invitation to madhurpan...@gmail.com.
> > > > >
> > > > > On Wed, Jan 16, 2019 at 7:27 AM Madhur Pant <
> madhurpan...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Team,
> > > > > >
> > > > > > I was wondering if I could get access to the HBase user slack
> > channel
> > > > > >
> > > > > > https://apache-hbase.slack.com
> > > > > >
> > > > > > It says here 
> > > > that I
> > > > > > should email you guys  :)
> > > > > >
> > > > > > Thanks,
> > > > > > Madhur Pant
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.

[Duo Zhang]

So the first priority is to make progress on HBCK2? If we all agree, let's
start to work.

Andrew Purtell  于2019年1月18日周五 下午12:31写道：

> Sorry, let me add... Check all the boxes on that list and I'm +1 for moving
> the stable pointer (modulo some time to pound on the candidate to really
> put it through its paces, like two weeks of chaos...)
>
> On Thu, Jan 17, 2019 at 8:28 PM Andrew Purtell 
> wrote:
>
> > I do not believe we should move the stable pointer to any 2.x until HBCK2
> > is feature complete. We can discuss what that milestone should look like.
> > At a minimum, I think we need:
> >
> >- Rebuild meta from region metadata in the filesystem, aka offline
> >meta rebuild.
> >- Fix assignment errors (undeployed regions, double assignments (yes,
> >should not be possible), etc)
> >- Fix region holes, overlaps, and other errors in the region chain
> >- Fix failed split and merge transactions that have failed to roll
> >back due to some bug (related to previous)
> >- Enumerate store files to determine file level corruption and
> >sideline corrupt files
> >- Fix hfile link problems (dangling / broken)
> >
> > This is a list of the real problems I have had to fix in production at
> > least once (in the past 10 years...).
> >
> > On Thu, Jan 17, 2019 at 8:19 PM 张铎(Duo Zhang) 
> > wrote:
> >
> >> There are still lots of small new features which we want to integrate
> into
> >> branch-2 so I'm -1 on making release directly from branch-2. Backporting
> >> at
> >> once before release is a pain I'd say, I've tried this many times
> >> recently,
> >> as we have to follow up the community version...Let's make a branch-2.2
> >> when we want to release 2.2.0, and maybe also retire the branch-2.0?
> >>
> >> For the stable pointer, I think 2.1.x maybe a good candidate? Though we
> >> know that we may still have some bugs for the AMv2, but actually we all
> >> know that the AMv1 for all the branch-1.x also has lots of bugs, that's
> >> why
> >> hbck is very important.
> >>
> >> And also +! on making progress on HBCK2, we need to port he useful
> >> features
> >> of HBCK1 to HBCK2. There is no software can guarantee that there is no
> >> bug,
> >> so FWIW we should have a way to fix broken clusters.
> >>
> >> Sean Busbey  于2019年1月18日周五 上午11:47写道：
> >>
> >> > There are a few related topics I'd like to discuss and I figured this
> >> > subject line is the most likely to get a bit of attention. :)
> >> >
> >> > First, I'd like us all to get on the same page wrt the current state
> >> > of branch-2. Personally, I don't think it can be released as-is with a
> >> > 2.y version because folks can't rolling upgrade from 2.0 or 2.1 to it
> >> > due to the current implementation of HBASE-20881. As Duo has mentioned
> >> > a couple of times, folks have to ensure there are no region
> >> > transitions around during the upgrade. I think that will be
> >> > prohibitive for folks looking to upgrade. What do other folks think?
> >> >
> >> > Second, I think our recent discussions around the need for shifting to
> >> > more minor releases for HBase 1.y also applies to the 2.y branches.
> >> > branch-2 hasn't had a release since 2.1.0 came out in July 2018.
> >> > That's a scary long amount of time. I think it contributes to us
> >> > ending up with changes like the above since it's easy to think about
> >> > the branch as something that has a lot of time before the next
> >> > release.
> >> >
> >> > Personally, I'd like to see us skip making minor-release specific
> >> > branches for a bit unless a CVE fix or something comes up. Ideally,
> >> > that would mean we work towards a 2.2.0 release directly from branch-2
> >> > and then 2.2.1, etc. When we have a feature that's ready to backport
> >> > from the master branch for a release we then update branch-2's version
> >> > to be 2.3.0.
> >> >
> >> > Or maybe we try set a regular cadence to feature releases by having
> >> > branch-2 release a new minor, two months of new maintenance releases,
> >> > followed by a new minor. That would mean after the last of the
> >> > maintenance releases we'd have a window of a few weeks where we can
> >> > all decide which features in master are mature enough to backport for
> >> > the new minor release.
> >> >
> >> > Lastly, what would it take for folks to feel confident moving the
> >> > 'stable' pointer to a HBase 2.y? Is there a major gap still on
> >> > assignment stability? Is it a more thorough look at performance? More
> >> > time to ensure HBCK2 has good coverage of failure modes that need it?
> >> >
> >>
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: Requesting access to hbase slack channel

[Manjeet Singh]

Done

On Fri, 18 Jan 2019, 11:24 Buchi Reddy Busi Reddy  Can you also invite mailtobu...@gmail.com please?
>
> On Thu, Jan 17, 2019 at 8:34 PM Manjeet Singh 
> wrote:
>
> > Seems someone else already did it
> >
> > Manjeet
> >
> > On Wed, 16 Jan 2019, 17:33 Nihal Jain  >
> > > Hi
> > >
> > > Could you please invite me: nihaljain...@gmail.com <
> > nihaljain...@gmail.com
> > > >?
> > >
> > > Regards,
> > > Nihal
> > >
> > > On Wed, 16 Jan, 2019, 2:54 PM Peter Somogyi  wrote:
> > >
> > > > Sent invitation to madhurpan...@gmail.com.
> > > >
> > > > On Wed, Jan 16, 2019 at 7:27 AM Madhur Pant 
> > > > wrote:
> > > >
> > > > > Hi Team,
> > > > >
> > > > > I was wondering if I could get access to the HBase user slack
> channel
> > > > >
> > > > > https://apache-hbase.slack.com
> > > > >
> > > > > It says here 
> > > that I
> > > > > should email you guys  :)
> > > > >
> > > > > Thanks,
> > > > > Madhur Pant
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (HBASE-21738) Latency spike happen when memstore flushing in 100% put case

[Zheng Hu (JIRA)]

Zheng Hu created HBASE-21738:


 Summary: Latency spike happen when memstore flushing in 100% put 
case
 Key: HBASE-21738
 URL: https://issues.apache.org/jira/browse/HBASE-21738
 Project: HBase
  Issue Type: Sub-task
  Components: Performance
Reporter: Zheng Hu
Assignee: Zheng Hu
 Attachments: image-2019-01-18-14-03-28-662.png

Made some performance test for 100% put case in branch-2 before. 

We can see that there are many  latency peak  in p999 latency curve , and the 
peak time are almost the point time which our region is flushing. 

 !image-2019-01-18-14-03-28-662.png! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Requesting access to hbase slack channel

[Buchi Reddy Busi Reddy]

Can you also invite mailtobu...@gmail.com please?

On Thu, Jan 17, 2019 at 8:34 PM Manjeet Singh 
wrote:

> Seems someone else already did it
>
> Manjeet
>
> On Wed, 16 Jan 2019, 17:33 Nihal Jain 
> > Hi
> >
> > Could you please invite me: nihaljain...@gmail.com <
> nihaljain...@gmail.com
> > >?
> >
> > Regards,
> > Nihal
> >
> > On Wed, 16 Jan, 2019, 2:54 PM Peter Somogyi  >
> > > Sent invitation to madhurpan...@gmail.com.
> > >
> > > On Wed, Jan 16, 2019 at 7:27 AM Madhur Pant 
> > > wrote:
> > >
> > > > Hi Team,
> > > >
> > > > I was wondering if I could get access to the HBase user slack channel
> > > >
> > > > https://apache-hbase.slack.com
> > > >
> > > > It says here 
> > that I
> > > > should email you guys  :)
> > > >
> > > > Thanks,
> > > > Madhur Pant
> > > >
> > >
> >
>

Requesting access to slack channel

[aman goyal]

Hi Team,

Pls provide access to hbase user slack channel
https://apache-hbase.slack.com

Pls send invitation to aman...@gmail.com

Thanks,
Aman

Re: Requesting access to hbase slack channel

[Manjeet Singh]

Seems someone else already did it

Manjeet

On Wed, 16 Jan 2019, 17:33 Nihal Jain  Hi
>
> Could you please invite me: nihaljain...@gmail.com  >?
>
> Regards,
> Nihal
>
> On Wed, 16 Jan, 2019, 2:54 PM Peter Somogyi 
> > Sent invitation to madhurpan...@gmail.com.
> >
> > On Wed, Jan 16, 2019 at 7:27 AM Madhur Pant 
> > wrote:
> >
> > > Hi Team,
> > >
> > > I was wondering if I could get access to the HBase user slack channel
> > >
> > > https://apache-hbase.slack.com
> > >
> > > It says here 
> that I
> > > should email you guys  :)
> > >
> > > Thanks,
> > > Madhur Pant
> > >
> >
>

Re: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.

[Andrew Purtell]

Sorry, let me add... Check all the boxes on that list and I'm +1 for moving
the stable pointer (modulo some time to pound on the candidate to really
put it through its paces, like two weeks of chaos...)

On Thu, Jan 17, 2019 at 8:28 PM Andrew Purtell  wrote:

> I do not believe we should move the stable pointer to any 2.x until HBCK2
> is feature complete. We can discuss what that milestone should look like.
> At a minimum, I think we need:
>
>- Rebuild meta from region metadata in the filesystem, aka offline
>meta rebuild.
>- Fix assignment errors (undeployed regions, double assignments (yes,
>should not be possible), etc)
>- Fix region holes, overlaps, and other errors in the region chain
>- Fix failed split and merge transactions that have failed to roll
>back due to some bug (related to previous)
>- Enumerate store files to determine file level corruption and
>sideline corrupt files
>- Fix hfile link problems (dangling / broken)
>
> This is a list of the real problems I have had to fix in production at
> least once (in the past 10 years...).
>
> On Thu, Jan 17, 2019 at 8:19 PM 张铎(Duo Zhang) 
> wrote:
>
>> There are still lots of small new features which we want to integrate into
>> branch-2 so I'm -1 on making release directly from branch-2. Backporting
>> at
>> once before release is a pain I'd say, I've tried this many times
>> recently,
>> as we have to follow up the community version...Let's make a branch-2.2
>> when we want to release 2.2.0, and maybe also retire the branch-2.0?
>>
>> For the stable pointer, I think 2.1.x maybe a good candidate? Though we
>> know that we may still have some bugs for the AMv2, but actually we all
>> know that the AMv1 for all the branch-1.x also has lots of bugs, that's
>> why
>> hbck is very important.
>>
>> And also +! on making progress on HBCK2, we need to port he useful
>> features
>> of HBCK1 to HBCK2. There is no software can guarantee that there is no
>> bug,
>> so FWIW we should have a way to fix broken clusters.
>>
>> Sean Busbey  于2019年1月18日周五 上午11:47写道：
>>
>> > There are a few related topics I'd like to discuss and I figured this
>> > subject line is the most likely to get a bit of attention. :)
>> >
>> > First, I'd like us all to get on the same page wrt the current state
>> > of branch-2. Personally, I don't think it can be released as-is with a
>> > 2.y version because folks can't rolling upgrade from 2.0 or 2.1 to it
>> > due to the current implementation of HBASE-20881. As Duo has mentioned
>> > a couple of times, folks have to ensure there are no region
>> > transitions around during the upgrade. I think that will be
>> > prohibitive for folks looking to upgrade. What do other folks think?
>> >
>> > Second, I think our recent discussions around the need for shifting to
>> > more minor releases for HBase 1.y also applies to the 2.y branches.
>> > branch-2 hasn't had a release since 2.1.0 came out in July 2018.
>> > That's a scary long amount of time. I think it contributes to us
>> > ending up with changes like the above since it's easy to think about
>> > the branch as something that has a lot of time before the next
>> > release.
>> >
>> > Personally, I'd like to see us skip making minor-release specific
>> > branches for a bit unless a CVE fix or something comes up. Ideally,
>> > that would mean we work towards a 2.2.0 release directly from branch-2
>> > and then 2.2.1, etc. When we have a feature that's ready to backport
>> > from the master branch for a release we then update branch-2's version
>> > to be 2.3.0.
>> >
>> > Or maybe we try set a regular cadence to feature releases by having
>> > branch-2 release a new minor, two months of new maintenance releases,
>> > followed by a new minor. That would mean after the last of the
>> > maintenance releases we'd have a window of a few weeks where we can
>> > all decide which features in master are mature enough to backport for
>> > the new minor release.
>> >
>> > Lastly, what would it take for folks to feel confident moving the
>> > 'stable' pointer to a HBase 2.y? Is there a major gap still on
>> > assignment stability? Is it a more thorough look at performance? More
>> > time to ensure HBCK2 has good coverage of failure modes that need it?
>> >
>>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.

[Andrew Purtell]

I do not believe we should move the stable pointer to any 2.x until HBCK2
is feature complete. We can discuss what that milestone should look like.
At a minimum, I think we need:

   - Rebuild meta from region metadata in the filesystem, aka offline meta
   rebuild.
   - Fix assignment errors (undeployed regions, double assignments (yes,
   should not be possible), etc)
   - Fix region holes, overlaps, and other errors in the region chain
   - Fix failed split and merge transactions that have failed to roll back
   due to some bug (related to previous)
   - Enumerate store files to determine file level corruption and sideline
   corrupt files
   - Fix hfile link problems (dangling / broken)

This is a list of the real problems I have had to fix in production at
least once (in the past 10 years...).

On Thu, Jan 17, 2019 at 8:19 PM 张铎(Duo Zhang)  wrote:

> There are still lots of small new features which we want to integrate into
> branch-2 so I'm -1 on making release directly from branch-2. Backporting at
> once before release is a pain I'd say, I've tried this many times recently,
> as we have to follow up the community version...Let's make a branch-2.2
> when we want to release 2.2.0, and maybe also retire the branch-2.0?
>
> For the stable pointer, I think 2.1.x maybe a good candidate? Though we
> know that we may still have some bugs for the AMv2, but actually we all
> know that the AMv1 for all the branch-1.x also has lots of bugs, that's why
> hbck is very important.
>
> And also +! on making progress on HBCK2, we need to port he useful features
> of HBCK1 to HBCK2. There is no software can guarantee that there is no bug,
> so FWIW we should have a way to fix broken clusters.
>
> Sean Busbey  于2019年1月18日周五 上午11:47写道：
>
> > There are a few related topics I'd like to discuss and I figured this
> > subject line is the most likely to get a bit of attention. :)
> >
> > First, I'd like us all to get on the same page wrt the current state
> > of branch-2. Personally, I don't think it can be released as-is with a
> > 2.y version because folks can't rolling upgrade from 2.0 or 2.1 to it
> > due to the current implementation of HBASE-20881. As Duo has mentioned
> > a couple of times, folks have to ensure there are no region
> > transitions around during the upgrade. I think that will be
> > prohibitive for folks looking to upgrade. What do other folks think?
> >
> > Second, I think our recent discussions around the need for shifting to
> > more minor releases for HBase 1.y also applies to the 2.y branches.
> > branch-2 hasn't had a release since 2.1.0 came out in July 2018.
> > That's a scary long amount of time. I think it contributes to us
> > ending up with changes like the above since it's easy to think about
> > the branch as something that has a lot of time before the next
> > release.
> >
> > Personally, I'd like to see us skip making minor-release specific
> > branches for a bit unless a CVE fix or something comes up. Ideally,
> > that would mean we work towards a 2.2.0 release directly from branch-2
> > and then 2.2.1, etc. When we have a feature that's ready to backport
> > from the master branch for a release we then update branch-2's version
> > to be 2.3.0.
> >
> > Or maybe we try set a regular cadence to feature releases by having
> > branch-2 release a new minor, two months of new maintenance releases,
> > followed by a new minor. That would mean after the last of the
> > maintenance releases we'd have a window of a few weeks where we can
> > all decide which features in master are mature enough to backport for
> > the new minor release.
> >
> > Lastly, what would it take for folks to feel confident moving the
> > 'stable' pointer to a HBase 2.y? Is there a major gap still on
> > assignment stability? Is it a more thorough look at performance? More
> > time to ensure HBCK2 has good coverage of failure modes that need it?
> >
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

[jira] [Reopened] (HBASE-21573) More sensible client default timeout values

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-21573:
---

Reopening

> More sensible client default timeout values
> ---
>
> Key: HBASE-21573
> URL: https://issues.apache.org/jira/browse/HBASE-21573
> Project: HBase
>  Issue Type: Wish
>  Components: Client
>Affects Versions: 2.1.1
>Reporter: Cosmin Lehene
>Assignee: Peter Somogyi
>Priority: Critical
> Attachments: HBASE-21573.master.001.patch
>
>
> I guess the goal is to have operations allow enough time to recover from 
> major failures.
> While this may make sense for large jobs, it's a PITA for OLTP scenarios and 
> could probably benefit from a faster failure mode in default
>  
> hbase.rpc.timeout = 6
> hbase.client.operation.timeout = 120
> hbase.client.meta.operation.timeout = 120
> The client meta ops timeout is not in defaults-xml and not documented in the 
> book either.
> [https://hbase.apache.org/book.html#config_timeouts]
>  
> Would it make sense to have aggressive defaults instead?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

RE: Requesting access to hbase slack channel

[Pankaj kr]

Hi,

Can you please send invitation me at:  pankaj...@huawei.com 

Regards,
Pankaj

-Original Message-
From: Toshihiro Suzuki [mailto:brfrn...@gmail.com] 
Sent: 18 January 2019 08:23
To: dev@hbase.apache.org
Subject: Re: Requesting access to hbase slack channel

Hi,

Could you please also invite me: brfrn...@gmail.com

Regards,
Toshihiro Suzuki

On Wed, Jan 16, 2019 at 9:35 PM Peter Somogyi  wrote:

> Invite sent.
>
> On Wed, Jan 16, 2019 at 1:03 PM Nihal Jain  wrote:
>
> > Hi
> >
> > Could you please invite me: nihaljain...@gmail.com <
> nihaljain...@gmail.com
> > >?
> >
> > Regards,
> > Nihal
> >
> > On Wed, 16 Jan, 2019, 2:54 PM Peter Somogyi  >
> > > Sent invitation to madhurpan...@gmail.com.
> > >
> > > On Wed, Jan 16, 2019 at 7:27 AM Madhur Pant 
> > > wrote:
> > >
> > > > Hi Team,
> > > >
> > > > I was wondering if I could get access to the HBase user slack channel
> > > >
> > > > https://apache-hbase.slack.com
> > > >
> > > > It says here 
> > that I
> > > > should email you guys  :)
> > > >
> > > > Thanks,
> > > > Madhur Pant
> > > >
> > >
> >
>

Re: [DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.

[Duo Zhang]

There are still lots of small new features which we want to integrate into
branch-2 so I'm -1 on making release directly from branch-2. Backporting at
once before release is a pain I'd say, I've tried this many times recently,
as we have to follow up the community version...Let's make a branch-2.2
when we want to release 2.2.0, and maybe also retire the branch-2.0?

For the stable pointer, I think 2.1.x maybe a good candidate? Though we
know that we may still have some bugs for the AMv2, but actually we all
know that the AMv1 for all the branch-1.x also has lots of bugs, that's why
hbck is very important.

And also +! on making progress on HBCK2, we need to port he useful features
of HBCK1 to HBCK2. There is no software can guarantee that there is no bug,
so FWIW we should have a way to fix broken clusters.

Sean Busbey  于2019年1月18日周五 上午11:47写道：

> There are a few related topics I'd like to discuss and I figured this
> subject line is the most likely to get a bit of attention. :)
>
> First, I'd like us all to get on the same page wrt the current state
> of branch-2. Personally, I don't think it can be released as-is with a
> 2.y version because folks can't rolling upgrade from 2.0 or 2.1 to it
> due to the current implementation of HBASE-20881. As Duo has mentioned
> a couple of times, folks have to ensure there are no region
> transitions around during the upgrade. I think that will be
> prohibitive for folks looking to upgrade. What do other folks think?
>
> Second, I think our recent discussions around the need for shifting to
> more minor releases for HBase 1.y also applies to the 2.y branches.
> branch-2 hasn't had a release since 2.1.0 came out in July 2018.
> That's a scary long amount of time. I think it contributes to us
> ending up with changes like the above since it's easy to think about
> the branch as something that has a lot of time before the next
> release.
>
> Personally, I'd like to see us skip making minor-release specific
> branches for a bit unless a CVE fix or something comes up. Ideally,
> that would mean we work towards a 2.2.0 release directly from branch-2
> and then 2.2.1, etc. When we have a feature that's ready to backport
> from the master branch for a release we then update branch-2's version
> to be 2.3.0.
>
> Or maybe we try set a regular cadence to feature releases by having
> branch-2 release a new minor, two months of new maintenance releases,
> followed by a new minor. That would mean after the last of the
> maintenance releases we'd have a window of a few weeks where we can
> all decide which features in master are mature enough to backport for
> the new minor release.
>
> Lastly, what would it take for folks to feel confident moving the
> 'stable' pointer to a HBase 2.y? Is there a major gap still on
> assignment stability? Is it a more thorough look at performance? More
> time to ensure HBCK2 has good coverage of failure modes that need it?
>

[DISCUSS] Moving towards a branch-2 line that can get the 'stable' pointer.

[Sean Busbey]

There are a few related topics I'd like to discuss and I figured this
subject line is the most likely to get a bit of attention. :)

First, I'd like us all to get on the same page wrt the current state
of branch-2. Personally, I don't think it can be released as-is with a
2.y version because folks can't rolling upgrade from 2.0 or 2.1 to it
due to the current implementation of HBASE-20881. As Duo has mentioned
a couple of times, folks have to ensure there are no region
transitions around during the upgrade. I think that will be
prohibitive for folks looking to upgrade. What do other folks think?

Second, I think our recent discussions around the need for shifting to
more minor releases for HBase 1.y also applies to the 2.y branches.
branch-2 hasn't had a release since 2.1.0 came out in July 2018.
That's a scary long amount of time. I think it contributes to us
ending up with changes like the above since it's easy to think about
the branch as something that has a lot of time before the next
release.

Personally, I'd like to see us skip making minor-release specific
branches for a bit unless a CVE fix or something comes up. Ideally,
that would mean we work towards a 2.2.0 release directly from branch-2
and then 2.2.1, etc. When we have a feature that's ready to backport
from the master branch for a release we then update branch-2's version
to be 2.3.0.

Or maybe we try set a regular cadence to feature releases by having
branch-2 release a new minor, two months of new maintenance releases,
followed by a new minor. That would mean after the last of the
maintenance releases we'd have a window of a few weeks where we can
all decide which features in master are mature enough to backport for
the new minor release.

Lastly, what would it take for folks to feel confident moving the
'stable' pointer to a HBase 2.y? Is there a major gap still on
assignment stability? Is it a more thorough look at performance? More
time to ensure HBCK2 has good coverage of failure modes that need it?

Re: Requesting access to hbase slack channel

[Toshihiro Suzuki]

Hi,

Could you please also invite me: brfrn...@gmail.com

Regards,
Toshihiro Suzuki

On Wed, Jan 16, 2019 at 9:35 PM Peter Somogyi  wrote:

> Invite sent.
>
> On Wed, Jan 16, 2019 at 1:03 PM Nihal Jain  wrote:
>
> > Hi
> >
> > Could you please invite me: nihaljain...@gmail.com <
> nihaljain...@gmail.com
> > >?
> >
> > Regards,
> > Nihal
> >
> > On Wed, 16 Jan, 2019, 2:54 PM Peter Somogyi  >
> > > Sent invitation to madhurpan...@gmail.com.
> > >
> > > On Wed, Jan 16, 2019 at 7:27 AM Madhur Pant 
> > > wrote:
> > >
> > > > Hi Team,
> > > >
> > > > I was wondering if I could get access to the HBase user slack channel
> > > >
> > > > https://apache-hbase.slack.com
> > > >
> > > > It says here 
> > that I
> > > > should email you guys  :)
> > > >
> > > > Thanks,
> > > > Madhur Pant
> > > >
> > >
> >
>

[jira] [Created] (HBASE-21737) Fix "Appendix A: HFile format" section in the doc

[Sakthi (JIRA)]

Sakthi created HBASE-21737:
--

 Summary: Fix "Appendix A: HFile format" section in the doc
 Key: HBASE-21737
 URL: https://issues.apache.org/jira/browse/HBASE-21737
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Sakthi
Assignee: Sakthi


There are few typos in the "Appendix A: HFile format" section in the doc. 
Fixing them in this jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21736) TestHbck is flakey

[Duo Zhang (JIRA)]

Duo Zhang created HBASE-21736:
-

 Summary: TestHbck is flakey
 Key: HBASE-21736
 URL: https://issues.apache.org/jira/browse/HBASE-21736
 Project: HBase
  Issue Type: Bug
Reporter: Duo Zhang


It will hang on stopping region server

https://builds.apache.org/job/HBase-Flaky-Tests/job/master/2348/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestHbck-output.txt/*view*/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21735) Port HBASE-18784 (Use of filesystem that requires hflush / hsync / append / etc should query outputstream capabilities) to branch-1

[Andrew Purtell (JIRA)]

Andrew Purtell created HBASE-21735:
--

 Summary: Port HBASE-18784 (Use of filesystem that requires hflush 
/ hsync / append / etc should query outputstream capabilities) to branch-1
 Key: HBASE-21735
 URL: https://issues.apache.org/jira/browse/HBASE-21735
 Project: HBase
  Issue Type: Sub-task
  Components: fs, wal
Reporter: Andrew Purtell
Assignee: Andrew Purtell
 Fix For: 1.5.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[ANNOUNCE] Apache HBase 1.2.10 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.2.10

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.2.10 is the tenth maintenance release in the HBase 1.2 line,
continuing on the theme of bringing a stable, reliable database to
the Hadoop and NoSQL communities. This release includes about a half
dozen bug fixes done in the two months since 1.2.10.

All users of previous 1.2.z releases are encouraged to upgrade to either
this release or the latest in our stable release line, which is currently
1.4.9. Releases in the 1.2.z line are expected to stop in late spring 2019.

Critical fixes include:

* HBASE-21387 - Race condition surrounding in progress snapshot handling in
snapshot cache leads to loss of snapshot files
* HBASE-21492 - CellCodec Written To WAL Before It's Verified
* HBASE-21504 - If enable FIFOCompactionPolicy, a compaction may write a
"empty" hfile whose maxTimeStamp is long max. This kind of
hfile will never be archived.
* HBASE-21582 - If call HBaseAdmin#snapshotAsync but forget call
isSnapshotFinished, then SnapshotHFileCleaner will skip to
run every time

The full list of fixes included in this release is available at:

 https://s.apache.org/hbase-1.2.10-jira-release-notes

 and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/hbase-1.2.10

The relevant checksums files are available at:


https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-src.tar.gz.sha512

https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-bin.tar.gz.sha512

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-src.tar.gz.asc
https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at:
dev@hbase.apache.org.

Cheers,
The HBase Dev Team

[jira] [Resolved] (HBASE-21595) Print thread's information and stack traces when RS is aborting forcibly

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell resolved HBASE-21595.

Resolution: Fixed

Pushed to branch-1. Looks like this has landed everywhere desired, so resolving 
the issue

> Print thread's information and stack traces when RS is aborting forcibly
> 
>
> Key: HBASE-21595
> URL: https://issues.apache.org/jira/browse/HBASE-21595
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 3.0.0, 1.5.0, 2.2.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Minor
>  Labels: operability
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21595.branch-1.patch, HBASE-21595.patch
>
>
> After HBASE-21325 RS terminate forcibly  on abort timeout.
> We should print the thread info before terminating, will be useful to analyze 
> the RS abort timeout problem.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HBASE-21595) Print thread's information and stack traces when RS is aborting forcibly

[Pankaj Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pankaj Kumar reopened HBASE-21595:
--

Changes are applicable to branch-1 also. Ping [~apurtell]

> Print thread's information and stack traces when RS is aborting forcibly
> 
>
> Key: HBASE-21595
> URL: https://issues.apache.org/jira/browse/HBASE-21595
> Project: HBase
>  Issue Type: Improvement
>  Components: regionserver
>Affects Versions: 3.0.0, 1.5.0, 2.2.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Minor
>  Labels: operability
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21595.branch-1.patch, HBASE-21595.patch
>
>
> After HBASE-21325 RS terminate forcibly  on abort timeout.
> We should print the thread info before terminating, will be useful to analyze 
> the RS abort timeout problem.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21734) Some optimization in FilterListWithOR

[Zheng Hu (JIRA)]

Zheng Hu created HBASE-21734:


 Summary: Some optimization in FilterListWithOR
 Key: HBASE-21734
 URL: https://issues.apache.org/jira/browse/HBASE-21734
 Project: HBase
  Issue Type: Sub-task
Reporter: Zheng Hu


In HBASE-21620,   [~KarthickRam] and [~mohamed.meeran]  complaind that their 
performance of filter list has been degraded after that patch in here [1].  

I wrote a UT for this, and test under my host.  It's true.   I gussed there may 
be two reasons: 
1.  the comparator.compare(nextKV, cell) > 0 StoreScanner; 
2.  the filter list concated by OR will choose the minimal forward step among 
all sub-filters. in this patch, we have stricter restrictions on all sub 
filters, include those sub-filter whose has non-null RC returned in 
calculateReturnCodeByPrevCellAndRC (previously, we will skip to merge this 
sub-filter's rc, but it's wrong in some case), and merge all of the 
sub-filter's RC, this is also some time cost.

The former one seems not the main problem, because the UT still cost ~ 3s even 
if I comment the compare.  the second one has some impact indeed, because after 
i skip to merge the sub-filters's RC if calculateReturnCodeByPrevCellAndRC 
return a non-null rc,  the UT cost ~ 1s,  it's improvement but the logic is not 
wrong.


1. 
https://issues.apache.org/jira/browse/HBASE-21620?focusedCommentId=16737100=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16737100



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HBASE-21733) SnapshotQuotaObserverChore should only fetch space quotas

[Yi Mei (JIRA)]

Yi Mei created HBASE-21733:
--

 Summary: SnapshotQuotaObserverChore should only fetch space quotas
 Key: HBASE-21733
 URL: https://issues.apache.org/jira/browse/HBASE-21733
 Project: HBase
  Issue Type: Bug
Reporter: Yi Mei


In SnapshotQuotaObserverChore.getSnapshotsFromTables method, it fetches space 
quotas using the following filter:
{code:java}
QuotaFilter filter = new QuotaFilter();
filter.addTypeFilter(QuotaType.SPACE);
{code}
but the QuotaType filter hasn't been implemented.

And if there is throttle quotas in quota table, it will encounter Exception as 
follows:
{code:java}
java.lang.IllegalStateException: Expected only one of namespace and tablename 
to be null

at 
org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:137)
at 
org.apache.hadoop.hbase.quotas.TestSnapshotQuotaObserverChore.testSnapshotsFromNamespaces(TestSnapshotQuotaObserverChore.java:184)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)