[jira] [Created] (HBASE-21484) [HBCK2] hbck2 should default to a released hbase version
Sean Busbey created HBASE-21484: --- Summary: [HBCK2] hbck2 should default to a released hbase version Key: HBASE-21484 URL: https://issues.apache.org/jira/browse/HBASE-21484 Project: HBase Issue Type: Bug Components: hbck2 Affects Versions: hbck2-1.0.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: hbck2-1.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21483) [HBCK2] version string checking should look for exactly the version we know doesn't work
Sean Busbey created HBASE-21483: --- Summary: [HBCK2] version string checking should look for exactly the version we know doesn't work Key: HBASE-21483 URL: https://issues.apache.org/jira/browse/HBASE-21483 Project: HBase Issue Type: Bug Components: hbck2 Affects Versions: hbck2-1.0.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: hbck2-1.0.0 Right now the version check looks for anything that starts with "2.1.0" and declares HBCK2 a lost cause. I presume this was to deal with "2.1.0-SNAPSHOT", but that no longer needs to be a concern. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-14294) Add new attribute to Scan to limit the return size of records
[ https://issues.apache.org/jira/browse/HBASE-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick.han resolved HBASE-14294. -- Resolution: Duplicate > Add new attribute to Scan to limit the return size of records > - > > Key: HBASE-14294 > URL: https://issues.apache.org/jira/browse/HBASE-14294 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick.han >Assignee: Nick.han >Priority: Minor > Attachments: 14294-v1.patch > > > Reason: > we all use PageFilter to limit the size of the return records,but the > PageFilter return limit * region number size of limit ,which is bad for a > huge number of region per table. > Solution: > ClientScanner has a attribute caching which determine the return size of > record per RPC,so we can use this attribute to limit the return size of hbase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21485) Add more debug logs for remote procedure execution
Duo Zhang created HBASE-21485: - Summary: Add more debug logs for remote procedure execution Key: HBASE-21485 URL: https://issues.apache.org/jira/browse/HBASE-21485 Project: HBase Issue Type: Improvement Components: proc-v2 Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0, 2.2.0, 2.1.2 In our internal testing, sometimes the RefreshPeerProcedure could hang for a very time. Add more debug logs for better debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21486) The current replication implementation for peer in STANDBY state breaks serial replication
Duo Zhang created HBASE-21486: - Summary: The current replication implementation for peer in STANDBY state breaks serial replication Key: HBASE-21486 URL: https://issues.apache.org/jira/browse/HBASE-21486 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang Fix For: 3.0.0 As we will drop recovered queues in STANDBY state, and also when transiting to DA, we will drain the replication queue, we may miss to update the last pushed sequence id, and cause the serial replication to be stuck. Need to find a way to deal with this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21482) TestHRegion fails due to 'Too many open files'
Ted Yu created HBASE-21482:
--
Summary: TestHRegion fails due to 'Too many open files'
Key: HBASE-21482
URL: https://issues.apache.org/jira/browse/HBASE-21482
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
TestHRegion fails due to 'Too many open files' in master branch.
Here is one failed subtest :
{code}
testCheckAndDelete_ThatDeleteWasWritten(org.apache.hadoop.hbase.regionserver.TestHRegion)
Time elapsed: 2.373 sec <<< ERROR!
java.lang.IllegalStateException: failed to create a child event loop
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4853)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4844)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4835)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testCheckAndDelete_ThatDeleteWasWritten(TestHRegion.java:2034)
Caused by: org.apache.hbase.thirdparty.io.netty.channel.ChannelException:
failed to open a new selector
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4853)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4844)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4835)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testCheckAndDelete_ThatDeleteWasWritten(TestHRegion.java:2034)
Caused by: java.io.IOException: Too many open files
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4853)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4844)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.initHRegion(TestHRegion.java:4835)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testCheckAndDelete_ThatDeleteWasWritten(TestHRegion.java:2034)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [DISCUSS] Gathering metrics on HBase versions in use
What about following, collection is still on the download page, but there we can attach a link for survey as Andrew did, and describe community's intention and emphasize voluntary and anonymity. Inspired by Peter that there are a lot of learning|test cases, here i come up with two questions: profession: engineer/researcher/student/others version: for engineer, ask about which version(s) in use in production; for the rest, ask which version they are going to download. It doesn't have to be a third party survey link, it (e.g. web application) can just generate an e-mail and send to PMC's mail list. -- Best regards, R.C From: Peter Somogyi Sent: 15 November 2018 16:48 To: dev@hbase.apache.org Subject: Re: [DISCUSS] Gathering metrics on HBase versions in use I like the idea to have some sort of metrics from the users. I agree with Allan that in many cases HBase cluster is in an internal network making the data collection difficult or not even possible. It could lead us to an incorrect view if these generally bigger clusters do not appear in the metrics but is full with stats from standalone test environments that were just started once and never again. Collecting download information could give us a better picture but in this statistics the latest version might be overrepresented and we won't know which releases are currently used in the field. What do you think about collecting page views of Reference guide tied to specific releases? Someone searching in 1.4 Ref guide probably using HBase 1.4 or in the process of setting it up. Thanks, Peter On Thu, Nov 15, 2018 at 4:56 AM 张铎(Duo Zhang) wrote: > +1 on collecting the download information. > > And collecting data when starting up is a bit dangerous I'd say, both > technically and legally... > > Maybe a possible way is to add a link on the master state page, or some > ASCII arts in the master start log, to guide the people to our survey? > > Allan Yang 于2018年11月15日周四 上午11:23写道： > > > I also think having metrics about the downloads from Apache/archives is a > > doable action. Most HBase clusters are running in user's Intranet with no > > public access, sending anonymous data from them may not be possible. And > > also we need to find a way to obtain their authorization I think... > > Best Regards > > Allan Yang > > > > Zach York 于2018年11月15日周四 上午5:35写道： > > > > > Can we have metrics around the downloads from Apache/archives? I'm not > > sure > > > how that is all set up, but might be a low cost way to get some > metrics. > > > > > > On Wed, Nov 14, 2018, 12:12 PM Andrew Purtell > wrote: > > > > > > > While it seems you are proposing some kind of autonomous ongoing > usage > > > > metrics collection, please note I ran an anonymous version usage > survey > > > via > > > > surveymonkey for 1.x last year. It was opt in and there were no PII > > > > concerns by its nature. All of the issues around data collection, > > > storage, > > > > and processing were also handled (by surveymonkey). Unfortunately I > > > > recently cancelled my account. > > > > > > > > For occasional surveys something like that might work. Otherwise > there > > > are > > > > a ton of questions: How do we generate the data? How do we get > per-site > > > > opt-in permission? How do we collect the data? Store it? Process it? > > > Audit > > > > it? Seems more trouble than it's worth and requires ongoing volunteer > > > > hosting and effort to maintain. > > > > > > > > > > > > On Wed, Nov 14, 2018 at 11:47 AM Misty Linville > > > wrote: > > > > > > > > > When discussing the 2.0.x branch in another thread, it came up that > > we > > > > > don’t have a good way to understand the version skew of HBase > across > > > the > > > > > user base. Metrics gathering can be tricky. You don’t want to > capture > > > > > personally identifiable information (PII) and you need to be > > > transparent > > > > > about what you gather, for what purpose, how long the data will be > > > > > retained, etc. The data can also be sensitive, for instance if a > > large > > > > > number of installations are running a version with a CVE or known > > > > > vulnerability against it. If you gather metrics, it really needs to > > be > > > > > opt-out rather than opt-in so that you actually get a reasonable > > amount > > > > of > > > > > data. You also need to stand up some kind of metrics-gathering > > service > > > > and > > > > > run it somewhere, and some kind of reporting / visualization > tooling. > > > The > > > > > flip side of all these difficulties is a more intelligent way to > > decide > > > > > when to retire a branch or when to communicate more broadly / > loudly > > > > asking > > > > > people in a certain version stream to upgrade, as well as where to > > > > > concentrate our efforts. > > > > > > > > > > I’m not sticking my hand up to implement such a monster. I only > > wanted > > > to > > > > > open a discussion
Re: [DISCUSS] Gathering metrics on HBase versions in use
I like the idea to have some sort of metrics from the users. I agree with Allan that in many cases HBase cluster is in an internal network making the data collection difficult or not even possible. It could lead us to an incorrect view if these generally bigger clusters do not appear in the metrics but is full with stats from standalone test environments that were just started once and never again. Collecting download information could give us a better picture but in this statistics the latest version might be overrepresented and we won't know which releases are currently used in the field. What do you think about collecting page views of Reference guide tied to specific releases? Someone searching in 1.4 Ref guide probably using HBase 1.4 or in the process of setting it up. Thanks, Peter On Thu, Nov 15, 2018 at 4:56 AM 张铎(Duo Zhang) wrote: > +1 on collecting the download information. > > And collecting data when starting up is a bit dangerous I'd say, both > technically and legally... > > Maybe a possible way is to add a link on the master state page, or some > ASCII arts in the master start log, to guide the people to our survey? > > Allan Yang 于2018年11月15日周四 上午11:23写道： > > > I also think having metrics about the downloads from Apache/archives is a > > doable action. Most HBase clusters are running in user's Intranet with no > > public access, sending anonymous data from them may not be possible. And > > also we need to find a way to obtain their authorization I think... > > Best Regards > > Allan Yang > > > > Zach York 于2018年11月15日周四 上午5:35写道： > > > > > Can we have metrics around the downloads from Apache/archives? I'm not > > sure > > > how that is all set up, but might be a low cost way to get some > metrics. > > > > > > On Wed, Nov 14, 2018, 12:12 PM Andrew Purtell > wrote: > > > > > > > While it seems you are proposing some kind of autonomous ongoing > usage > > > > metrics collection, please note I ran an anonymous version usage > survey > > > via > > > > surveymonkey for 1.x last year. It was opt in and there were no PII > > > > concerns by its nature. All of the issues around data collection, > > > storage, > > > > and processing were also handled (by surveymonkey). Unfortunately I > > > > recently cancelled my account. > > > > > > > > For occasional surveys something like that might work. Otherwise > there > > > are > > > > a ton of questions: How do we generate the data? How do we get > per-site > > > > opt-in permission? How do we collect the data? Store it? Process it? > > > Audit > > > > it? Seems more trouble than it's worth and requires ongoing volunteer > > > > hosting and effort to maintain. > > > > > > > > > > > > On Wed, Nov 14, 2018 at 11:47 AM Misty Linville > > > wrote: > > > > > > > > > When discussing the 2.0.x branch in another thread, it came up that > > we > > > > > don’t have a good way to understand the version skew of HBase > across > > > the > > > > > user base. Metrics gathering can be tricky. You don’t want to > capture > > > > > personally identifiable information (PII) and you need to be > > > transparent > > > > > about what you gather, for what purpose, how long the data will be > > > > > retained, etc. The data can also be sensitive, for instance if a > > large > > > > > number of installations are running a version with a CVE or known > > > > > vulnerability against it. If you gather metrics, it really needs to > > be > > > > > opt-out rather than opt-in so that you actually get a reasonable > > amount > > > > of > > > > > data. You also need to stand up some kind of metrics-gathering > > service > > > > and > > > > > run it somewhere, and some kind of reporting / visualization > tooling. > > > The > > > > > flip side of all these difficulties is a more intelligent way to > > decide > > > > > when to retire a branch or when to communicate more broadly / > loudly > > > > asking > > > > > people in a certain version stream to upgrade, as well as where to > > > > > concentrate our efforts. > > > > > > > > > > I’m not sticking my hand up to implement such a monster. I only > > wanted > > > to > > > > > open a discussion and see what y’all think. It seems to me that a > few > > > > > must-haves are: > > > > > > > > > > - Transparency: Release notes, logging about the status of > > > > > metrics-gathering (on or off) at master or RS start-up, logging > about > > > > > exactly when and what metrics are sent > > > > > - Low frequency: Would we really need to wake up and send metrics > > more > > > > > often than weekly? > > > > > - Conservative approach: Only collect what we can find useful > today, > > > > don’t > > > > > collect the world. > > > > > - Minimize PII: This probably means not trying to group together > > > > > time-series results for a given server or cluster at all, but could > > > make > > > > > the data look like there were a lot more clusters running in the > > world > > > > than > > > > > really are. > > > > > - Who has access to the data?
Re: [DISCUSS] Gathering metrics on HBase versions in use
Hi All, Just to add my 2 cents. I have to agree with the people before me that it's quite impossible to get runtime data of HBase clusters so we have to look for workarounds. Collecting data of downloads and aggregated visitor informations of wiki pages or reference guides are also useful, but still they don't provide enough information. In my last job we have faced the same issue and ended up with a solution embedded in the application which provided a way for sysadmins to get the needed data in a processable format for us (json). So I would suggest to add a functionality to HBase which would collect *aggregated and anonymised* data of the HBase cluster setup (no DNS, nothing which the users might want to keep secret) only version numbers, number of servers, etc. (the content may be parametrized). This functionality could be called *manually* and the result could be uploaded to a place provided by the HBase community for analysis. This approach still has a problem, it's depends on manual interaction, but since the data can be verified by the users it increases the probability of the uploads itself. I don't think we can have a way to collect data like Apache Ambari or Cloudera Manager does, but still we could get more insight this way. Regards, Tamaas On Thu, Nov 15, 2018 at 10:40 AM Reid Chan wrote: > What about following, collection is still on the download page, but there > we can attach a link for survey as Andrew did, and describe community's > intention and emphasize voluntary and anonymity. > > Inspired by Peter that there are a lot of learning|test cases, here i come > up with two questions: > profession: engineer/researcher/student/others > version: for engineer, ask about which version(s) in use in > production; for the rest, ask which version they are going to download. > > It doesn't have to be a third party survey link, it (e.g. web application) > can just generate an e-mail and send to PMC's mail list. > > > -- > > Best regards, > R.C > > > > > From: Peter Somogyi > Sent: 15 November 2018 16:48 > To: dev@hbase.apache.org > Subject: Re: [DISCUSS] Gathering metrics on HBase versions in use > > I like the idea to have some sort of metrics from the users. > > I agree with Allan that in many cases HBase cluster is in an internal > network making the data collection difficult or not even possible. It could > lead us to an incorrect view if these generally bigger clusters do not > appear in the metrics but is full with stats from standalone test > environments that were just started once and never again. > > Collecting download information could give us a better picture but in this > statistics the latest version might be overrepresented and we won't know > which releases are currently used in the field. > > What do you think about collecting page views of Reference guide tied to > specific releases? Someone searching in 1.4 Ref guide probably using HBase > 1.4 or in the process of setting it up. > > Thanks, > Peter > > On Thu, Nov 15, 2018 at 4:56 AM 张铎(Duo Zhang) > wrote: > > > +1 on collecting the download information. > > > > And collecting data when starting up is a bit dangerous I'd say, both > > technically and legally... > > > > Maybe a possible way is to add a link on the master state page, or some > > ASCII arts in the master start log, to guide the people to our survey? > > > > Allan Yang 于2018年11月15日周四 上午11:23写道： > > > > > I also think having metrics about the downloads from Apache/archives > is a > > > doable action. Most HBase clusters are running in user's Intranet with > no > > > public access, sending anonymous data from them may not be possible. > And > > > also we need to find a way to obtain their authorization I think... > > > Best Regards > > > Allan Yang > > > > > > Zach York 于2018年11月15日周四 上午5:35写道： > > > > > > > Can we have metrics around the downloads from Apache/archives? I'm > not > > > sure > > > > how that is all set up, but might be a low cost way to get some > > metrics. > > > > > > > > On Wed, Nov 14, 2018, 12:12 PM Andrew Purtell > > wrote: > > > > > > > > > While it seems you are proposing some kind of autonomous ongoing > > usage > > > > > metrics collection, please note I ran an anonymous version usage > > survey > > > > via > > > > > surveymonkey for 1.x last year. It was opt in and there were no PII > > > > > concerns by its nature. All of the issues around data collection, > > > > storage, > > > > > and processing were also handled (by surveymonkey). Unfortunately I > > > > > recently cancelled my account. > > > > > > > > > > For occasional surveys something like that might work. Otherwise > > there > > > > are > > > > > a ton of questions: How do we generate the data? How do we get > > per-site > > > > > opt-in permission? How do we collect the data? Store it? Process > it? > > > > Audit > > > > > it? Seems more trouble than it's worth and requires ongoing > volunteer > >
Re: [ANNOUNCE] Please welcome Zach York to the HBase PMC
Congratulations Zach.
-
Srinivas
- Typed on tiny keys. pls ignore typos.{mobile app}
On Thu 15 Nov, 2018, 12:48 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com wrote:
> Congratulations Zach!!!
>
> On Thu, Nov 15, 2018 at 3:10 AM Misty Linville wrote:
>
> > Thanks for your continuing and increasing commitment to the project and
> the
> > community!
> >
> > On Wed, Oct 17, 2018, 11:14 PM OpenInx >
> > > Congratulations Zach!
> > >
> > > On Wed, Oct 17, 2018 at 8:14 PM ramkrishna vasudevan <
> > > ramkrishna.s.vasude...@gmail.com> wrote:
> > >
> > > > Congratulations Zach!!!
> > > >
> > > > On Wed, Oct 17, 2018 at 11:15 AM Yu Li wrote:
> > > >
> > > > > Congratulations and welcome, Zach!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Wed, 17 Oct 2018 at 13:02, Pankaj kr
> > wrote:
> > > > >
> > > > > > Congratulations Zach..!! :)
> > > > > >
> > > > > > Regards,
> > > > > > Pankaj
> > > > > >
> > > > > > -Original Message-
> > > > > > From: Sean Busbey [mailto:bus...@apache.org]
> > > > > > Sent: 12 October 2018 01:32
> > > > > > To: dev
> > > > > > Subject: [ANNOUNCE] Please welcome Zach York to the HBase PMC
> > > > > >
> > > > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > Zach
> > > > York
> > > > > > has accepted our invitation to become a PMC member on the Apache
> > > HBase
> > > > > > project. We appreciate Zach stepping up to take more
> responsibility
> > > in
> > > > > the
> > > > > > HBase project.
> > > > > >
> > > > > > Please join me in welcoming Zach to the HBase PMC!
> > > > > >
> > > > > > As a reminder, if anyone would like to nominate another person
> as a
> > > > > > committer or PMC member, even if you are not currently a
> committer
> > or
> > > > PMC
> > > > > > member, you can always drop a note to priv...@hbase.apache.org
> to
> > > let
> > > > us
> > > > > > know.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [ANNOUNCE] Please welcome Zach York to the HBase PMC
Congratulations Zach!
On Thu, Nov 15, 2018 at 12:53 PM Srinivas Reddy
wrote:
> Congratulations Zach.
>
> -
> Srinivas
>
> - Typed on tiny keys. pls ignore typos.{mobile app}
>
> On Thu 15 Nov, 2018, 12:48 ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com wrote:
>
> > Congratulations Zach!!!
> >
> > On Thu, Nov 15, 2018 at 3:10 AM Misty Linville wrote:
> >
> > > Thanks for your continuing and increasing commitment to the project and
> > the
> > > community!
> > >
> > > On Wed, Oct 17, 2018, 11:14 PM OpenInx > >
> > > > Congratulations Zach!
> > > >
> > > > On Wed, Oct 17, 2018 at 8:14 PM ramkrishna vasudevan <
> > > > ramkrishna.s.vasude...@gmail.com> wrote:
> > > >
> > > > > Congratulations Zach!!!
> > > > >
> > > > > On Wed, Oct 17, 2018 at 11:15 AM Yu Li wrote:
> > > > >
> > > > > > Congratulations and welcome, Zach!
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Wed, 17 Oct 2018 at 13:02, Pankaj kr
> > > wrote:
> > > > > >
> > > > > > > Congratulations Zach..!! :)
> > > > > > >
> > > > > > > Regards,
> > > > > > > Pankaj
> > > > > > >
> > > > > > > -Original Message-
> > > > > > > From: Sean Busbey [mailto:bus...@apache.org]
> > > > > > > Sent: 12 October 2018 01:32
> > > > > > > To: dev
> > > > > > > Subject: [ANNOUNCE] Please welcome Zach York to the HBase PMC
> > > > > > >
> > > > > > > On behalf of the Apache HBase PMC I am pleased to announce that
> > > Zach
> > > > > York
> > > > > > > has accepted our invitation to become a PMC member on the
> Apache
> > > > HBase
> > > > > > > project. We appreciate Zach stepping up to take more
> > responsibility
> > > > in
> > > > > > the
> > > > > > > HBase project.
> > > > > > >
> > > > > > > Please join me in welcoming Zach to the HBase PMC!
> > > > > > >
> > > > > > > As a reminder, if anyone would like to nominate another person
> > as a
> > > > > > > committer or PMC member, even if you are not currently a
> > committer
> > > or
> > > > > PMC
> > > > > > > member, you can always drop a note to priv...@hbase.apache.org
> > to
> > > > let
> > > > > us
> > > > > > > know.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
