Re: If anyone has experience in changing the regex engine in RegexStringComparator to joni?

[Sean Busbey]

That sounds reasonable. Could you file an issue in our issue tracker? Are
you up for working on a PR?


On Wed, Jul 13, 2022 at 2:27 AM Minwoo Kang 
wrote:

> Hello,
>
> I checked whether JONI can be used in RegexStringComparator.
> After changing the engine of RegexStringComparator to JONI, when a regex
> filter request was sent, the heap memory usage spiked and the RegionServer
> did not work due to GC.
>
> When I checked the reason, it is said that when using UTF8Encoding, an
> infinite loop can occur if an invalid UTF8 is entered.[1]
> For trino, using NonStrictUTF8Encoding instead of UTF8Encoding.
>
> After changing the encoding of JoniRegexEngine to NonStrictUTF8Encoding in
> RegexStringComparator, it was confirmed that the heap memory usage spike
> was gone.[2]
>
> In HBase, like trino, it seems to be necessary to use NonStrictUTF8Encoding
> instead of UTF8Encoding for JoniRegexEngine's encoding.
> What do you think about changing JoniRegexEngine's encoding to
> NonStrictUTF8Encoding?
>
> Best Regards,
> Minwoo
>
> On 2022/06/27 04:41:41 Minwoo Kang wrote:
> > (I sent the mail title in Korean for the first time. I'm so sorry.)
> >
> > Hello,
> >
> > Recently, java.util.regex in the Regex filter (RegexStringComparator) had
> > been running forever.
> > It is said that java.util.regex can run forever or stack overflow in the
> > worst case.
> >
> > Looking at RegexStringComparator, I saw that two regex implementations
> > (java, joni) were provided.
> > I was wondering if anyone has experience in changing the regex engine
> > in RegexStringComparator to joni and operating it.
> >
> > Best Regards,
> > Minwoo
> >
> > On 2022/06/27 04:37:11 Minwoo Kang wrote:
> > > Hello,
> > >
> > > Recently, java.util.regex in the Regex filter (RegexStringComparator)
> had
> > > been running forever.
> > > It is said that java.util.regex can run forever or stack overflow in
> the
> > > worst case.
> > >
> > > Looking at RegexStringComparator, I saw that two regex implementations
> > > (java, joni) were provided.
> > > I was wondering if anyone has experience in changing the regex engine
> > > in RegexStringComparator to joni and operating it.
> > >
> > > Best Regards,
> > > Minwoo
> > >
> >
>

Re: HBase Backup and Restore

[Sean Busbey]

These questions are more appropriate for the dev@HBase list because you are
asking how to make use of things that have note been sanctioned by the
project for downstream use yet.

If you’d like to see a particular timeline or want to see the feature
available in active releases it would be more effective to spend your time
helping Mallikarjun with outstanding issues. (Also something for the
dev@HBase list)

On Thu, Jul 14, 2022 at 6:00 PM anup ahire  wrote:

> Any particular timeline that we are expecting for 2.x backup ?
>
> We are wondering if we should wait for official support or backport
> ourselves. Is it possible for you to guide us on which PRs to backport?
>
> Thanks for your help. Really appreciate it.
>
> On Tue, Jul 12, 2022 at 6:14 PM Mallikarjun 
> wrote:
>
> > Backport is possible. We have backported to 2.1.x and using it.
> >
> > Waiting for some PRs to merge so that I can put some effort in getting
> the
> > backport available for active 2.x releases.
> >
> > On Tue, Jul 12, 2022, 10:42 PM anup ahire  wrote:
> >
> > > Thanks Mallikarjun. Will it be possible to backport this to 2.x ?
> trying
> > to
> > > understand if there are any hard dependencies on 3.0.0.
> > >
> > > On Tue, Jul 12, 2022 at 10:01 AM Mallikarjun  >
> > > wrote:
> > >
> > > > 3.0.0 should be the first release to include Backup and Restore. So
> far
> > > > there are no releases.
> > > >
> > > > On Tue, Jul 12, 2022, 10:26 PM anup ahire 
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I am trying to understand, why is
> > > > > https://github.com/apache/hbase/tree/master/hbase-backup part of
> > > master
> > > > > but
> > > > > getting left out from release tags ?
> > > > >
> > > > > Also, What is the minimum required version of the HBase to use
> Backup
> > > and
> > > > > Restore ? I am on 2.2.6
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Anup
> > > > >
> > > >
> > >
> >
>

[DISCUSS] looking for maintainers for HBase 1.y

[Sean Busbey]

Hi folks!

Over on dev@hbase we're discussing EOM for HBase 1.7.z and 1.y generally.

Our rate of contributions to HBase 1.y has slowed considerably over
the last year, as have releases.

If you are a current user of HBase 1 and you would like to see
continued releases, please subscribe to the dev@hbase list and join in
the discussion there about what it will take to keep releases
happening.

The specific thread has the subject "[DISCUSS] EOM 1.7 and branch-1?
(proposal: 2022/07/21)"

it is linked here on lists.a.o:
https://lists.apache.org/thread/8qy2lqtdwzc10tqnfdhrqvo35z6vkrvk

[ANNOUNCE] end of maintenance (EOM) for Apache HBase 2.3 releases

[Sean Busbey]

Hi folks!

The Apache HBase 2.3 release line reached the end of maintenance in
October 2021. Our apologies for the late notification.

The final release was 2.3.7 and is available from archive.apache.org.
No further releases from branch-2.3 are planned.

Users of HBase 2.3.z should upgrade to the current stable release
(currently 2.4.10) as quickly as possible.

Thanks to all the contributors who helped make HBase 2.3 possible over
its lifespan.

Re: HBase Logging Not Outputting to Desired File

[Sean Busbey]

Thanks for the problem report. This sounds like an edge case we should warn
folks about.



On Mon, Mar 7, 2022 at 7:09 PM Claude M  wrote:

> The problem had to do w/ putting hbck2-1.1.0.jar in the HBase lib
> directory.  This was causing a conflict and moving it out of that directory
> solved the issue.
>
> On Mon, Mar 7, 2022 at 10:01 AM Claude M  wrote:
>
> > Hello,
> >
> > I recently upgraded HBase from 1.3.2 to 2.3.5 in an environment and the
> > logging is not working properly.  I've configured the logging to go to
> > /var/log/hbase/hbase.log yet the log messages are going to syslog.
> Looking
> > at the process, it does show the log configuration is set correctly:
> >
> > opt/java/oracle/jre1.8.0_271/bin/java -Dproc_master
> > -XX:OnOutOfMemoryError=kill -9 %p -Djava.net.preferIPv4Stack=true
> > -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:-ResizePLAB
> > -XX:MaxGCPauseMillis=100 -XX:ParallelGCThreads=6 -verbose:gc
> > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/extra/gc.out
> > -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false -Xmx768m
> > -Dcom.sun.management.jmxremote.port=10102 -Dhbase.log.dir=/var/log/hbase
> > -Dhbase.log.file=hbase.log -Dhbase.home.dir=/opt/hbase-2.3.5
> > -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA
> > -Djava.library.path=/opt/hbase-2.3.5/lib/native/Linux-amd64-64
> > -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster
> > start
> >
> > In another environment in which I have HBase 1.3.2 installed, the logging
> > is going to the correct location /var/log/hbase/hbase.log.  What has
> > changed in HBase 2.3.5 to cause the logging to behave this way?
> >
> >
> > Thanks
> >
>

Re: [DISCUSS] Keep EOL HBase and Hadoop versions in our support matrix for longer time

[Sean Busbey]

We already publish version specific reference guides as a part of our
binary tarballs. My understanding is that's a big part of why they're
still in the main source repository; that we get docs built as a part
of our assembly.

e.g. just picking a 2.4 release I have handy:

(base) sbusbey@seans-mbp ~ % tar tzf Downloads/hbase-2.4.8-bin.tar.gz|
grep pdf
hbase-2.4.8/docs/apache_hbase_reference_guide.pdf
(base) sbusbey@seans-mbp ~ % tar tzf Downloads/hbase-2.4.8-bin.tar.gz|
grep book.html
hbase-2.4.8/docs/book.html

Historically these do get maintained for each minor version. My
understanding is that the diligence on backporting across both
contributors to docs and release managers has varied considerably over
time.

If we're only going to have one version of the docs, then we should
break the docs out of the main repo entirely so that we can simplify
their generation.

IIRC we have only gone through the trouble of publishing to the
website version specific API docs / ref guides for release lines that
made it to be the "stable" branch.

On Sat, Jan 8, 2022 at 9:37 AM 张铎(Duo Zhang)  wrote:
>
> I agree with Andrew that maybe it is beyond our ability to maintain
> ref guides for different release lines and keep them all in sync...
>
> What about this, we just make the ref guide for the master branch in
> sync, which contains all release lines. We will still remove
> information of the EOL releases as needed, to keep the ref guide
> clean, but as said in the title, less aggressive.
> And when we decide to EOL a release line, we copy the ref guide of the
> current master branch to the specific branch, and generate the ref
> guide for that release line for the last time.
> In this way the release manager does not need to always think of
> keeping the ref guide in sync, we all just need to consider the master
> one, only one extra work needs to be done when EOLing a release line.
>
> WDYT?
>
> Thanks.
>
> Andrew Purtell  于2022年1月8日周六 07:16写道：
> >
> > There are some challenges with respect to keeping multiple versions of the
> > documentation around. Each minor release needs a new version? Each RM
> > managing one or more code line(s) needs to update trunk docs and also
> > backport to branch docs (or not, depending)? There's been a JIRA open
> > forever for a 2.4 version of the book and honestly I haven't found time to
> > do it, because it seems both low priority and nontrivial, and there's never
> > enough time for everything... although that may be a personal failing.
> >
> > On Fri, Jan 7, 2022 at 9:05 AM Sean Busbey  wrote:
> >
> > > We should have a version specific version of the ref guide that
> > > contains that information.
> > >
> > > e.g.
> > >
> > > https://hbase.apache.org/1.4/book.html#hadoop
> > >
> > > https://hbase.apache.org/2.3/book.html#hadoop
> > >
> > > Can we do a better job of making these discoverable to folks rather
> > > than keeping stuff around?
> > >
> > > On Fri, Jan 7, 2022 at 1:14 AM 张铎(Duo Zhang) 
> > > wrote:
> > > >
> > > > Recently we've seen several emails on the user list asking whether
> > > > some hbase versions support specific hadoop versions, usually it will
> > > > be some versions which are already EOL, so there is no information in
> > > > our ref guide.
> > > >
> > > > For me I think we could leave the EOL versions in the support matrix
> > > > for a bit longer. It will be useful for our users.
> > > >
> > > > Thanks.
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk

Re: [DISCUSS] Keep EOL HBase and Hadoop versions in our support matrix for longer time

[Sean Busbey]

We should have a version specific version of the ref guide that
contains that information.

e.g.

https://hbase.apache.org/1.4/book.html#hadoop

https://hbase.apache.org/2.3/book.html#hadoop

Can we do a better job of making these discoverable to folks rather
than keeping stuff around?

On Fri, Jan 7, 2022 at 1:14 AM 张铎(Duo Zhang)  wrote:
>
> Recently we've seen several emails on the user list asking whether
> some hbase versions support specific hadoop versions, usually it will
> be some versions which are already EOL, so there is no information in
> our ref guide.
>
> For me I think we could leave the EOL versions in the support matrix
> for a bit longer. It will be useful for our users.
>
> Thanks.

[ANNOUNCE] Apache HBase Operator Tools 1.1.0 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase Operator Tools 1.1.0.

Apache HBase™ Operator Tools provides HBCK2 which is the repair tool for
Apache HBase 2 clusters.
To learn more about HBase and HBase Operator Tools, see
https://hbase.apache.org/.

The full list of issues can be found in the included CHANGES.md and
RELEASENOTES.md, or via our issue tracker:

https://s.apache.org/hbase-operator-tools-1.1.0-jira

To download please follow the links and instructions on our website:

https://hbase.apache.org/downloads.html

Question, comments, and problems are always welcome at: d...@hbase.apache.org

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

Re: problem building hbase 3.0.0

[Sean Busbey]

A few things:

1) it sounds like you do not need to work on the under development hbase
codebase, so I'd recommend focusing on using an hbase 2.x release
(preferably 2.3.4 since that's what you mention running)

If something comes up later where you do need the under development
codebase, please bring that up on dev@hbase (folks not on that mailing list
should only use the things the project has published to downstream)

2) for HBase 2.3.4 you should be able to rely on released artifacts.

Our download page has links for both the source and already built
convenience artifacts for HBase 2.3.4

http://hbase.apache.org/downloads.html

You can find the client source files there.

3) if your application uses maven then you should add a dependency at
compile scope for hbase-shaded-client 2.3.4 and at test scope for
hbase-shaded-testing-util 2.3.4 if you have unit tests that need a mini
hbase.


If that doesn't work then please give us specific failure conditions and
details so we can make sure the easy/expected path to use is functional.


On Fri, Jan 29, 2021, 15:43 richard t  wrote:

>  Hi Josh, first of all thanks for any help you can provide... I do
> appreciate it.
> `-DskipTests` is the standard Maven "ism" to skip tests. Your output
> appears to indicate that you were running tests, so perhaps your
> invocation was incorrect? You've not provided enough information for us
> to know why exactly your build failed.
> well, in the pom file, it does say that you should use the invocation "mvn
> package" to build.as far as the -DskipTests, I did use that BUT there
> were some tests still being run in the actual build, my assumption is that
> there were some poms that might have overridden this but I would have to
> look...
> it does turn out that the specific one that I had the problem hbase-http,
> I did set the configuration to skipTests and that did work...
>
> You do not have to build HBase from source in order to build an
> application. Every official release which this project creates has
> artifacts which are published in Maven Central for you to consume directly.
> ok, except that I when building just the client source code I couldn't
> find this version (3.0.0-SNAPSHOT) for a dependencywhich I did open a jira
> ticket on that...
> another guy named sean suggested I need to build from the top down in
> order for some of those dependencies to properly builton the apache web
> site, it does have a link that seems to point to that dependency but
> unfortunately in going to the link it returned a 404
>
> One final note: you are building from the master branch (3.0.0-SNAPSHOT)
> which is still under development. I'd suggest that you add a dependency
> in your application to the Maven dependency
> org.apache.hbase:hbase-client for the version of HBase that you are
> actually running.
> I have an actual running hbase (2.3.4) that does run but (and this is an
> important but)... I can't seem to get a client to build with the
> hbase-clientalone, and I couldn't seem to find the client source code to
> use to test out the java api / connect to my version of hbase...any help
> you can provide on that would be the real solution...
>
> If you do not yet have a HBase which is already running, you can
> download a pre-built release from
> https://hbase.apache.org/downloads.html. Be sure to grab a version of
> Hadoop which is compatible with the HBase release which you are running.
> Thanks again for the response!RichardOn Friday, January 29, 2021,
> 04:19:42 PM EST, Josh Elser  wrote:
>
>  `-DskipTests` is the standard Maven "ism" to skip tests. Your output
> appears to indicate that you were running tests, so perhaps your
> invocation was incorrect? You've not provided enough information for us
> to know why exactly your build failed.
>
> You do not have to build HBase from source in order to build an
> application. Every official release which this project creates has
> artifacts which are published in Maven Central for you to consume directly.
>
> One final note: you are building from the master branch (3.0.0-SNAPSHOT)
> which is still under development. I'd suggest that you add a dependency
> in your application to the Maven dependency
> org.apache.hbase:hbase-client for the version of HBase that you are
> actually running.
>
> If you do not yet have a HBase which is already running, you can
> download a pre-built release from
> https://hbase.apache.org/downloads.html. Be sure to grab a version of
> Hadoop which is compatible with the HBase release which you are running.
>
> On 1/29/21 8:54 AM, richard t wrote:
> > hi,
> > my ultimate goal is to have a basic CRUD client to hbase using the java
> api.
> > in trying to get there, I downloaded the source for hbase 3.0.0 because
> of the client source and wanted to build this client and test it against my
> hbase instance that is running.
> > the hbase source is not building due to this error:BUILD FAILURE
> >
> [INFO] 
> 

Re: Region server idle

[Sean Busbey]

You can change the logging for just the balancer system so that not so
much goes into the logs. The logger to change is
org.apache.hadoop.hbase.master.balancer

On Tue, Jan 26, 2021 at 9:46 AM Marc Hoppins  wrote:
>
> I did change it on the master. The log filled up with a bunch of reads and 
> writes, so I discontinued it after about an hour.  Certainly, this was 
> because another IT-er had kicked off HDFS balance. Obviously, I didn't really 
> want to try running one tool while the other was already running.
>
> -Original Message-
> From: Sean Busbey 
> Sent: Tuesday, January 26, 2021 3:59 PM
> To: Hbase-User 
> Subject: Re: Region server idle
>
> EXTERNAL
>
> Hi Marc!
>
> Did turning up the balancer's logging confirm that it is actually making 
> plans? Did it confirm what weights are being used for the various balancer 
> functions?
>
> On Tue, Jan 26, 2021 at 8:47 AM Marc Hoppins  wrote:
> >
> > Region counts have crept up (very slowly) to
> >
> > Hbase19 - 15 regions
> > Hbase20 - 8 regions
> >
> > So, would either continual reads or, (more likely) continual writes have 
> > such an effect on region splits and/or hbase balancing ?
> >
> > It is confusing as 3 new servers (NN, RS) were added to the cluster back in 
> > October 2020 and they were well-integrated within a day or so.
> >
> > -Original Message-
> > From: Josh Elser 
> > Sent: Sunday, January 24, 2021 10:22 PM
> > To: user@hbase.apache.org
> > Subject: Re: Region server idle
> >
> > EXTERNAL
> >
> > Yes, each RegionServer has its own writeahead log (which is named with that 
> > RS' hostname).
> >
> > You'd want to look at the HBase master log, specifically for reasons as to 
> > why balancing is not naturally happening. It very well could be that you 
> > have other regions in transition (which may prevent balancing from 
> > happening). This is just one reason balancing may not naturally happening, 
> > but you should be able to see this in the active master log (potentially 
> > with enabling DEBUG on org.apache.hadoop.hbase, first).
> > Don't forget about the hbase shell command to request the balancer to 
> > immediately run (so you can look for that logging at a specific point in 
> > time).
> >
> > On 1/18/21 7:23 AM, Marc Hoppins wrote:
> > > I have been checking for days and there are no outstanding RITs.  Region 
> > > servers do not have their own WAL files, do they?
> > >
> > > What gives me pause is that, although the affected servers (hbase19 & 
> > > hbase20) have 11 and 3 regions respectively, there must be very little 
> > > useable data as the requests per second are negligible for hbase19 and 
> > > zero  for hbase20.
> > >
> > > I would have expected SOME movement to distribute data and work onto 
> > > these 'vacant' systems after more than 2 weeks.
> > >
> > > The circumstance behind hbase19 going offline is that a memory module had 
> > > vailed and dump data was constantly filling up tmp storage so the oncall 
> > > guy made the decision to shut the system down. Given that a lot of hbase 
> > > work is done in memory is there any possible way something still lingers 
> > > in memory somewhere that has not been flushed?
> > >
> > > As for hbase20, an IT guy decommissioned the host in the Cloudera console 
> > > and recommissioned it as a test to see if region balancing proceeded as 
> > > normal. Obviously, it hasn't. For obvious reasons, a second test has not 
> > > been performed.
> > >
> > > -Original Message-
> > > From: Josh Elser 
> > > Sent: Tuesday, January 12, 2021 4:56 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: Region server idle
> > >
> > > EXTERNAL
> > >
> > > Yes, in general, HDFS rebalancing will cause a decrease in the 
> > > performance of HBase as it removes the ability for HBase to short-circuit 
> > > some read logic. It should not, however, cause any kind of errors or lack 
> > > of availability.
> > >
> > > You should feel free to investigate the RITs you have now, rather than 
> > > wait for a major compaction to finish. As a reminder, you can also force 
> > > one to happen now via the `major_compact` HBase shell command, for each 
> > > table (or at least the tables which are most important). Persistent RITs 
> > > will prevent balancing from happening, that may be your smoking gun.
> > >
> > > It may also be helpful for

Re: Region server idle

[Sean Busbey]

erberos tgt)] [Caused by javax.security.sasl.SaslException: GSS initiate 
> >>> failed [Caused by GSSException: No valid credentia
> >>>   
> >>>ls provided (Mechanism level: 
> >>> Failed to find any Kerberos tgt)]]
> >>>
> >>> -Original Message-
> >>> From: Stack 
> >>> Sent: Saturday, January 9, 2021 1:52 AM
> >>> To: Hbase-User 
> >>> Subject: Re: Region server idle
> >>>
> >>> EXTERNAL
> >>>
> >>> Looking at code around exception, can you check your quota settings? See 
> >>> refguide on how to list quotas. Look for table or namespace that is empty 
> >>> or non-existant and fill in missing portion.
> >>>
> >>> This is master-side log? It is from a periodic task so perhaps something 
> >>> else is in the way of the non-assign? Anything else in there about 
> >>> balancing or why we are skipping assign to these servers? Try a balance 
> >>> run in the shell and then check master log to see why no work done?
> >>>
> >>> S
> >>>
> >>> On Fri, Jan 8, 2021 at 2:51 AM Marc Hoppins  wrote:
> >>>
> >>>> Apologies again.  Here is the full error message.
> >>>>
> >>>> 2021-01-08 11:34:15,831 ERROR org.apache.hadoop.hbase.ScheduledChore:
> >>>> Caught error
> >>>> java.lang.IllegalStateException: Expected only one of namespace and
> >>>> tablename to be null
> >>>>at
> >>>> org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:198)
> >>>>at
> >>>> org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:126)
> >>>>at
> >>>> org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:113)
> >>>>at
> >>>> org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> >>>>at
> >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>>>at 
> >>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >>>>at
> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >>>>at
> >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >>>>at
> >>>> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> >>>>at
> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>>>at
> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>>>at java.lang.Thread.run(Thread.java:748)
> >>>>
> >>>> -Original Message-
> >>>> From: Marc Hoppins 
> >>>> Sent: Friday, January 8, 2021 10:57 AM
> >>>> To: user@hbase.apache.org
> >>>> Subject: RE: Region server idle
> >>>>
> >>>> EXTERNAL
> >>>>
> >>>> So, I tried decommission that RS and recommission it.  No change.
> >>>> Server still idle.
> >>>>
> >>>> Tried decommission another server and see if HBASE sets itself right.
> >>>> Now I have two RS that are idle.
> >>>>
> >>>> ba-hbase18.jumbo.hq.com,16020,1604413480001 Tue Nov 03 15:24:40 CET
> >>>> 20201 s 2.1.0-cdh6.3.2  13  471
> >>>> ba-hbase19.jumbo.hq.com,16020,1610095488001 Fri Jan 08 09:44:48 CET
> >>>> 20210 s 2.1.0-cdh6.3.2  0   6
> >>>> ba-hbase20.jumbo.hq.com,16020,1610096850259 Fri Jan 08 10:07:30 CET
> >>>> 20210 s 2.1.0-cdh6.3.2  0   0
> >>>> ba-hbase21.jumbo.hq.com,16020,1604414101652 Tue Nov 03 15:35:01 CET
> >>>> 20201 s 2.1.0-cdh6.3.2  15  447
> >>>>
> >>>>From the logs:
> >>>> 2021-01-08 10:25:36,875

Re: Region server idle

[Sean Busbey]

Sounds like https://issues.apache.org/jira/browse/HBASE-24139

The description of that jira has a workaround.

On Thu, Jan 7, 2021, 05:23 Marc Hoppins  wrote:

> Hi all,
>
> I have a setup with 67 region servers. 29 Dec one system had to be shut
> down to have EMM module swapped out - which took one work day. Host was
> back online 30 Dec.
>
> My HBASE is very basic so I appreciate your patience.
>
> My understanding of the defaults that are setup is that a major compaction
> should occur every 7 days.  Moreover, do I assume that more extensive
> balancing may occur after this happens?
>
> When I check (via hbase master UI) the status of HBASE, I see the
> following:
>
> ServerName
>
> Start time
>
> Last contact
>
> Version
>
> Requests Per Second
>
> Num. Regions
>
> ba-hbase16.jumbo.hq.com,16020,1604413068640<
> http://ba-hbase16.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:17:48 CET 2020
>
> 3 s
>
> 2.1.0-cdh6.3.2
>
> 46
>
> 462
>
> ba-hbase17.jumbo.hq.com,16020,1604413274393<
> http://ba-hbase17.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:21:14 CET 2020
>
> 1 s
>
> 2.1.0-cdh6.3.2
>
> 19
>
> 462
>
> ba-hbase18.jumbo.hq.com,16020,1604413480001<
> http://ba-hbase18.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:24:40 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 62
>
> 461
>
> ba-hbase19.jumbo.hq.com,16020,1609326754985<
> http://ba-hbase19.jumbo.hq.eset.com:16030/rs-status>
>
> Wed Dec 30 12:12:34 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 0
>
> 0
>
> ba-hbase20.jumbo.hq.com,16020,1604413895967<
> http://ba-hbase20.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:31:35 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 62
>
> 503
>
> ba-hbase21.jumbo.hq.com,16020,1604414101652<
> http://ba-hbase21.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:35:01 CET 2020
>
> 3 s
>
> 2.1.0-cdh6.3.2
>
> 59
>
> 442
>
> ba-hbase22.jumbo.hq.com,16020,1604414308289<
> http://ba-hbase22.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:38:28 CET 2020
>
> 0 s
>
> 2.1.0-cdh6.3.2
>
> 40
>
> 438
>
>
> Why, after more than 7 days, is this host not hosting more (any) regions?
> Should I initiate some kind of rebalancing?
>
> Thanks in advance.
>
> M
>

Re: failed building hbase C++ native client

[Sean Busbey]

Please note that the Apache HBase community has not yet released a
downstream usable hbase-native-client.

If you would like to work with us towards making it ready please join the
dev@hbase mailing list and bring issues up there.



On Wed, Dec 16, 2020, 11:59 Bharath Vissapragada 
wrote:

> What commit of HBase and native-client are you building? I fixed that
> problem here
> <
> https://github.com/apache/hbase/commit/fe1fc25fba52b6b07454b4e1b6dc9afe533ecbbe
> >
> .
>
> On Wed, Dec 16, 2020 at 9:13 AM 杭研-闵涛  wrote:
>
> > Hi all, We are trying to use a C++ hbase client to access hbase service.
> > We found https://github.com/apache/hbase-native-client may be what we
> > needs. After prepared the building environment and successfully excuted
> > ‘cmake  .’ ,  the error occurred during processing ‘make -j’.
> >
> > Error info :
> > [  4%] Built target boost
> > [  4%] Built target ZooKeeper
> > [  6%] Built target facebook-folly-proj
> > [  8%] Built target facebook-wangle-proj
> > [  8%] Building CXX object
> > CMakeFiles/hbaseclient-static.dir/src/hbase/serde/rpc-serde.cc.o
> > [  9%] Building CXX object
> > CMakeFiles/hbaseclient-shared.dir/src/hbase/serde/rpc-serde.cc.o
> > /usr/src/hbase/hbase-native-client/src/hbase/serde/rpc-serde.cc:32:27:
> > fatal error: utils/version.h: No such file or directory
> > compilation terminated.
> > CMakeFiles/hbaseclient-static.dir/build.make:2254: recipe for target
> > 'CMakeFiles/hbaseclient-static.dir/src/hbase/serde/rpc-serde.cc.o' failed
> > make[2]: ***
> > [CMakeFiles/hbaseclient-static.dir/src/hbase/serde/rpc-serde.cc.o] Error
> 1
> > CMakeFiles/Makefile2:107: recipe for target
> > 'CMakeFiles/hbaseclient-static.dir/all' failed
> > make[1]: *** [CMakeFiles/hbaseclient-static.dir/all] Error 2
> > make[1]: *** Waiting for unfinished jobs
> > /usr/src/hbase/hbase-native-client/src/hbase/serde/rpc-serde.cc:32:27:
> > fatal error: utils/version.h: No such file or directory
> > compilation terminated.
> > CMakeFiles/hbaseclient-shared.dir/build.make:2254: recipe for target
> > 'CMakeFiles/hbaseclient-shared.dir/src/hbase/serde/rpc-serde.cc.o' failed
> > make[2]: ***
> > [CMakeFiles/hbaseclient-shared.dir/src/hbase/serde/rpc-serde.cc.o] Error
> 1
> > CMakeFiles/Makefile2:599: recipe for target
> > 'CMakeFiles/hbaseclient-shared.dir/all' failed
> > make[1]: *** [CMakeFiles/hbaseclient-shared.dir/all] Error 2
> > Makefile:138: recipe for target 'all' failed
> > make: *** [all] Error 2
> >  According to the error info, It seems forget to put utils/version.h into
> > including directories. I tried to find the version.h but failed. Have
> > anyone
> > successfully built this C++ client library? Any advice would be
> > appreciated.
> >
> > Thanks,
> >
> > Mintao
> >
> >
>

Re: [DISCUSS] The breaking changes in HBASE-24966

[Sean Busbey]

I think we should change what they throw directly and label it
incompatible. I think this is in line with our previous expectation setting
about how we'll handle mistakes in the API.

That change would be source incompatible but would still be binary
compatible.

I think we should do it in a major release. esp since there's not a way in
Java to say "this deprecation is just about the thrown exceptions" and it
will be awkward to write code that is source compatible with the existing
api and with the exception removed.

On Sat, Nov 28, 2020, 22:07 张铎(Duo Zhang)  wrote:

> In HBASE-24966, we found that in AsyncTableRegionLocator, we accidentally
> declared 3 methods
>
> getStartKeys
> getEndKeys
> getStartEndKeys
>
> to throw IOException directly.
>
> This should be a copy paste mistake, as typically, for a method which
> returns CompletableFuture, the exception should be returned through the
> CompletableFuture, and this is exactly the behavior of these methods.
>
> So the actual problem is only that we have a wrong method signature. but
> since this interface is IA.Public, and it has already been included in
> several releases, according to our compatibility rule, we can not just
> remove the throws part from the method. Instead, we need to deprecate them
> and create new methods. But there will be another problem that we want to
> align the method names between the sync and async client, so if we change
> the names of the methods, we'd better also change the name of methods for
> sync client, which will make our users do more unnecessary work.
>
> So here I want to discuss that, since we all know that, this is a mistake,
> and the methods will never throw IOException directly, is it OK for us to
> just remove the throws part and tell users directly that this is a mistake,
> and release it in the next minor release or a major release as an
> incompatible change?
>
> Thanks.
>

Re: Hbase 2.2.5 Unable to load .regioninfo from table

[Sean Busbey]

Let me check back in my notes for what step we're missing.

On Thu, Sep 24, 2020 at 5:32 AM Martin Braun  wrote:
>
> Hello all,
>
> i digged a bit deeper into this:
>
> the WALPlayer is not able to replay recovered.edits files, the source code 
> http://hbase.apache.org/2.2/devapidocs/src-html/org/apache/hadoop/hbase/mapreduce/WALInputFormat.html
>
> seems to expect an endtime coded into the filename:
>
> long fileStartTime = Long.parseLong(name.substring(idx+1));
> 323if (fileStartTime <= endTime) {
> 324  LOG.info("Found: " + file);
> 325  result.add(file);
> 326}
> 327  } catch (NumberFormatException x) {
> 328idx = 0;
>
> But the files in recovered.edits are named differently (just a numbers like 
> 00195).
>
> I have also found also this issue:
>
> https://issues.apache.org/jira/browse/HBASE-22976
> [HBCK2] Add RecoveredEditsPlayer
>
> But what can I do now to fix this and replay the WAL files in the recovered 
> edits?
>
> Any ideas?
>
> best,
> Martin
>
> > On 22. Sep 2020, at 18:38, Sean Busbey  wrote:
> >
> > hurm. following the instructions from the reference guide works for
> > me. Is there a specific reason you're passing the
> > '--internal-classpath' flag? Do other hadoop jobs work?
> >
> > what if you submit it as a proper MR job? unfortunately the ref guide
> > is thin on explaining this atm, but it looks like:
> >
> > HADOOP_CLASSPATH="${HBASE_CONF_DIR}:$("${HBASE_HOME}/bin/hbase"
> > mapredcp)" yarn jar
> > "${HBASE_HOME}/lib/shaded-clients/hbase-shaded-mapreduce-2.2.5.jar"
> > WALPlayer some/path/to/wals/ 'some:example'
> >
> > On Tue, Sep 22, 2020 at 10:24 AM Martin Braun  
> > wrote:
> >>
> >> Hello Sean,
> >>
> >> thank you for you quick response!
> >>
> >> Replaying the wal files would be OK-  however I am struggling using the 
> >> WALPlayer:
> >>
> >>
> >> hbase --internal-classpath org.apache.hadoop.hbase.mapreduce.WALPlayer 
> >> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits
> >>  tt_ix_parent_item
> >> Exception in thread "main" java.lang.NoClassDefFoundError: 
> >> org/codehaus/jackson/map/JsonMappingException
> >>at org.apache.hadoop.mapreduce.Job.getJobSubmitter(Job.java:1325)
> >>at org.apache.hadoop.mapreduce.Job.submit(Job.java:1336)
> >>at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
> >>at 
> >> org.apache.hadoop.hbase.mapreduce.WALPlayer.run(WALPlayer.java:428)
> >>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> >>at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> >>at 
> >> org.apache.hadoop.hbase.mapreduce.WALPlayer.main(WALPlayer.java:417)
> >> Caused by: java.lang.ClassNotFoundException: 
> >> org.codehaus.jackson.map.JsonMappingException
> >>at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> >>at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> >>at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> >>at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> >>... 7 more
> >>
> >> Could you provide some hints how to use the WALPlayer correctly?
> >>
> >>
> >>
> >> best,
> >> Martin
> >>
> >>> On 22. Sep 2020, at 16:52, Sean Busbey  wrote:
> >>>
> >>> bulk loading stuff works with hfiles. recovered.edits files are
> >>> formatted the same as WAL files rather than as HFiles. for wal files
> >>> you can use the wal replayer to ensure those edits are all present in
> >>> the table.
> >>>
> >>> IIRC there is an unknown sequence of events that can result in the
> >>> recovered edits sticking around for a region after they've already
> >>> been recovered. Presuming your use case will work for having the same
> >>> edit played multiple times (basically if you do not mess about with
> >>> cell level timestamps or keeping multiple versions around) then it
> >>> should be fine to sideline those edits and then replay them using the
> >>> wal player.
> >>>
> >>> If your use case isn't fine with that, then you can use the wal pretty
> >>> printer to examine the edits th

Re: Hbase 2.2.5 Unable to load .regioninfo from table

[Sean Busbey]

hurm. following the instructions from the reference guide works for
me. Is there a specific reason you're passing the
'--internal-classpath' flag? Do other hadoop jobs work?

what if you submit it as a proper MR job? unfortunately the ref guide
is thin on explaining this atm, but it looks like:

HADOOP_CLASSPATH="${HBASE_CONF_DIR}:$("${HBASE_HOME}/bin/hbase"
mapredcp)" yarn jar
"${HBASE_HOME}/lib/shaded-clients/hbase-shaded-mapreduce-2.2.5.jar"
WALPlayer some/path/to/wals/ 'some:example'

On Tue, Sep 22, 2020 at 10:24 AM Martin Braun  wrote:
>
> Hello Sean,
>
> thank you for you quick response!
>
> Replaying the wal files would be OK-  however I am struggling using the 
> WALPlayer:
>
>
> hbase --internal-classpath org.apache.hadoop.hbase.mapreduce.WALPlayer 
> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits
>  tt_ix_parent_item
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/codehaus/jackson/map/JsonMappingException
> at org.apache.hadoop.mapreduce.Job.getJobSubmitter(Job.java:1325)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1336)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
> at org.apache.hadoop.hbase.mapreduce.WALPlayer.run(WALPlayer.java:428)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at 
> org.apache.hadoop.hbase.mapreduce.WALPlayer.main(WALPlayer.java:417)
> Caused by: java.lang.ClassNotFoundException: 
> org.codehaus.jackson.map.JsonMappingException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 7 more
>
> Could you provide some hints how to use the WALPlayer correctly?
>
>
>
> best,
> Martin
>
> > On 22. Sep 2020, at 16:52, Sean Busbey  wrote:
> >
> > bulk loading stuff works with hfiles. recovered.edits files are
> > formatted the same as WAL files rather than as HFiles. for wal files
> > you can use the wal replayer to ensure those edits are all present in
> > the table.
> >
> > IIRC there is an unknown sequence of events that can result in the
> > recovered edits sticking around for a region after they've already
> > been recovered. Presuming your use case will work for having the same
> > edit played multiple times (basically if you do not mess about with
> > cell level timestamps or keeping multiple versions around) then it
> > should be fine to sideline those edits and then replay them using the
> > wal player.
> >
> > If your use case isn't fine with that, then you can use the wal pretty
> > printer to examine the edits that are there and check to ensure the
> > cells are already in the table in a current region.
> >
> > sounds like we should update the troubleshooting tips to include some
> > coverage of stray recovered.edits files.
> >
> > On Tue, Sep 22, 2020 at 8:58 AM Martin Braun  
> > wrote:
> >>
> >> Hello all,
> >>
> >> I have an issue with hbase 2.2.5 (and hadoop-2.8.5) after a full disk 
> >> event I have 38 inconsistencies, when I do a
> >>
> >> hbase --internal-classpath hbck
> >>
> >> I get a bunch of these errors:
> >>
> >> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table 
> >> tt_ix_bizStep_inserting in hdfs dir 
> >> hdfs://localhost:9000/hbase/data/default/tt_ix_bizStep_inserting/8a1acb499bf454b072daeee5960daa73!
> >>   It may be an invalid format or version file.  Treating as an orphaned 
> >> regiondir.
> >> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table 
> >> tt_ix_bizStep_inserting in hdfs dir 
> >> hdfs://localhost:9000/hbase/data/default/tt_ix_bizStep_inserting/8f64025b68958ebddeb812297facdfc6!
> >>   It may be an invalid format or version file.  Treating as an orphaned 
> >> regiondir.
> >>
> >>
> >> When looking into these directories I see that there is indeed no 
> >> .regioninfo file:
> >>
> >> hdfs dfs -ls -R 
> >> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0
> >>
> >> drwxr-xr-x   - jenkins supergroup  0 2020-09-21 11:23 
> >> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c

Re: Hbase 2.2.5 Unable to load .regioninfo from table

[Sean Busbey]

bulk loading stuff works with hfiles. recovered.edits files are
formatted the same as WAL files rather than as HFiles. for wal files
you can use the wal replayer to ensure those edits are all present in
the table.

IIRC there is an unknown sequence of events that can result in the
recovered edits sticking around for a region after they've already
been recovered. Presuming your use case will work for having the same
edit played multiple times (basically if you do not mess about with
cell level timestamps or keeping multiple versions around) then it
should be fine to sideline those edits and then replay them using the
wal player.

If your use case isn't fine with that, then you can use the wal pretty
printer to examine the edits that are there and check to ensure the
cells are already in the table in a current region.

sounds like we should update the troubleshooting tips to include some
coverage of stray recovered.edits files.

On Tue, Sep 22, 2020 at 8:58 AM Martin Braun  wrote:
>
> Hello all,
>
> I have an issue with hbase 2.2.5 (and hadoop-2.8.5) after a full disk event I 
> have 38 inconsistencies, when I do a
>
> hbase --internal-classpath hbck
>
> I get a bunch of these errors:
>
> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table 
> tt_ix_bizStep_inserting in hdfs dir 
> hdfs://localhost:9000/hbase/data/default/tt_ix_bizStep_inserting/8a1acb499bf454b072daeee5960daa73!
>   It may be an invalid format or version file.  Treating as an orphaned 
> regiondir.
> ERROR: Orphan region in HDFS: Unable to load .regioninfo from table 
> tt_ix_bizStep_inserting in hdfs dir 
> hdfs://localhost:9000/hbase/data/default/tt_ix_bizStep_inserting/8f64025b68958ebddeb812297facdfc6!
>   It may be an invalid format or version file.  Treating as an orphaned 
> regiondir.
>
>
> When looking into these directories I see that there is indeed no .regioninfo 
> file:
>
> hdfs dfs -ls -R 
> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0
>
> drwxr-xr-x   - jenkins supergroup  0 2020-09-21 11:23 
> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits
> -rw-r--r--   3 jenkins supergroup  74133 2020-09-21 11:11 
> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/285
> -rw-r--r--   3 jenkins supergroup  74413 2020-09-16 19:03 
> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/286
> -rw-r--r--   3 jenkins supergroup  74693 2020-09-16 19:05 
> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/287
> -rw-r--r--   3 jenkins supergroup  79427 2020-09-16 18:27 
> hdfs://localhost:9000/hbase/data/default/tt_ix_parent_item/ae1553c4d6140110c51c535ba1dbc1a0/recovered.edits/305
>
>
> The hbck2 manual  from the hbase-operator tools tells me for Orphan Data to 
> read http://hbase.apache.org/book.html#arch.bulk.load.complete.strays, 
> chapter “72.4.1. 'Adopting' Stray Data"
>
> However it seems that this is another case a completebuldload on the named 
> directories seems to do nothing…
>
> A scan 'hbase:meta', {COLUMN=>'info:regioninfo’} does not show any errors.
>
>
> How can I resolve these inconsistencies of the missing .regioninfo?
>
> TIA
>
> best,
> Martin
>


-- 
Sean

Re: A tool to rewrite corrupted HFile

[Sean Busbey]

I think that would be a useful tool. It would also fit well into the
project's hbase-operator-tools repo.

Mind filing a jira to set some initial goals? Or look for suggestions on
initial goal over on dev@hbase?

On Wed, Aug 19, 2020, 18:10 Andrey Elenskiy
 wrote:

> Hello,
>
> Over years I have been dealing with corrupted hfiles by just removing them.
> It always seemed wasteful to throw away the entire HFile even only if one
> block is corrupted.
>
> Do you happen to know if a tool to rewrite the hfile with omitting/skipping
> corrupt blocks already exists? If not, would such a tool be of use to the
> community? The goal of such a tool is to try and recover as much data as
> possible. We can always drop data on the hbase block boundaries as well.
>
> Andrey
>

Re: Logo Merchandise

[Sean Busbey]

I do not recall the PMC approving it, probably should start with a heads-up
to private@hbase so we can check.


On Sat, Aug 1, 2020, 16:47 Mike Drob  wrote:

> Do we know if this usage was approved by VP Brand? Does the rest of this
> conversation need to happen in conjunction with trademark enforcement?
>
> On 2020/07/31 20:57:53, Tamas Penzes  wrote:
> > RedBubble does sell HBase swag:
> >
> https://www.redbubble.com/i/sticker/HBase-column-oriented-database-by-taivop/28077314.O9UDB
> > Sticker, t-shirt, etc.
> >
> > Regards, Tamaas
> > On Fri, Jul 31, 2020 at 7:42 PM Mike Drob  wrote:
> >
> > > HBase Community,
> > >
> > > Do we have official logo merchandise available for purchase anywhere? I
> > > know lots of other ASF projects use RedBubble, but I don't see HBase
> > > included on what I think is the official storefront.
> > >
> > > https://www.redbubble.com/people/comdev/shop
> > >
> > > Is there a different venue to buy? Preferably somewhere that still
> sends a
> > > portion of the proceeds back to the project/foundation.
> > >
> > > Mike
> > >
> >
>

Re: Logo Merchandise

[Sean Busbey]

seems the bar for additions is low if someone wants to get it going.

 - Did you know that you can have swag from your favorite Apache project
> added to the ComDev store on RedBubble
> https://www.redbubble.com/people/comdev ? Grab the project logo from
> http://www.apache.org/logos/ and request it be added by emailing
> d...@community.apache.org


from:

https://blogs.apache.org/foundation/date/20190809


On Fri, Jul 31, 2020 at 2:28 PM Sean Busbey  wrote:

> I do not know of any. All of the swag I can think of was made by
> interested third parties who got one-off approvals.
>
> On Fri, Jul 31, 2020 at 12:44 PM Mike Drob  wrote:
>
>> HBase Community,
>>
>> Do we have official logo merchandise available for purchase anywhere? I
>> know lots of other ASF projects use RedBubble, but I don't see HBase
>> included on what I think is the official storefront.
>>
>> https://www.redbubble.com/people/comdev/shop
>>
>> Is there a different venue to buy? Preferably somewhere that still sends a
>> portion of the proceeds back to the project/foundation.
>>
>> Mike
>>
>

Re: Logo Merchandise

[Sean Busbey]

I do not know of any. All of the swag I can think of was made by interested
third parties who got one-off approvals.

On Fri, Jul 31, 2020 at 12:44 PM Mike Drob  wrote:

> HBase Community,
>
> Do we have official logo merchandise available for purchase anywhere? I
> know lots of other ASF projects use RedBubble, but I don't see HBase
> included on what I think is the official storefront.
>
> https://www.redbubble.com/people/comdev/shop
>
> Is there a different venue to buy? Preferably somewhere that still sends a
> portion of the proceeds back to the project/foundation.
>
> Mike
>

Re: HBase 2.1.0 - NoSuchMethodException org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy

[Sean Busbey]

that is the detail message of why it determined that the FileSystem
doesn't support setStoragePolicy. As opposed to e.g. a security
manager denying access to introspect the methods available.

On Tue, Jul 21, 2020 at 8:24 AM Debraj Manna  wrote:
>
> I understood the "util.CommonFSUtils: FileSystem doesn't support
> setStoragePolicy;" part.
>
> Can you let me know why it is saying "java.lang.NoSuchMethodException:
> org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy2020-07-20
> 06:02:24,859 WARN  [StoreOpener-1588230740-1]" ?
>
>
> On Tue, Jul 21, 2020 at 6:43 PM zheng wang <18031...@qq.com> wrote:
>
> > This log info just as a warning that cant make it disappear for now, but
> > will not impact anything, so you can just ignore it in local mode.
> >
> >
> > --原始邮件--
> > 发件人:
> >   "user"
> > <
> > subharaj.ma...@gmail.com;
> > 发送时间:2020年7月21日(星期二) 晚上9:19
> > 收件人:"Hbase-User" >
> > 主题:Re: HBase 2.1.0 - NoSuchMethodException
> > org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy
> >
> >
> >
> > Thanks for replying.
> >
> > Yes it is a single node hbase cluster. I am not specifying any storage
> > policy. Looking at the HStore
> > <
> > https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java#L274
> > ;
> > code it appears even if no storage policy is specified it will take HOT.
> >
> > Can you explain this a bit more how can I get around this error or in a
> > single node hbase cluster I should be ignoring this?
> >
> >
> > On Tue, Jul 21, 2020 at 3:03 PM zheng wang <18031...@qq.com wrote:
> >
> >  LocalFileSystem? Thenbsp;setStoragePolicy could only be used in
> >  distributed hdfs.
> >  nbsp;
> > 
> > 
> >  --nbsp;原始邮件nbsp;--
> >  发件人:
> > 
> > "user"
> > 
> > <
> >  subharaj.ma...@gmail.comgt;;
> >  发送时间:nbsp;2020年7月21日(星期二) 下午5:58
> >  收件人:nbsp;"Hbase-User" > 
> >  主题:nbsp;HBase 2.1.0 - NoSuchMethodException
> >  org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy
> > 
> > 
> > 
> >  Hi
> > 
> >  I am using HBase 2.1.0 with Hadoop 3.0.0. In hbase master logs I am
> >  seeing a warning like below
> > 
> >  2020-07-20 06:02:24,859 WARNnbsp; [StoreOpener-1588230740-1]
> >  util.CommonFSUtils: FileSystem doesn't support setStoragePolicy;
> >  HDFS-6584, HDFS-9345 not available. This is normal and expected on
> >  earlier Hadoop versions.
> >  java.lang.NoSuchMethodException:
> > 
> > 
> > org.apache.hadoop.fs.LocalFileSystem.setStoragePolicy(org.apache.hadoop.fs.Path,
> >  java.lang.String)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> >  java.lang.Class.getDeclaredMethod(Class.java:2130)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:577)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:558)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:526)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:194)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > org.apache.hadoop.hbase.regionserver.HStore. > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5731)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> >  org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1059)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> >  org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1056)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> >  java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> >  java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> > 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > 
> > nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp; at
> >  java.lang.Thread.run(Thread.java:748)
> > 
> >  As per my understanding, this error should not be coming with Hadoop
> >  3.0.0. Can someone let me know if my understanding is correct or what
> >  could be going wrong here?



-- 
Sean

Re: Could not iterate StoreFileScanner - during compaction

[Sean Busbey]

File attachments won't work on the mailing list. Can you put the files on
some hosting service?

Can you reproduce the problem on hbase 2.2? HBase 2.1 has been EOM since
May.


On Fri, Jul 3, 2020, 18:20 Mohamed Meeran 
wrote:

> Hi,
>
> We are using HBase-2.1.9 (Hadoop-3.1.3) in our setup. In the logs, we see
> major compaction failed for some of the regions with the following error
> logs.
>
> Caused by: java.io.IOException: Could not iterate
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://TestCluster/hbasedata/data/Test/Test/6472f3839fc9b0a1d4b64e182043bc52/hb/2ec37243628b4a03ae3d937da4c27081,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=332,
> currentSize=485.88 MB, freeSize=333.32 MB, maxSize=819.20 MB,
> heapSize=485.88 MB, minSize=778.24 MB, minFactor=0.95, multiSize=389.12 MB,
> multiFactor=0.5, singleSize=194.56 MB, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false,
> firstKey=Optional[10259783_10101578129/hb:B/1490097103780/Put/seqid=0],
> lastKey=Optional[10260211_100965800470017/hb:H/1490097295354/Put/seqid=0],
> avgKeyLen=43, avgValueLen=213357, entries=10134, length=2163318554,
> cur=10259783_10101578851/hb:B/1490097148981/Put/vlen=16591695/seqid=0]
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:120)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:654)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6593)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6757)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6527)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3158)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3407)
> ... 5 more
> Caused by: java.io.IOException: Invalid onDisksize=-969694035: expected to
> be at least 33 and at most 2147483647, or -1
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.checkAndGetSizeAsInt(HFileBlock.java:1673)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1746)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1610)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1496)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readNextDataBlock(HFileReaderImpl.java:931)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.isNextBlock(HFileReaderImpl.java:1064)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.positionForNextBlock(HFileReaderImpl.java:1058)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl._next(HFileReaderImpl.java:1076)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.next(HFileReaderImpl.java:1097)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:208)
> ... 13 more
>
> We analysed a file using the hfile tool. Attaching the output for
> printblocks and printblockheaders.
>
> Any help to fix this would be greatly appreciated.
> --
> Thanks in advance,
>  Meeran
>

Re: [DISCUSS] Removing problematic terms from our project

[Sean Busbey]

How about "manager"?

(It would help me if folks could explain what is lacking in "coordinator".)

On Thu, Jun 25, 2020, 13:32 Nick Dimiduk  wrote:

> On Wed, Jun 24, 2020 at 10:14 PM 张铎(Duo Zhang) 
> wrote:
>
> > -0/+1/+1/+1
> >
> > I’m the one who asked whether ‘master’ is safe to use without ‘slave’ in
> > the private list.
> >
> > I’m still not convinced that it is really necessary and I do not think
> > other words like ‘coordinator’ can fully describe the role of HMaster in
> > HBase. HBase is more than 10 years old. In the context of HBase, the word
> > ‘HMaster’ has its own meaning. Changing the name will hurt our users and
> > make them confusing, especially for us non native English speakers...
> >
>
> Is there a word that's not "master" and not "coordinator" that is clear and
> suitable for (diverse, polyglot) community?
>
> Stack 于2020年6月25日 周四06:34写道：
> >
> > > +1/+1/+1/+1 where hbase3 adds the deprecation and hbase4 follows hbase3
> > > soon after sounds good to me. I'm up for working on this.
> > > S
> > >
> > > On Wed, Jun 24, 2020 at 2:26 PM Xu Cang  wrote:
> > >
> > > > Strongly agree with what Nick said here:
> > > >
> > > >  " From my perspective, we gain nothing as a project or as a
> community
> > be
> > > > willfully retaining use of language that is well understood to be
> > > > problematic or hurtful, On the contrary, we have much to gain by
> > > > encouraging
> > > > contributions from as many people as possible."
> > > >
> > > > +1 to Andrew's proposal.
> > > >
> > > > It might be good to have a source of truth web page or README file
> for
> > > > developers and users to refer to regarding all naming transitions.
> It's
> > > > going to help both developers changing the code and users looking for
> > > some
> > > > answers online that use old namings.
> > > >
> > > > Xu
> > > >
> > > > On Wed, Jun 24, 2020 at 2:21 PM Nick Dimiduk 
> > > wrote:
> > > >
> > > > > On Tue, Jun 23, 2020 at 13:11 Sean Busbey 
> wrote:
> > > > >
> > > > > > I would like to make sure I am emphatically clear that "master"
> by
> > > > itself
> > > > > > is not okay if the context is the same as what would normally be
> a
> > > > > > master/slave context. Furthermore our use of master is clearly
> > such a
> > > > > > context.
> > > > >
> > > > >
> > > > > I agree: to me “Master”, as in “HMaster” caries with it the
> > > master/slave
> > > > > baggage. As an alternative, I prefer the term “coordinator” over
> > > > “leader”.
> > > > > Thus we would have daemons called “coordinator” and “region
> server”.
> > > > >
> > > > > To me, “master” as in “master branch” does not carry the same
> > baggage,
> > > > but
> > > > > I’m also in favor changing the name of our default branch to a word
> > > that
> > > > is
> > > > > less conflicted. I see nothing that we gain as a community by
> > > continuing
> > > > to
> > > > > use this word.
> > > > >
> > > > > It seems to me we have, broadly speaking, consensus around making
> > > *some*
> > > > > > changes. I haven't seen a strong push for "break everything in
> the
> > > name
> > > > > of
> > > > > > expediency" (I would personally be fine with this). So barring
> > > > additional
> > > > > > discussion that favors breaking changes, current approaches
> should
> > > > > comport
> > > > > > with our existing project compatibility goals.
> > > > > >
> > > > > > Maybe we could stop talking about what-ifs and look at actual
> > > practical
> > > > > > examples? If anyone is currently up for doing the work of a PR we
> > can
> > > > > look
> > > > > > at for one of these?
> > > > > >
> > > > > > If folks would prefer we e.g. just say "we should break whatever
> we
> > > > need
> > > > > to
> > > > > > in 3.0.0 to make this happen" then it would be good to speak up.
> > &g

Re: [DISCUSS] Removing problematic terms from our project

[Sean Busbey]

I would like to make sure I am emphatically clear that "master" by itself
is not okay if the context is the same as what would normally be a
master/slave context. Furthermore our use of master is clearly such a
context.

It seems to me we have, broadly speaking, consensus around making *some*
changes. I haven't seen a strong push for "break everything in the name of
expediency" (I would personally be fine with this). So barring additional
discussion that favors breaking changes, current approaches should comport
with our existing project compatibility goals.

Maybe we could stop talking about what-ifs and look at actual practical
examples? If anyone is currently up for doing the work of a PR we can look
at for one of these?

If folks would prefer we e.g. just say "we should break whatever we need to
in 3.0.0 to make this happen" then it would be good to speak up. Otherwise
likely we would be done with needed changes circa hbase 4, probably late
2021 or 2022.


On Tue, Jun 23, 2020, 03:03 zheng wang <18031...@qq.com> wrote:

> IMO, master is ok if not used with slave together.
>
>
> -1/+1/+1/+1
>
>
> --原始邮件--
> 发件人:"Andrew Purtell" 发送时间:2020年6月23日(星期二) 凌晨5:24
> 收件人:"Hbase-User" 抄送:"dev" 主题:Re: [DISCUSS] Removing problematic terms from our project
>
>
>
> In observing something like voting happening on this thread to express
> alignment or not, it might be helpful to first, come up with a list of
> terms to change (if any), and then propose replacements, individually. So
> far we might break this apart into four proposals:
>
> 1. Replace "master"/"hmaster" with ??? ("coordinator" is one option), this
> one has by far the most significant impact and both opinion and
> interpretation on this one is mixed.
>
> 2. Replace "slave" with "follower", seems to impact the cross cluster
> replication subsystem only.
>
> 3. Replace "black list" with "deny list".
>
> 4. Replace "white list" with "accept list".
>
> Perhaps if you are inclined to respond with a +1/-1/+0/-0, it would be
> useful to give such an indication for each line item above. Or, offer
> alternative proposals. Or, if you have a singular opinion, that's fine too.
>
>
>
> On Mon, Jun 22, 2020 at 2:09 PM Geoffrey Jacoby  wrote:
>
>  For most of the proposals (slave - worker, blacklist -
> denylist,
>  whitelist- allowlist), I'm +1 (nonbinding). Denylist and
> acceptlist even
>  have the advantage of being clearer than the terms they're replacing.
> 
>  However, I'm not convinced about changing "master" to "coordinator",
> or
>  something similar. Unlike "slave", which is negative in any context,
>  "master" has many definitions, including some common ones which do not
>  appear problematic. See
> https://www.merriam-webster.com/dictionary/master
>  <https://www.merriam-webster.com/dictionary/master>; for
>  examples. In particular, the progression of an artisan was from
>  "apprentice" to "journeyman" to "master". A master smith, carpenter,
> or
>  artist would run a shop managing lots of workers and apprentices who
> would
>  hope to become masters of their own someday. So "master" and "worker"
> can
>  still go together.
> 
>  Since it's the least problematic term, and by far the hardest term to
>  change (both within HBase and with effects on downstream projects
> such as
>  Ambari), I'm -0 (nonbinding) on changing "master".
> 
>  Geoffrey
> 
>  On Mon, Jun 22, 2020 at 1:32 PM Rushabh Shah
>   
>   +1 to renaming.
>  
>  
>   Rushabh Shah
>  
>   - Software Engineering SMTS | Salesforce
>   -
>   - Mobile: 213 422 9052
>  
>  
>  
>   On Mon, Jun 22, 2020 at 1:18 PM Josh Elser  wrote:
>  
>+1
>   
>On 6/22/20 4:03 PM, Sean Busbey wrote:
> We should change our use of these terms. We can be
> equally or more
>   clear
>in
> what we are trying to convey where they are present.
>
> That they have been used historically is only useful
> if the advantage
>   we
> gain from using them through that shared context
> outweighs the
>   potential
> friction they add. They make me personally less
> enthusiastic about
> contributing. That's enough friction for me to
> advocate removing
>  them.
>
> AFAICT reworking our replication stuff in terms of
> "active" and
>   "passive"
> clusters did not result in a big spike of folks asking
> new questions
>about
> where authority f

Re: [DISCUSS] Removing problematic terms from our project

[Sean Busbey]

We should change our use of these terms. We can be equally or more clear in
what we are trying to convey where they are present.

That they have been used historically is only useful if the advantage we
gain from using them through that shared context outweighs the potential
friction they add. They make me personally less enthusiastic about
contributing. That's enough friction for me to advocate removing them.

AFAICT reworking our replication stuff in terms of "active" and "passive"
clusters did not result in a big spike of folks asking new questions about
where authority for state was.

On Mon, Jun 22, 2020, 13:39 Andrew Purtell  wrote:

> In response to renewed attention at the Foundation toward addressing
> culturally problematic language and terms often used in technical
> documentation and discussion, several projects have begun discussions, or
> made proposals, or started work along these lines.
>
> The HBase PMC began its own discussion on private@ on June 9, 2020 with an
> observation of this activity and this suggestion:
>
> There is a renewed push back against classic technology industry terms that
> have negative modern connotations.
>
> In the case of HBase, the following substitutions might be proposed:
>
> - Coordinator instead of master
>
> - Worker instead of slave
>
> Recommendations for these additional substitutions also come up in this
> type of discussion:
>
> - Accept list instead of white list
>
> - Deny list instead of black list
>
> Unfortunately we have Master all over our code base, baked into various
> APIs and configuration variable names, so for us the necessary changes
> amount to a new major release and deprecation cycle. It could well be worth
> it in the long run. We exist only as long as we draw a willing and
> sufficient contributor community. It also wouldn’t be great to have an
> activist fork appear somewhere, even if unlikely to be successful.
>
> Relevant JIRAs are:
>
>- HBASE-12677 :
>Update replication docs to clarify terminology
>- HBASE-13852 :
>Replace master-slave terminology in book, site, and javadoc with a more
>modern vocabulary
>- HBASE-24576 :
>Changing "whitelist" and "blacklist" in our docs and project
>
> In response to this proposal, a member of the PMC asked if the term
> 'master' used by itself would be fine, because we only have use of 'slave'
> in replication documentation and that is easily addressed. In response to
> this question, others on the PMC suggested that even if only 'master' is
> used, in this context it is still a problem.
>
> For folks who are surprised or lacking context on the details of this
> discussion, one PMC member offered a link to this draft RFC as background:
> https://tools.ietf.org/id/draft-knodel-terminology-00.html
>
> There was general support for removing the term "master" / "hmaster" from
> our code base and using the terms "coordinator" or "leader" instead. In the
> context of replication, "worker" makes less sense and perhaps "destination"
> or "follower" would be more appropriate terms.
>
> One PMC member's thoughts on language and non-native English speakers is
> worth including in its entirety:
>
> While words like blacklist/whitelist/slave clearly have those negative
> references, word master might not have the same impact for non native
> English speakers like myself where the literal translation to my mother
> tongue does not have this same bad connotation. Replacing all references
> for word *master *on our docs/codebase is a huge effort, I guess such a
> decision would be more suitable for native English speakers folks, and
> maybe we should consider the opinion of contributors from that ethinic
> minority as well?
>
> These are good questions for public discussion.
>
> We have a consensus in the PMC, at this time, that is supportive of making
> the above discussed terminology changes. However, we also have concerns
> about what it would take to accomplish meaningful changes. Several on the
> PMC offered support in the form of cycles to review pull requests and
> patches, and two PMC members offered  personal bandwidth for creating and
> releasing new code lines as needed to complete a deprecation cycle.
>
> Unfortunately, the terms "master" and "hmaster" appear throughout our code
> base in class names, user facing API subject to our project compatibility
> guidelines, and configuration variable names, which are also implicated by
> compatibility guidelines given the impact of changes to operators and
> operations. The changes being discussed are not backwards compatible
> changes and cannot be executed with swiftness while simultaneously
> preserving compatibility. There must be a deprecation cycle. First, we must
> tag all implicated public API and configuration variables as deprecated,
> and release HBase 3 with these 

Re: On org.apache.hadoop.hbase.constraint

[Sean Busbey]

I think it being labeled IA.Private is incorrect.

The red guide talks about the feature and directly points folks to the
javadoc of one of the IA.Private classes.

http://hbase.apache.org/book.html#_constraints

I'm all for us figuring out what the public surface should be and
correcting this gap, but we need to be mindful as though it were more
public than that annotation claims.

On Sun, Jun 7, 2020, 19:30 Stack  wrote:

> If IA.Private, it was for internal use only? Doesn't need a deprecation to
> change it?
>
> Please speak up if you are using the Constraint feature!
>
> Thanks,
> S
>
> On Sat, Jun 6, 2020 at 12:40 AM 张铎(Duo Zhang) 
> wrote:
>
> > The related classes are marked as IA.Private which means it is not part
> of
> > our public API...
> >
> > That's why I check for shell support, as if there is no shell support,
> then
> > users have no way to make use of it without breaking the
> InterfaceAudience
> > rule...
> >
> > Jesse Yates  于2020年6月6日周六 上午1:04写道：
> >
> > > Not particularly. Just because there is no shell integration though,
> > > doesn't mean it isn't used -  it has been around for 5 years, which
> means
> > > someone likely has picked it up. You should probably ask on the user
> list
> > > and/or do a deprecation cycle to before just removing.
> > > ---
> > > Jesse Yates
> > > @jesse_yates
> > > jesseyates.com 
> > >
> > >
> > > On Fri, Jun 5, 2020 at 8:50 AM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > Seems only this issue has been finished.
> > > >
> > > > https://issues.apache.org/jira/browse/HBASE-4605
> > > >
> > > > Which brought in these classes, but the later approach on adding
> shell
> > > > support had been resolved as incomplete.
> > > >
> > > > https://issues.apache.org/jira/browse/HBASE-4879
> > > >
> > > > So I guess there is no actual use in HBase yet.
> > > >
> > > > Do you still want to finish this feature?
> > > >
> > > > Thanks.
> > > >
> > > > Jesse Yates  于2020年6月5日周五 下午11:29写道：
> > > >
> > > > > Here is the original JIRA for the constraint work:
> > > > > https://issues.apache.org/jira/browse/HBASE-4999
> > > > >
> > > > > Its a common DB feature, but not sure if folks actually use it in
> > > HBase.
> > > > > ---
> > > > > Jesse Yates
> > > > > @jesse_yates
> > > > > jesseyates.com 
> > > > >
> > > > >
> > > > > On Fri, Jun 5, 2020 at 4:06 AM 张铎(Duo Zhang) <
> palomino...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > When removing HTableDescriptor on master branch, I found that it
> > has
> > > > been
> > > > > > referenced by org.apache.hadoop.hbase.constraint package.
> > > > > >
> > > > > > The problem here is that, the API design is to pass in an
> > > > > HTableDescriptor
> > > > > > and modify it directly, but now, TableDescriptor is immutable, so
> > we
> > > > need
> > > > > > to redesign the API.
> > > > > >
> > > > > > But the problem is that, all the classes are marked as
> IA.Private,
> > > and
> > > > > the
> > > > > > only references to these classes are in the test code. And I can
> > not
> > > > find
> > > > > > any useful information through the git log, the earliest one is
> > > > > >
> > > > > > commit 390f32d79fd0c0464fcab008150ad182f4c0abef
> > > > > > Author: Michael Stack 
> > > > > > Date:   Sat May 26 05:56:04 2012 +
> > > > > >
> > > > > > HBASE-4336 Convert source tree into maven modules
> > > > > >
> > > > > > git-svn-id:
> > https://svn.apache.org/repos/asf/hbase/trunk@1342856
> > > > > > 13f79535-47bb-0310-9956-ffa450edef68
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/hbase/trunk@134285613f79535-47bb-0310-9956-ffa450edef68
> > > > > >
> > > > > >
> > > > > > Does anyone still use this feature? Or does anyone have some
> > > background
> > > > > on
> > > > > > how this feature works?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Please welcome Lijin Bin to the HBase PMC

[Sean Busbey]

Congratulations! Thanks for taking on the additional responsibilities of
being on the PMC.

On Mon, May 25, 2020, 09:40 Guanghao Zhang  wrote:

> On behalf of the Apache HBase PMC I am pleased to announce that Lijin Bin
> has accepted our invitation to become a PMC member on the Apache HBase
> project. We appreciate Lijin Bin stepping up to take more responsibility in
> the HBase project.
>
> Please join me in welcoming Lijin Bin to the HBase PMC!
>

[ANNOUNCE] New HBase committer Wei-Chiu Chuang

[Sean Busbey]

Folks,

On behalf of the Apache HBase PMC I am pleased to announce that Wei-Chiu
Chuang has accepted the PMC's invitation to become a committer on the
project.

We appreciate all of the great contributions Wei-Chiu has made to the
community thus far and we look forward to his continued involvement.

Allow me to be the first to congratulate Wei-Chiu on his new role!

thanks,
busbey

Re: DISCUSS: Move hbase-thrift and hbase-rest out of core to hbase-connectors project?

[Sean Busbey]

By "works with it" do you mean has documented steps to work with it or do
you mean that the convenience binary that ships for 2.3.0 will have the
same deployment model as prior 2.y releases where I can run those services
directly from the download?

On Fri, Apr 24, 2020, 16:34 Stack  wrote:

> Taking a sounding
>
> We've talked of moving the hbase-rest and hbase-thrift modules out of core
> over to hbase-connectors project [2, 3]. The connectors project [1] was
> meant for the likes of REST and thrift. I'm thinking of trying to do the
> move in the next few days BEFORE 2.3.0RC0. Any objections? I'd make a
> release from hbase-connectors as part of this effort and would make sure it
> works w/ 2.3.0.
>
> Thank you,
> S
>
>
> 1. https://github.com/apache/hbase-connectors
> 2. https://issues.apache.org/jira/browse/HBASE-20999
> 3. https://issues.apache.org/jira/browse/HBASE-20998
>

Re: Issue in running hbase-2.2.3 in cygwin

[Sean Busbey]

Please be aware that running in Windows is not well tested by the rest of
the community as far as I know.

That error looks like hbase couldn't talk to the zookeeper quorum.

Are you following a specific set of instructions for getting things
started? It'll be easier to work from a common base rather then go through
all the possible configuration issues that could cause an issue.

On Sun, Mar 1, 2020, 00:32 PRAKASH GOPALSAMY 
wrote:

>   Hi Team,
> I am trying to run the hbase-2.2.3 in cygwint in windows. While running the
> hbase shell, I am getting the below exception. Kindly help me to solve this
> issue
>
> Mar 01, 2020 11:46:01 AM
> org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1 exec
> WARNING: 0x434c186b to localhost:2181 failed for get of /hbase/hbaseid,
> code = CONNECTIONLOSS, retries = 14
> Mar 01, 2020 11:46:02 AM org.apache.zookeeper.ClientCnxn$SendThread
> logStartConnect
> INFO: Opening socket connection to server kubernetes.docker.internal/
> 127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
> error)
> Mar 01, 2020 11:46:03 AM org.apache.zookeeper.ClientCnxn$SendThread run
> WARNING: Session 0x0 for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connection refused: no further information
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
>

Re: Spark reading from Hbase throws java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods

[Sean Busbey]

Hi Mich!

Please try to keep your thread on a single mailing list. It's much easier
to have things show up on a new list if you give a brief summary of the
discussion and a pointer to the original thread (lists.apache.org is great
for this).

It looks like you're using "SHC" aka the "Spark HBase Connector". This is a
toolset from a third-party and isn't associated with either the Apache
Spark or Apache HBase communities. You should address your concerns to the
provider of said tool.

If you are interested in reading/writing with HBase from Spark jobs, the
Apache HBase community provides its own integration through our "Apache
HBase Connectors" project.

The project's reference guide includes some examples of using the
integration:

http://hbase.apache.org/book.html#spark

And the bits are available from our download page:

http://hbase.apache.org/downloads.html

The current documentation for deployment is thin, but if can bring specific
questions to the user@hbase mailing list about it that'll help get push
along improving that.


On Sun, Feb 23, 2020, 13:13 Mich Talebzadeh 
wrote:

> Hi,
>
> Does anyone has any more suggestion for the error I reported below please?
>
> Thanks,
>
> Mich
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 17 Feb 2020 at 22:27, Mich Talebzadeh 
> wrote:
>
> > I stripped everything from the jar list. This is all I have
> >
> > sspark-shell --jars shc-core-1.1.1-2.1-s_2.11.jar, \
> >   json4s-native_2.11-3.5.3.jar, \
> >   json4s-jackson_2.11-3.5.3.jar, \
> >   hbase-client-1.2.3.jar, \
> >   hbase-common-1.2.3.jar
> >
> > Now I still get the same error!
> >
> > scala> val df = withCatalog(catalog)
> > java.lang.NoSuchMethodError:
> >
> org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue;
> >   at
> >
> org.apache.spark.sql.execution.datasources.hbase.HBaseTableCatalog$.apply(HBaseTableCatalog.scala:257)
> >   at
> >
> org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.(HBaseRelation.scala:80)
> >   at
> >
> org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:51)
> >   at
> >
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
> >   at
> >
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
> >   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
> >   at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
> >   at withCatalog(:54)
> >
> > Thanks
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
> >
> >
> >
> > On Mon, 17 Feb 2020 at 21:37, Mich Talebzadeh  >
> > wrote:
> >
> >>
> >> Dr Mich Talebzadeh
> >>
> >>
> >>
> >> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> <
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >*
> >>
> >>
> >>
> >> http://talebzadehmich.wordpress.com
> >>
> >>
> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >> any loss, damage or destruction of data or any other property which may
> >> arise from relying on this email's technical content is explicitly
> >> disclaimed. The author will in no case be liable for any monetary
> damages
> >> arising from such loss, damage or destruction.
> >>
> >>
> >> Many thanks both.
> >>
> >> Let me check and confirm.
> >>
> >> regards,
> >>
> >> Mich
> >>
> >>
> >> On Mon, 17 Feb 2020 at 21:33, Jörn Franke  wrote:
> >>
> >>> Is there a reason why different Scala (it seems at least 2.10/2.11)
> >>> versions are mixed? This never works.
> >>> Do you include by accident a dependency to with an old Scala version?
> Ie
> >>> the Hbase datasource maybe?
> >>>
> >>>
> >>> Am 17.02.2020 um 22:15 schrieb Mich Talebzadeh <
> >>> mich.talebza...@gmail.com>:
> >>>
> >>> 
> >>> Thanks Muthu,
> >>>
> >>>
> >>> I am using the following jar files for now in local mode i.e.
> spark-shell_local
> >>> --jars …..
> >>>
> >>> json4s-jackson_2.10-3.2.10.jar
> >>> json4s_2.11-3.2.11.jar
> >>> json4s-native_2.10-3.4.0.jar
> >>>
> >>> Which one is the incorrect one please/
> >>>
> >>> Regards,
> >>>
> >>> Mich
> >>>
> >>>
> >>>
> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >>> any loss, damage or destruction of data or any other 

Re: [ANNOUNCE] New HBase committer Bharath Vissapragada

[Sean Busbey]

Congrats Bharath!

On Wed, Feb 5, 2020, 21:36 Nick Dimiduk  wrote:

> On behalf of the Apache HBase PMC I am pleased to announce that Bharath
> Vissapragada has accepted the PMC's invitation to become a commiter on the
> project. We appreciate all of Bharath's generous contributions thus far and
> look forward to his continued involvement.
>
> Allow me to be the first to congratulate and welcome Bharath into his new
> role!
>
> Thanks,
> Nick
>

[ANNOUNCE] Apache HBase "stable" release line is now 2.2.3+

[Sean Busbey]

Hi Community!

The Apache HBase developers would like you to know that we have updated the
"stable" release pointer to the 2.2.3 release. As future 2.2.z releases are
made the stable pointer will track them. All users of releases prior to
2.2.3 are advised to upgrade to the new stable release line.

The prior stable release line was 1.4.z. Since this change marks a move
across a major HBase version, users still on HBase 1.y can use the new
"stable-1" release pointer to track the community's best guess on a
non-disruptive upgrade choice that is likely to receive updated releases.
Approximately monthly releases are expected for the "stable-1" pointer for
the next 6 months. Be advised that it is likely the "stable-1" pointer will
track new minor releases of HBase 1 rather than maintenance releases of
HBase 1.4.

If you are currently a user of a release line older than 2.2 and there's
anything that would make your transition to the stable release line easier,
please let us know either via the mailing list or JIRAs.

Re: replication fix and hbase 1.4.13?

[Sean Busbey]

As a side note, the current "stable" pointer is the 2.2.3 release as
of like a week ago.

http://www.apache.org/dist/hbase/stable/

it's noted on the download page:

http://hbase.apache.org/downloads.html

it looks like the text at the top of the asf mirroring page hasn't
been updated and I don't see an announcement to user@. I'll try to got
both of those oversights cleared up.

On Mon, Jan 27, 2020 at 11:19 AM Whitney Jackson  wrote:
>
> Thanks for the reply! Very helpful.
>
> > One more thing I'd ask is that you remember that we're all volunteers
> here.
>
> Absolutely. Appreciate the excellent work! I can see how the "stable"
> question is less cut and dried than it might seem from the outside. And
> your answer on that is definitely useful in thinking through which version
> to run.
>
> Whitney
>
>
> On Mon, Jan 27, 2020 at 9:18 AM Josh Elser  wrote:
>
> > That fix is already slated for 1.4.13. There would be an exceptional
> > issue for it to not be included at this point. The "-1" you saw is just
> > an automated, nightly test which are not always perfect.
> >
> > The general cadence for releases is about 1 month. It looks like the
> > Christmas season got in the way and we're due for 1.4.13 soon. You can
> > watch the chatter on the dev mailing list to stay in the loop.
> >
> > Your question about "stable" is not so easy to answer. It's a label
> > assigned by the HBase developers to try to give a simple answer to a
> > very complex question. Developers decide on what we feel is acceptable
> > to call "stable", and will move it when we think it's
> > appropriate/necessary. Our goal is to try to make the best experience
> > for our users to save you all pain/heartache/grief.
> >
> > One more thing I'd ask is that you remember that we're all volunteers
> > here. Day jobs, families, and all sorts of other things can get in the
> > way. This doesn't mean that something isn't, persay, urgent or critical
> > to "do" (fix, investigate, release), but just that we don't have enough
> > bandwidth to do everything we're asked to do.
> >
> > Specifically around releases, we're always looking for more people to
> > help drive the release process. Those who can corral Jira issues, do
> > testing, and stage release candidates are very welcome and desired to
> > help make our releases happen on a regular cadence. If you have the
> > time/resources to help our, let us know on the dev list.
> >
> > - Josh
> >
> > On 1/27/20 11:04 AM, Whitney Jackson wrote:
> > > Hi,
> > >
> > > I've been running 1.4.12 with replication and experiencing the "random
> > > region server aborts" as described here:
> > >
> > > https://issues.apache.org/jira/browse/HBASE-23169
> > >
> > > The underlying problem and fix (woohoo!) seems to be here:
> > >
> > > https://issues.apache.org/jira/browse/HBASE-23205
> > >
> > > I do see there is a failed jdk7 check noted at the end of 23205. Will
> > that
> > > prevent the fix from getting into 1.4.13? Also, when is 1.4.13 likely to
> > > drop?
> > >
> > > On a related note, how does the "stable" label work? I'm running 1.4.12
> > in
> > > large part because it has that label. But as I discovered it also has
> > this
> > > known bug with a core feature (replication). It seems serious to me but
> > the
> > > urgency on fixing it seems to be low. That surprises me given the
> > > importance of the bug and the "stable" label. Would I be better off
> > keeping
> > > up with the latest and greatest 2.x if my biggest concern is stability?
> > >
> > > Whitney
> > >
> >

Re: [DISCUSS] Bump hadoop versions

[Sean Busbey]

That's me. I'll update with a plan later in the week.

On Sat, Jan 4, 2020, 09:02 张铎(Duo Zhang)  wrote:

> Anyway if we want to start releasing 3.x alpha versions, at least we need
> to have a release manager... Dang
>
> Andrew Purtell  于2020年1月4日周六 上午11:38写道：
>
> > Releasing HBase 3 snapshots off master would be fine by me.
> >
> > > On Jan 3, 2020, at 7:37 PM, Sean Busbey  wrote:
> > >
> > > A while back I offered to start making hbase 3 alpha releases so we
> had
> > > something we could all refer to in test environments. That offer still
> > > stands.
> > >
> > >
> > > I personally would still rather those releases be off of the master
> > branch
> > > because we still have too many branches given our tooling for
> backports.
> > >
> > >> On Fri, Jan 3, 2020, 21:30 Andrew Purtell 
> > wrote:
> > >>
> > >> Why not start a branch-3 and begin SNAPSHOT releasing of this branch
> > right
> > >> now?
> > >>
> > >> +1 to dropping Hadoop 2 support in HBase 3. We need the major
> increment
> > to
> > >> make this kind of change, let’s take the opportunity.
> > >>
> > >> Regarding Hadoop 2, the discussion I have seen indicates Hadoop thinks
> > it
> > >> will be releasing 2.x for up to two more years. I don’t know how many
> > >> releases there will actually be but let’s assume at least one more
> 2.9,
> > and
> > >> a few 2.10. My employer is expected to use these versions along a
> > >> transition path to 3.x for at least the next eighteen months. We are
> > >> probably typical. We won’t need Hadoop 2 support for a HBase 3 but
> will
> > >> need it for HBase 1 and HBase 2 for “a couple years”.
> > >>
> > >>>> On Jan 3, 2020, at 6:52 PM, Sean Busbey  wrote:
> > >>>
> > >>> I personally like having Hadoop 2 support still, but I agree the
> > cadence
> > >>> out of Hadoop has been problematic.
> > >>>
> > >>> I would prefer we not change the state of Hadoop support in the
> master
> > >>> branch until we have a release plan of some kind for HBase 3. I'd
> > rather
> > >>> that be sooner rather than later.
> > >>>
> > >>>> On Fri, Jan 3, 2020, 18:08 张铎(Duo Zhang) 
> > wrote:
> > >>>>
> > >>>> Support Hadoop 2.x in 3.0.0 means we need to carry it over the whole
> > 3.x
> > >>>> release lines, which seems to be a problem since the Hadoop
> community
> > do
> > >>>> not want to new 2.x release line any more...
> > >>>>
> > >>>> Nick Dimiduk 于2020年1月4日 周六06:55写道：
> > >>>>
> > >>>>> On Wed, Dec 25, 2019 at 5:38 PM 张铎(Duo Zhang) <
> palomino...@gmail.com
> > >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> We will only remove the hadoop 2.x support from hbase 3.x, which
> > does
> > >>>> not
> > >>>>>> have a formal release plan yet, for 2.x we will still support
> hadoop
> > >>>> 2.x.
> > >>>>>>
> > >>>>>
> > >>>>> Indeed there is no formal release plan for HBase-3.0, but I hope
> it's
> > >>>>> sooner than 2+ years away! What's the motivation for dropping
> Hadoop2
> > >>>>> support?
> > >>>>>
> > >>>>> Wei-Chiu Chuang  于2019年12月26日周四 上午8:57写道：
> > >>>>>>
> > >>>>>>> With my Hadoop hat's on, we have not yet officially declared
> Hadoop
> > >>>> 2.8
> > >>>>>>> EOL. I think the 2.8 download missing from the web page is just a
> > >>>>>> mistake.
> > >>>>>>>
> > >>>>>>> That being said, some of the biggest Hadoop users (LinkedIn,
> Yahoo,
> > >>>>>>> Microsoft) that I am aware of are moving up from 2.7/2.8 to 2.10,
> > and
> > >>>>>> that
> > >>>>>>> 2.8.5 (the last version in the 2.8 line) was released in Sep
> 2018,
> > >>>> more
> > >>>>>>> than a year ago. It doesn't look like the community has the
> desire
> > to
> > >>>>>>> continue the 2.8 line.
> > >>>>>>>
> > >>>>>>> I think it is a little extreme to remove hadoop2 profile, given
> > that
> > >>>>>> Hadoop
> > >>>>>>> 2.9 and 2.10 are still active and I expect Hadoop 2 to stay
> around
> > >>>> for
> > >>>>> at
> > >>>>>>> least 2 years out.
> > >>>>>>>
> > >>>>>>> On Thu, Dec 26, 2019 at 8:41 AM 张铎(Duo Zhang) <
> > palomino...@gmail.com
> > >>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hadoop 2.8.x has been removed from the download page of hadoop
> so
> > I
> > >>>>>> think
> > >>>>>>>> it is time to bump the hadoop dependency to 2.9.x, on master an
> > >>>>>> branch-2.
> > >>>>>>>>
> > >>>>>>>> And the hadoop community is going to make 2.10.x the last minor
> > >>>>> release
> > >>>>>>>> line for 2.x
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://lists.apache.org/thread.html/cab84265d632b90d66dcd1ad957a7439a2c76a987c7e62feafb4812e%40%3Ccommon-dev.hadoop.apache.org%3E
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> I think this is a sign that the community is moving forward to
> > 3.x.
> > >>>>> So
> > >>>>>> I
> > >>>>>>>> propose we make the master branch hadoop3 only, This requires
> > >>>>> changing
> > >>>>>>> the
> > >>>>>>>> pom a bit to active hadoop3 profile by default and remove the
> > >>>> hadoop2
> > >>>>>>>> profile.
> > >>>>>>>>
> > >>>>>>>> Thoughts? Thanks.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
>

Re: [DISCUSS] Bump hadoop versions

[Sean Busbey]

A while back I offered to start making hbase 3 alpha releases so we had
something we could all refer to in test environments. That offer still
stands.


I personally would still rather those releases be off of the master branch
because we still have too many branches given our tooling for backports.

On Fri, Jan 3, 2020, 21:30 Andrew Purtell  wrote:

> Why not start a branch-3 and begin SNAPSHOT releasing of this branch right
> now?
>
> +1 to dropping Hadoop 2 support in HBase 3. We need the major increment to
> make this kind of change, let’s take the opportunity.
>
> Regarding Hadoop 2, the discussion I have seen indicates Hadoop thinks it
> will be releasing 2.x for up to two more years. I don’t know how many
> releases there will actually be but let’s assume at least one more 2.9, and
> a few 2.10. My employer is expected to use these versions along a
> transition path to 3.x for at least the next eighteen months. We are
> probably typical. We won’t need Hadoop 2 support for a HBase 3 but will
> need it for HBase 1 and HBase 2 for “a couple years”.
>
> > On Jan 3, 2020, at 6:52 PM, Sean Busbey  wrote:
> >
> > I personally like having Hadoop 2 support still, but I agree the cadence
> > out of Hadoop has been problematic.
> >
> > I would prefer we not change the state of Hadoop support in the master
> > branch until we have a release plan of some kind for HBase 3. I'd rather
> > that be sooner rather than later.
> >
> >> On Fri, Jan 3, 2020, 18:08 张铎(Duo Zhang)  wrote:
> >>
> >> Support Hadoop 2.x in 3.0.0 means we need to carry it over the whole 3.x
> >> release lines, which seems to be a problem since the Hadoop community do
> >> not want to new 2.x release line any more...
> >>
> >> Nick Dimiduk 于2020年1月4日 周六06:55写道：
> >>
> >>> On Wed, Dec 25, 2019 at 5:38 PM 张铎(Duo Zhang) 
> >>> wrote:
> >>>
> >>>> We will only remove the hadoop 2.x support from hbase 3.x, which does
> >> not
> >>>> have a formal release plan yet, for 2.x we will still support hadoop
> >> 2.x.
> >>>>
> >>>
> >>> Indeed there is no formal release plan for HBase-3.0, but I hope it's
> >>> sooner than 2+ years away! What's the motivation for dropping Hadoop2
> >>> support?
> >>>
> >>> Wei-Chiu Chuang  于2019年12月26日周四 上午8:57写道：
> >>>>
> >>>>> With my Hadoop hat's on, we have not yet officially declared Hadoop
> >> 2.8
> >>>>> EOL. I think the 2.8 download missing from the web page is just a
> >>>> mistake.
> >>>>>
> >>>>> That being said, some of the biggest Hadoop users (LinkedIn, Yahoo,
> >>>>> Microsoft) that I am aware of are moving up from 2.7/2.8 to 2.10, and
> >>>> that
> >>>>> 2.8.5 (the last version in the 2.8 line) was released in Sep 2018,
> >> more
> >>>>> than a year ago. It doesn't look like the community has the desire to
> >>>>> continue the 2.8 line.
> >>>>>
> >>>>> I think it is a little extreme to remove hadoop2 profile, given that
> >>>> Hadoop
> >>>>> 2.9 and 2.10 are still active and I expect Hadoop 2 to stay around
> >> for
> >>> at
> >>>>> least 2 years out.
> >>>>>
> >>>>> On Thu, Dec 26, 2019 at 8:41 AM 张铎(Duo Zhang)  >>>
> >>>>> wrote:
> >>>>>
> >>>>>> Hadoop 2.8.x has been removed from the download page of hadoop so I
> >>>> think
> >>>>>> it is time to bump the hadoop dependency to 2.9.x, on master an
> >>>> branch-2.
> >>>>>>
> >>>>>> And the hadoop community is going to make 2.10.x the last minor
> >>> release
> >>>>>> line for 2.x
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://lists.apache.org/thread.html/cab84265d632b90d66dcd1ad957a7439a2c76a987c7e62feafb4812e%40%3Ccommon-dev.hadoop.apache.org%3E
> >>>>>>
> >>>>>>
> >>>>>> I think this is a sign that the community is moving forward to 3.x.
> >>> So
> >>>> I
> >>>>>> propose we make the master branch hadoop3 only, This requires
> >>> changing
> >>>>> the
> >>>>>> pom a bit to active hadoop3 profile by default and remove the
> >> hadoop2
> >>>>>> profile.
> >>>>>>
> >>>>>> Thoughts? Thanks.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>

Re: [DISCUSS] Bump hadoop versions

[Sean Busbey]

I personally like having Hadoop 2 support still, but I agree the cadence
out of Hadoop has been problematic.

I would prefer we not change the state of Hadoop support in the master
branch until we have a release plan of some kind for HBase 3. I'd rather
that be sooner rather than later.

On Fri, Jan 3, 2020, 18:08 张铎(Duo Zhang)  wrote:

> Support Hadoop 2.x in 3.0.0 means we need to carry it over the whole 3.x
> release lines, which seems to be a problem since the Hadoop community do
> not want to new 2.x release line any more...
>
> Nick Dimiduk 于2020年1月4日 周六06:55写道：
>
> > On Wed, Dec 25, 2019 at 5:38 PM 张铎(Duo Zhang) 
> > wrote:
> >
> > > We will only remove the hadoop 2.x support from hbase 3.x, which does
> not
> > > have a formal release plan yet, for 2.x we will still support hadoop
> 2.x.
> > >
> >
> > Indeed there is no formal release plan for HBase-3.0, but I hope it's
> > sooner than 2+ years away! What's the motivation for dropping Hadoop2
> > support?
> >
> > Wei-Chiu Chuang  于2019年12月26日周四 上午8:57写道：
> > >
> > > > With my Hadoop hat's on, we have not yet officially declared Hadoop
> 2.8
> > > > EOL. I think the 2.8 download missing from the web page is just a
> > > mistake.
> > > >
> > > > That being said, some of the biggest Hadoop users (LinkedIn, Yahoo,
> > > > Microsoft) that I am aware of are moving up from 2.7/2.8 to 2.10, and
> > > that
> > > > 2.8.5 (the last version in the 2.8 line) was released in Sep 2018,
> more
> > > > than a year ago. It doesn't look like the community has the desire to
> > > > continue the 2.8 line.
> > > >
> > > > I think it is a little extreme to remove hadoop2 profile, given that
> > > Hadoop
> > > > 2.9 and 2.10 are still active and I expect Hadoop 2 to stay around
> for
> > at
> > > > least 2 years out.
> > > >
> > > > On Thu, Dec 26, 2019 at 8:41 AM 张铎(Duo Zhang)  >
> > > > wrote:
> > > >
> > > > > Hadoop 2.8.x has been removed from the download page of hadoop so I
> > > think
> > > > > it is time to bump the hadoop dependency to 2.9.x, on master an
> > > branch-2.
> > > > >
> > > > > And the hadoop community is going to make 2.10.x the last minor
> > release
> > > > > line for 2.x
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/cab84265d632b90d66dcd1ad957a7439a2c76a987c7e62feafb4812e%40%3Ccommon-dev.hadoop.apache.org%3E
> > > > >
> > > > >
> > > > > I think this is a sign that the community is moving forward to 3.x.
> > So
> > > I
> > > > > propose we make the master branch hadoop3 only, This requires
> > changing
> > > > the
> > > > > pom a bit to active hadoop3 profile by default and remove the
> hadoop2
> > > > > profile.
> > > > >
> > > > > Thoughts? Thanks.
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Please welcome Guangxu Cheng the HBase PMC

[Sean Busbey]

Congrats and thanks for all you do Guangxu!

On Mon, Dec 9, 2019, 03:47 Duo Zhang  wrote:

> On behalf of the Apache HBase PMC I am pleased to announce that Guangxu
> Cheng has accepted our invitation to become a PMC member on the Apache
> HBase project. We appreciate Guangxu Cheng stepping up to take more
> responsibility in the HBase project.
>
> Please join me in welcoming Guangxu Cheng to the HBase PMC!
>

[ANNOUNCE] Apache HBase 1.4.12 is now available for download.

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.4.12.

Download from http://hbase.apache.org/downloads

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.4.12 is a maintenance release in HBase's current stable release
line, continuing on the theme of bringing a stable, reliable database to the
Hadoop and NoSQL communities.

All users of previous releases are encouraged to upgrade to this release.

Critical changes include:

* HBASE-23227 - Upgrade jackson-databind to 2.9.10.1 to avoid recent
CVEs
* HBASE-23287 - WALs not aged off of HDFS because LogCleaner is not
added to choreService

The full list of fixes included in this release is available in the
CHANGES.txt file included in the distribution and at

https://s.apache.org/hbase-1.4.12-jira-release-notes

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS

Thanks to all the contributors who made this release possible!
Question, comments, and problems are always welcome at: d...@hbase.apache.org

Cheers,
The HBase Dev Team

Re: Why HBase Delete Reverts Back to Previous Value instead of Totally Deleting it

[Sean Busbey]

What version of hbase and how are you issuing the delete.

On Tue, Nov 26, 2019, 21:06 Trixia Belleza <
trixia.bell...@cheetahdigital.com> wrote:

> Hbase Apache,
>
> Let's say I have a column named signup_date with a value 2019-09-09. Then
> I will update this column to 2019-11-11. So signup_date is now 2019-11-11.
>
> After that, if I delete the signup_date column, I expect it to be gone in
> the Hbase BUT, it remains there and it's reverted back to 2019-09-09.
>
> Is this a bug in hbase where deleted columns are reverted back to previous
> values?

Re: HBase table snapshots compatibility between 1.0 and 2.1

[Sean Busbey]

Snapshots should work between HBase 1 and HBase 2. There are only
three gotchas I know about for releases from the project.

1) Minimum Hfile Version bump - HBase 2 can still read HFile V2 and
HFile V3 and it can only write HFile V3. It can't read HFile V1. I
have seen clusters with old snapshots that still have HFile V1 files
present.
2) Removal of PREFIX_TREE data block encoding.

AFAIK both of the above can be checked with the check added in HBASE-20649

hbase pre-upgrade validate-hfile

http://hbase.apache.org/2.1/book.html#_hfile_content_validation

3) If exporting a snapshot into the cluster takes too long, the
cleaner in the destination will destroy the hfiles. HBASE-23202
"ExportSnapshot (import) will fail if copying files to root directory
takes longer than cleaner TTL" AFAIK this is still open. and the work
around is probably disabling the cleaner while you're doing the
snapshot export.

On Tue, Nov 12, 2019 at 10:56 AM Josh Elser  wrote:
>
> Hey Shuai,
>
> You're likely to get some more traction with this question via
> contacting Cloudera's customer support channels. We try to keep this
> forum focused on Apache HBase versions.
>
> If you are not seeing records after restoring, it sounds like there is
> some (missing?) metadata in the old version which is not handled in the
> newer versions.
>
> As far as your procedurces, you could combine your
> create/disable/restore_snapshot to just use clone_snapshot instead.
> However, if there is some incompatibility, this is of little consequence.
>
> You could try to use CopyTable instead of snapshots.
>
> On 11/11/19 1:47 AM, Shuai Lin wrote:
> > Hi all,
> >
> > TL;DR Could table snapshots taken in hbase 1.0 be used in hbase 2.1?
> >
> > We have an existing production hbase 1.0 cluster (CDH 5.4) , and we're
> > setting up a new cluster with hbase 2.1 (CDH 6.3). Let's call the old
> > cluster C1 and new one C2.
> >
> > To migrate the existing data from C1 to C2, we plan to use the "snapshot +
> > replication" approach (snapshot would capture the existing part, and
> > replication would do the incremental part) . However when I was testing the
> > feasibility of this approach locally, I found that the snapshot could be
> > successfully export to c2, but but the restored table on C2 has no data.
> >
> > Here is a minimal reproducible example:
> >
> > 1. on C1: take the snapshot and export it to C2
> >
> > hbase shell:
> >  create "t1", {"NAME"=>"f1", "REPLICATION_SCOPE" => 1}
> >  put "t1", "r1", "f1:c1", "value"
> >  snapshot 't1', 't1s1'
> >
> > sudo -u hdfs hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > -snapshot t1s1 \
> >  -copy-to hdfs://c2:8020/hbase -mappers 1
> >
> > 2. Then on C2 restore the table
> >
> > hbase shell:
> >  create "t1", {"NAME"=>"f1", "REPLICATION_SCOPE" => 1}
> >  disable "t1"
> >  restore_snapshot "t1s1"
> >  enable "t1"
> >  scan "t1"
> >
> > All these steps succeeds, except that the final "scan" command shows no
> > data at all. Also worth noting that on the master web ui on C2 it shows the
> > table t1 has two regions and one is not assigned - It shall have only one
> > region obviously.
> >
> > So my question is:  Could table snapshots taken in hbase 1.0 be used in
> > hbase 2.1?
> > - If yes, anything I'm doing wrong here?
> > - If no, is there any workaround? (e.g. performing some preprocessing on
> > the snapshot data on hbase 2.1 side before restoring it?)
> >
> > If this can't work, the only alternative way to migrate the data is too
> > install hbase 1.0 on C2 (so it could use the snapshot from C1), and upgrade
> > it to hbase 2.1 after restoring the snapshot. I I'd like to avoid going
> > this way as much as possible because it would be too cumbersome.
> >
> > Any information would be much appreciated, thx!
> >

Re: facing this error while starting hbase services in ubuntu

[Sean Busbey]

What version of hbase and how are you starting it?

Are you already following our quickstart guide?

http://hbase.apache.org/book.html#quickstart

On Sat, Nov 9, 2019, 11:24 Gourav Verma 
wrote:

> facing this error while starting hbase services in ubuntu
>
> Could not find or load main class
> org.apache.hadoop.hbase.util.HBaseConfTool
>
> Please let us know the solution for this issue
>

Re: Region server web port doesn't respond

[Sean Busbey]

what's your specific JDK version. Capturing the output of "java
-version" would be the easiest way to get an answer.

On Tue, Nov 5, 2019 at 8:09 PM yuhang li <89389114...@gmail.com> wrote:
>
> My jdk version is 1.8.
>
> I checked the jetty code and found that jetty does some work for jvm bugs
> in jdk1.6 and 1.7. But what I use is jdk 1.8. This should be a bug of jetty.
>
> The workearounds jetty takes is to check the time cost of JAVA NIO
> selector's select operation, and thinks it's BUG appearing when the time
> cost is too small. see https://wiki.eclipse.org/Jetty/Feature/JVM_NIO_Bug
>
> According to jetty code, some system property can be set to control the
> behaviour of this workearound. I'v set
> "-Dorg.mortbay.io.nio.JVMBUG_THRESHHOLD=2147483647
> -Dorg.mortbay.io.nio.MONITOR_PERIOD=0", maybe it cound work.

Re: Region server web port doesn't respond

[Sean Busbey]

What's your JDK version

On Mon, Nov 4, 2019, 21:39 yuhang li <89389114...@gmail.com> wrote:

> Hi all,
>
> I'm using hbase 1.2.9.
>
> Every time after starting regionserver several days, I cannot open its web
> ui.
>
> Using nc test tcp port, it's OK. But using curl, it just show 'curl: (52)
> empty reply from server '.
>
> Some abnormal message can be found in logs(in every minute):
>
> 2019-11-04 20:01:54,099 INFO [738111983@qtp-732189840-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:26030] mortbay.log:
> org.mortbay.io.nio.SelectorManager$SelectSet@59f478d5 JVM BUG(s) -
> injecting delay58 times
> 2019-11-04 20:01:54,099 INFO [738111983@qtp-732189840-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:26030] mortbay.log:
> org.mortbay.io.nio.SelectorManager$SelectSet@59f478d5 JVM BUG(s) -
> recreating selector 58 times, canceled keys 928 times
>

[ANNOUNCE] Apache HBase 1.4.11 is now available for download.

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.4.11.

Download from http://hbase.apache.org/downloads

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.4.11 is a maintenance release in HBase's current stable release
line, continuing on the theme of bringing a stable, reliable database to the
Hadoop and NoSQL communities.

All users of previous releases are encouraged to upgrade to this release.

Critical changes include:

* HBASE-22784 Clusters in a cyclical replication topology that only have
  incoming writes from replication never clean out WALs.
* HBASE-23101 replication of bulkloaded files must handle cyclical
  topologies. Includes binary incompatible change to Region
  interface.
* HBASE-23015 Stop including Jackson libraries in classpaths except when
  needed by the HBase REST Proxy or by Apache Hadoop.
* HBASE-22728, HBASE-23174 update HBase REST Proxy to use Jackson 2 for
  serialization (CVEs).
* HBASE-22627 Recovered WAL directories not getting cleaned up when
  running different WAL and base filesystems.
* HBASE-15666 New utility artifact for testing shaded client:
  hbase-shaded-testing-util.
* HBASE-22874 Define a public interface for Canary and move existing
  implementation to LimitedPrivate

The full list of fixes included in this release is available in the
CHANGES.txt file included in the distribution and at

https://s.apache.org/hbase-1.4.11-jira-release-notes

For instructions on verifying ASF release downloads, please see
https://www.apache.org/dyn/closer.cgi#verify

Project member signature keys can be found at
https://www.apache.org/dist/hbase/KEYS

Thanks to all the contributors who made this release possible!
Question, comments, and problems are always welcome at: d...@hbase.apache.org

Cheers,
The HBase Dev Team

[ANNOUNCE] Please welcome Balazs Meszaros to the Apache HBase PMC

[Sean Busbey]

On behalf of the Apache HBase PMC I am pleased to announce that
Balazs Meszaros has accepted our invitation to become a PMC member on the
HBase project. We appreciate Balazs stepping up to take more
responsibility in the HBase project.

Please join me in welcoming Balazs to the HBase PMC!



As a reminder, if anyone would like to nominate another person as a
committer or PMC member, even if you are not currently a committer or
PMC member, you can always drop a note to priv...@hbase.apache.org to
let us know.

[ANNOUNCE] Please welcome Wellington Chevreuil to the Apache HBase PMC

[Sean Busbey]

On behalf of the Apache HBase PMC I am pleased to announce that
Wellington Chevreuil has accepted our invitation to become a PMC member on the
HBase project. We appreciate Wellington stepping up to take more
responsibility in the HBase project.

Please join me in welcoming Wellington to the HBase PMC!



As a reminder, if anyone would like to nominate another person as a
committer or PMC member, even if you are not currently a committer or
PMC member, you can always drop a note to priv...@hbase.apache.org to
let us know.

Re: [ANNOUNCE] Please welcome Sakthi to the Apache HBase PMC

[Sean Busbey]

Please join me in welcoming Sakthi to the HBase PMC, of course.

Welcome Sakthi! Well deserved!

On Wed, Oct 23, 2019 at 3:14 PM Sean Busbey  wrote:
>
> On behalf of the Apache HBase PMC I am pleased to announce that
> Sakthi has accepted our invitation to become a PMC member on the
> HBase project. We appreciate Sakthi stepping up to take more
> responsibility in the HBase project.
>
> Please join me in welcoming Jan to the HBase PMC!
>
>
>
> As a reminder, if anyone would like to nominate another person as a
> committer or PMC member, even if you are not currently a committer or
> PMC member, you can always drop a note to priv...@hbase.apache.org to
> let us know.

[ANNOUNCE] Please welcome Sakthi to the Apache HBase PMC

[Sean Busbey]

On behalf of the Apache HBase PMC I am pleased to announce that
Sakthi has accepted our invitation to become a PMC member on the
HBase project. We appreciate Sakthi stepping up to take more
responsibility in the HBase project.

Please join me in welcoming Jan to the HBase PMC!



As a reminder, if anyone would like to nominate another person as a
committer or PMC member, even if you are not currently a committer or
PMC member, you can always drop a note to priv...@hbase.apache.org to
let us know.

Re: Failure to compile package hbase-operator-tools

[Sean Busbey]

For help with vendor specific builds of HBase you should reach out to that
vendor.

Did you already try passing the "bypass version check" flag to the
invocation of the pre-built HBCK2?

On Tue, Oct 15, 2019 at 7:28 AM 青椒肉丝 <1336318...@qq.com> wrote:

> hi ,busbey
>
>The reason I want to compile the HBase operator tools package myself is
> that I can't execute HBCK commands using the existing jar package,I
> downloaded the executable bin package from the following address，But when I
> execute the bypass command, I get an error as follows
> commond : hbase hbck -j
> /packages/hbase-operator-tools-1.0.0/hbase-hbck2/hbase-hbck2-1.0.0.jar
> bypass -o 993
> “
> Exception in thread "main" java.lang.UnsupportedOperationException: bypass
> not supported on server version=2.1.0-cdh6.1.0; needs at least a server
> that matches or exceeds [2.0.3, 2.1.1, 2.2.0, 3.0.0]
> at org.apache.hbase.HBCK2.checkHBCKSupport(HBCK2.java:134)
> at org.apache.hbase.HBCK2.bypass(HBCK2.java:335)
> at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:686)
> at org.apache.hbase.HBCK2.run(HBCK2.java:631)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hbase.HBCK2.main(HBCK2.java:865)
> ”
>
> So I want to compile the jar package that HBase 2.1.0 can use. But I
> download the SRC package from the above image address and compile it. The
> following errors will still be reported. Can HBase 2.1.0-cdh 6.1.0 not use
> HBCK2?
> “
> [INFO] Apache HBase Operator Tools  SUCCESS [
> 1.584 s]
> [INFO] Apache HBase - HBCK2 ... FAILURE [
> 23.380 s]
> [INFO] Apache HBase Operator Tools - Assembly . SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time:  25.196 s
> [INFO] Finished at: 2019-10-15T19:49:33+08:00
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hbase-hbck2: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools-1.0.0/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[49,38]
> cannot find symbol
> [ERROR]   symbol:   class Hbck
> [ERROR]   location: package org.apache.hadoop.hbase.client
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools-1.0.0/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[139,28]
> cannot find symbol
> [ERROR]   symbol:   class Hbck
> [ERROR]   location: class org.apache.hbase.HBCK2
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools-1.0.0/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[266,22]
> cannot find symbol
> [ERROR]   symbol:   class Hbck
> [ERROR]   location: class org.apache.hbase.HBCK2
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools-1.0.0/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[283,24]
> cannot find symbol
> [ERROR]   symbol:   class Hbck
> [ERROR]   location: class org.apache.hbase.HBCK2
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools-1.0.0/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[340,33]
> cannot find symbol
> [ERROR]   symbol:   class Hbck
> [ERROR]   location: class org.apache.hbase.HBCK2
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :hbase-hbck2
> ”
> -- 原始邮件 --
> *发件人:* "busbey";
> *发送时间:* 2019年10月14日(星期一) 晚上9:21
> *收件人:* "Hbase-User";
> *主题:* Re: Failure to compile package hbase-operator-tools
>
> Hi!
>
> Git holds the in progress work of the project and isn't intended for
> downstream use. Please try to use the published hbase-operator-tools and
> let us know if you have an issue. You can find a link to the current
> release on our downloads page:
>
> http://hbase.apache.org/downloads.html
>
>
> Alternatively, feel free to subscribe to d...@hbase.apache.org and bring up
> any issues you have trying to work with the in progress stuff.
>
> On Mon, Oct 14, 2019, 02:50 DavdShao <1336318...@qq.com> wrote:
>
> > hi guys,
> >   When I compiled the toolkit downloaded from gitup, the
> > display failed.
> >  My commands are as follows：
> >   git clone
> > https://github.com/apache/hbase-operator-tools.git
> >  ; 
> > cd hbase-operator-tools
> >   vi pom.xml (Modify 

Re: Failure to compile package hbase-operator-tools

[Sean Busbey]

Hi!

Git holds the in progress work of the project and isn't intended for
downstream use. Please try to use the published hbase-operator-tools and
let us know if you have an issue. You can find a link to the current
release on our downloads page:

http://hbase.apache.org/downloads.html


Alternatively, feel free to subscribe to d...@hbase.apache.org and bring up
any issues you have trying to work with the in progress stuff.

On Mon, Oct 14, 2019, 02:50 DavdShao <1336318...@qq.com> wrote:

> hi guys,
>   When I compiled the toolkit downloaded from gitup, the
> display failed.
>  My commands are as follows：
>   git clone
> https://github.com/apache/hbase-operator-tools.git
>  ; 
> cd hbase-operator-tools
>   vi pom.xml (Modify HBase version configuration)
>   mvn clean package
>
>
> But after the "mvn clean ..." command is executed, the error is reported
> as follows
>
>
> " INFO] Apache HBase Operator Tools  SUCCESS
> [ 3.644 s]
> [INFO] Apache HBase - HBCK2 ... FAILURE
> [ 6.522 s]
> [INFO] Apache HBase Operator Tools - Assembly . SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 10.347 s
> [INFO] Finished at: 2019-10-14T15:29:27+08:00
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile
> (default-compile) on project hbase-hbck2: Compilation failure: Compilation
> failure:
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[49,38]
> cannot find symbol
> [ERROR] symbol: class Hbck
> [ERROR] location: package org.apache.hadoop.hbase.client
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[138,28]
> cannot find symbol
> [ERROR] symbol: class Hbck
> [ERROR] location: class org.apache.hbase.HBCK2
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[267,22]
> cannot find symbol
> [ERROR] symbol: class Hbck
> [ERROR] location: class org.apache.hbase.HBCK2
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[284,24]
> cannot find symbol
> [ERROR] symbol: class Hbck
> [ERROR] location: class org.apache.hbase.HBCK2
> [ERROR]
> /home/cdcshaod/packages/hbase-operator-tools/hbase-hbck2/src/main/java/org/apache/hbase/HBCK2.java:[341,33]
> cannot find symbol
> [ERROR] symbol: class Hbck
> [ERROR] location: class org.apache.hbase.HBCK2
> [ERROR] - [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn  "
>
>
> I wonder if you have ever encountered such a mistake.
>
>
>
>
>
>
> DavidShao
>
>
>  

Re: Hbase errors for " ERROR: org.apache.hadoop.hbase.PleaseHoldException:Master is initializing"

[Sean Busbey]

che.hadoop.hbase.master.assignment.AssignmentManager: STUCK
> Region-In-Transition rit=OPENING, location=dtla1apps21,16020,1566557484692,
> table=KYLIN_2ZJ8M4PT96, region=1a3f6b1f03c417bc04234bcdc97146c6
> 2019-10-11 15:13:00,294 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK
> Region-In-Transition rit=OPENING, location=dtla1apps21,16020,1566557484692,
> table=KYLIN_CGMROZNC85, region=f46ca8c4f65a7c99ccd0dab22e4b83dd
>  ”
>
>
> Actually, now I can see from the HBase page which regions are in the
> opening state. And the regions of these openinng states can now be queried，
> but I don't know how to deal with this error of"
> ERROR:org.apache.hadoop.hbase.PleaseHoldException: Master is initializing"
>
>
>
>
> DavidShao
>
>
>
>
> -- Original --
> From:  "Sean Busbey";;
> Send time: Friday, Oct 11, 2019 2:16 PM
> To: "Hbase-User";
>
> Subject:  Re: Hbase errors for " ERROR:
> org.apache.hadoop.hbase.PleaseHoldException:Master is initializing"
>
>
>
> what version of hbase are you using?
>
> If master is still initializing, then probably one of the regions in
> transition is the region for meta, or a region for a system table (e.g.
> hbase:namespace). You should check the master logs to figure out which one
> and what their state is.
>
> On Fri, Oct 11, 2019 at 1:09 AM 青椒肉丝 <1336318...@qq.com> wrote:
>
> > hi ,guys:
> >My HBase made the following mistakes when creating a build table or
> > disable table
> > " ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is
> > initializing
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2998)
> > at
> > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1968)
> > at
> >
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:624)
> > at
> >
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> > at
> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> > at
> > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "
> >
> >
> > And the HBase rit state is abnormal，HBase page display "41 region(s) in
> > transition for more than 6 milliseconds.",  Now you can't see any
> user
> > tables from the HBase page, but you can see the list command in the HBase
> > shell, If any friend has ever met this kind of problem, please help to
> > answer it. Thank you.
> >
> >
> >
> >
> > DavidShao

Re: Hbase errors for " ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing"

[Sean Busbey]

what version of hbase are you using?

If master is still initializing, then probably one of the regions in
transition is the region for meta, or a region for a system table (e.g.
hbase:namespace). You should check the master logs to figure out which one
and what their state is.

On Fri, Oct 11, 2019 at 1:09 AM 青椒肉丝 <1336318...@qq.com> wrote:

> hi ,guys:
>My HBase made the following mistakes when creating a build table or
> disable table
> " ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is
> initializing
> at
> org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2998)
> at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1968)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:624)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) "
>
>
> And the HBase rit state is abnormal，HBase page display "41 region(s) in
> transition for more than 6 milliseconds.",  Now you can't see any user
> tables from the HBase page, but you can see the list command in the HBase
> shell, If any friend has ever met this kind of problem, please help to
> answer it. Thank you.
>
>
>
>
> DavidShao

Re: NoSuchMethodError: org.apache.hadoop.hbase.CellComparator.getInstance() while trying to bulk load in hbase using spark

[Sean Busbey]

Please use a released version of the hbase-connectors project. The master
branch isn't meant for downstream use (or feel free to come over to
dev@hbase and help us work on the not-yet-released version).

Please be aware hbase 2.0 is EOM and you should upgrade.

FYI, I've only seen testing of hbase-connectors with hbase 2.1+, so this
may or may not be possible on HBase 2.0.

That error message looks like you have an older hbase jar on the classpath.
That method is present both in hbase 2.0 and 2.1.

Can you get the full classpath out of the executor logs and see if it has
more hbase references?

On Tue, Oct 8, 2019, 10:03 Karthik Bikkumala 
wrote:

> Hi,
>
> I am trying to Bulk Load data from HDFS to HBase. I used the following
> example
>
> https://github.com/apache/hbase-connectors/blob/master/spark/hbase-spark/src/main/java/org/apache/hadoop/hbase/spark/example/hbasecontext/JavaHBaseBulkLoadExample.java
>
>
> I built the module with following command
>
> mvn -Dspark.version=2.4.3 -Dscala.version=2.11.7
> -Dscala.binary.version=2.11 clean install
>
> when i tried to run the example using the spark-submit command, i am
> getting the following error:
>
> Caused by: java.lang.NoSuchMethodError:
>
> org.apache.hadoop.hbase.CellComparator.getInstance()Lorg/apache/hadoop/hbase/CellComparator;
>
> at
>
> org.apache.hadoop.hbase.regionserver.StoreFileWriter$Builder.(StoreFileWriter.java:348)
>
> at org.apache.hadoop.hbase.spark.HBaseContext.org
>
> $apache$hadoop$hbase$spark$HBaseContext$$getNewHFileWriter(HBaseContext.scala:928)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$2.apply(HBaseContext.scala:1023)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$2.apply(HBaseContext.scala:972)
>
> at scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:79)
>
> at org.apache.hadoop.hbase.spark.HBaseContext.org
>
> $apache$hadoop$hbase$spark$HBaseContext$$writeValueToHFile(HBaseContext.scala:972)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$bulkLoad$3$$anonfun$apply$7.apply(HBaseContext.scala:677)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$bulkLoad$3$$anonfun$apply$7.apply(HBaseContext.scala:675)
>
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>
> at
>
> org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$bulkLoad$3.apply(HBaseContext.scala:675)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$bulkLoad$3.apply(HBaseContext.scala:664)
>
> at org.apache.hadoop.hbase.spark.HBaseContext.org
>
> $apache$hadoop$hbase$spark$HBaseContext$$hbaseForeachPartition(HBaseContext.scala:490)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$foreachPartition$1.apply(HBaseContext.scala:106)
>
> at
>
> org.apache.hadoop.hbase.spark.HBaseContext$$anonfun$foreachPartition$1.apply(HBaseContext.scala:106)
>
> at
>
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
>
> at
>
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:935)
>
> at
>
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
>
> at
>
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:121)
>
> at
>
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
>
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> Please find the code here (pom.xml, mvn dependency tree, source file):
> https://gist.github.com/bikkumala/d2e349c7bfaffc673e8a641ff3ec9d33
>
> I tried with the following versions
> Spark : 2.4.x
> HBase : 2.0.x
> Hadoop : 2.7.x
>
>
> I created a JIRA for the same:
> https://jira.apache.org/jira/browse/HBASE-23135
>
> I've also tried excluding hbase-common from dependency and explicitly
> adding it as explained by Zheng Hu
>  in
> above JIRA, but i am facing the same issue.
>
> Any help is appreciated.
>
>
> Thank You,
>
> Bikkumala.
>

[ANNOUNCE] end of maintenance (EOM) for Apache HBase 2.0 releases

[Sean Busbey]

hi folks!

As previously planned the 2.0 release line is now end of maintenance.
Apache HBase 2.0.6 was the final release and is still available from
archive.apache.org.

Users of 2.0 based releases should upgrade to a later HBase 2.y based
release as soon as possible.

Thanks you to all the contributors who made branch-2.0 possible over
the last 2.25 years.

-- 
Sean

Re: Building latest hbase-native-client

[Sean Busbey]

to my knowledge we do not currently have a meaningful performance
comparison. you should ask on the dev@hbase list though, since I have
also not been paying close attention to that part of the project.

On Fri, Sep 20, 2019 at 12:56 PM Sudhir Babu Pothineni
 wrote:
>
> Oh thank you!
>
> Is there drastic performance advantage compared to Java API?
>
> > On Sep 20, 2019, at 12:04 PM, Sean Busbey  wrote:
> >
> > the hbase-native-client work is not suitable for use by downstream
> > folks. It is currently only internal to the hbase project.
> >
> > If you'd like to help make it usable and ready for folks outside of
> > the project please join the d...@hbase.apache.org mailing list and
> > bring the topic up there.
> >
> > On Fri, Sep 20, 2019 at 11:50 AM Sudhir Babu Pothineni
> >  wrote:
> >>
> >> Any wiki for building https://github.com/apache/hbase-native-client
> >>
> >> This one is refactored from hbase, any wiki?
> >>
> >> Thanks
> >> Sudhir

Re: Building latest hbase-native-client

[Sean Busbey]

the hbase-native-client work is not suitable for use by downstream
folks. It is currently only internal to the hbase project.

If you'd like to help make it usable and ready for folks outside of
the project please join the d...@hbase.apache.org mailing list and
bring the topic up there.

On Fri, Sep 20, 2019 at 11:50 AM Sudhir Babu Pothineni
 wrote:
>
> Any wiki for building https://github.com/apache/hbase-native-client
>
> This one is refactored from hbase, any wiki?
>
> Thanks
> Sudhir

Re: NOTICE: hbase-2.2.1, 2.1.6 and 2.0.6 have a bug may cause data loss

[Sean Busbey]

I'm scheduled to remove 2.0 related materials tomorrow, so I'd really
like us to avoid restarting that clock.

As far as known issues, just make sure the affected version(s) include
2.0.6 and call it a day.

On Thu, Sep 19, 2019 at 11:04 AM Stack  wrote:
>
> First, good find Guanghao. Thanks for the fix.
>
> I can roll RCs for branch-2 (Would be good to do it after
> hbase-operator-tools goes out because then these new releases become
> candidates for the stable pointer).
>
> On branch-2.0, given its EOM, I could add a 'KNOWN_ISSUES.md' file to
> http://apache.mirrors.lucidnetworks.net/hbase/2.1.6/ and list this issue in
> it. Then update the note in the download page to point at new known issues
> doc? (Suggestions appreciated).
>
> S
>
>
>
> On Wed, Sep 18, 2019 at 5:55 PM Guanghao Zhang  wrote:
>
> > Only 2.2.1, 2.1.6 and 2.0.6 are effected. Earlier releases are fine. Please
> > avoid downloading these releases. If you already running these versions in
> > your cluster, please use splitormerge_switch to disable region merge
> > now. See HBASE-23044 for more details.
> >
> > Need to rolling new release for branch-2.1 and branch-2.2 line. Ping @stack
> > for branch-2.0, do we need a more release 2.0.7?
> >
> > Thanks
> > Guanghao Zhang
> >

Re: region-in-transtion and recovering-regions

[Sean Busbey]

please upgrade to a version of HBase that is not EOM. As a community
we dropped 0.98 in April 2017. 0.98.7 is from almost 5 years ago.

Also attachments are dropped by the mailing list, so if you'd like to
reference one please upload the image somewhere and provide a link.

On Tue, Sep 17, 2019 at 1:15 PM Jignesh Patel  wrote:
>
> We are running Hadoop 2.6.0 and hbase 0.98.7 and I do see following issue in 
> our system.
>
>
> I did followed the discussion  HBase - Region in Transition, however after 
> restarting in HBase I still see the same status of region-in-transtion and 
> recovering-regions.
>
> How do I fix the error.

Re: HBase

[Sean Busbey]

Version 3.0.0 of hbase has no specific plan for release. If you'd like to
discuss it please subscribe to dev@hbase and bring it up there.

On Mon, Aug 26, 2019, 17:49 עידו Gmail  wrote:

> Hi,
>
> Thank you for the follow up.
>
> Can you please share an ETA when version 3 will be released?
>
> Thanks!
>
> Ido
>
>
> > On 26 Aug 2019, at 10:23, 李响  wrote:
> >
> > Hi Ido,
> >
> > 1) The latest version is 2.2.0 (for v2.x) and 1.4.10 (for v1.x).
> > 2) 3.0.0-SNAPSHOT is for development only in master branch. You could
> only build HBase bin of this version from source code by yourself... This
> version is not for end user I believe.
> >
> >> On Mon, Aug 26, 2019 at 1:23 PM Ido Kaplan 
> wrote:
> >> Hi,
> >>
> >>
> >> What is the latest version of HBase? I see in the download page that
> it's
> >> version 2.2.0, but I see in the documentations "version 3.0.0-SNAPSHOT".
> >>
> >> Please advise,
> >>
> >> Thanks,
> >>
> >> Ido
> >
> >
> > --
> >
> >李响 Xiang Li
> >
> > 手机 cellphone ：+86-136-8113-8972
> > 邮件 e-mail  ：wate...@gmail.com
>

Re: HBase

[Sean Busbey]

Xiang Li has it correct. Any version you should use is linked from our
download page:

http://hbase.apache.org/downloads.html

As well as in the ASF mirror system:

https://www.apache.org/dyn/closer.lua/hbase/

Additionally, if you're unsure of what version to use, we recommend you
stick with our "stable" release.

https://www.apache.org/dyn/closer.lua/hbase/stable/

It's currently 1.4.10.

On Mon, Aug 26, 2019, 02:23 李响  wrote:

> Hi Ido,
>
> 1) The latest version is 2.2.0 (for v2.x) and 1.4.10 (for v1.x).
> 2) 3.0.0-SNAPSHOT is for development only in master branch. You could only
> build HBase bin of this version from source code by yourself... This
> version is not for end user I believe.
>
> On Mon, Aug 26, 2019 at 1:23 PM Ido Kaplan  wrote:
>
> > Hi,
> >
> >
> > What is the latest version of HBase? I see in the download page that it's
> > version 2.2.0, but I see in the documentations "version 3.0.0-SNAPSHOT".
> >
> > Please advise,
> >
> > Thanks,
> >
> > Ido
> >
>
>
> --
>
>李响 Xiang Li
>
> 手机 cellphone ：+86-136-8113-8972
> 邮件 e-mail  ：wate...@gmail.com
>

Re: Mob reference tags go missing

[Sean Busbey]

Also the hfile version setting and hbase version on your masters, presuming
you are using the mob compaction chore in the master.

On Sat, Aug 17, 2019, 09:59 Sean Busbey  wrote:

> That sounds like some of your hfiles are being written as hfile v2 instead
> of v3. Can you check versions and configs on all your region servers?
>
> On Sat, Aug 17, 2019, 08:42 Omri Cohen  wrote:
>
>> Sorry, accidentally pressed "send" before i finished the massage.
>>
>> Hello,
>>
>> We recently encountered a problem in our production hbase cluster (CDH
>> deployment, version 5.13.1).
>>
>> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
>> the threshold is the default 102400 bytes). The cluster has been active for
>> about a year. Recently, our clients started to receive the mob reference
>> instead of the mob data when trying to access certain rows (in most tables
>> about 10% of the rows are bad, in one table it is about 50% of rows).
>>
>> When we investigated the HFiles, we saw that the "bad" cells are missing
>> two tags that exist in the "good" rows, it looks something like that:
>>
>> K: {rowkey of a "good" row} {Column family, column qualifier}...
>> vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]:  T[1]: {table name}
>> K: {rowkey of a "bad" row} {Column family, column qualifier}...
>> vlen=76/seqid=... V: \x00\x13\x1A{MOB file name}
>>
>> The "good" row returns the expected data when queried, while the "bad"
>> row returned the mob file reference instead of the data. This happened when
>> we queried from the REST server, the native java client, and the hbase
>> shell.
>>
>> When we examined the mob files, we saw that all the data was there.
>>
>>   *   Has anyone encountered a similar situation?
>>   *   Is it possible to manually add the missing mob reference tags to
>> the "bad" rows?
>>
>> No changes where made to the table in recently. We never encountered this
>> issue before.
>>
>> We would appreciate any help on this issue.
>> 
>> From: Omri Cohen
>> Sent: Saturday, August 17, 2019 4:28 PM
>> To: hbase-u...@hadoop.apache.org 
>> Subject: Mob reference tags go missing
>>
>> Hello,
>>
>> We recently encountered a problem in our production hbase cluster.
>> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
>> the threshold is the default 102400 bytes). The cluster has been active for
>> about a year. Recently, our clients started to receive the mob reference
>> instead of the mob data when trying to access certain rows (in most tables
>> about 10% of the rows are bad, in one table it is about 50% of rows).
>>
>> When we investigated the HFiles, we saw that the "bad" cells are missing
>> two tags that exist in the "good" rows, it looks something like that:
>>
>

Re: Mob reference tags go missing

[Sean Busbey]

That sounds like some of your hfiles are being written as hfile v2 instead
of v3. Can you check versions and configs on all your region servers?

On Sat, Aug 17, 2019, 08:42 Omri Cohen  wrote:

> Sorry, accidentally pressed "send" before i finished the massage.
>
> Hello,
>
> We recently encountered a problem in our production hbase cluster (CDH
> deployment, version 5.13.1).
>
> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
> the threshold is the default 102400 bytes). The cluster has been active for
> about a year. Recently, our clients started to receive the mob reference
> instead of the mob data when trying to access certain rows (in most tables
> about 10% of the rows are bad, in one table it is about 50% of rows).
>
> When we investigated the HFiles, we saw that the "bad" cells are missing
> two tags that exist in the "good" rows, it looks something like that:
>
> K: {rowkey of a "good" row} {Column family, column qualifier}...
> vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]:  T[1]: {table name}
> K: {rowkey of a "bad" row} {Column family, column qualifier}...
> vlen=76/seqid=... V: \x00\x13\x1A{MOB file name}
>
> The "good" row returns the expected data when queried, while the "bad" row
> returned the mob file reference instead of the data. This happened when we
> queried from the REST server, the native java client, and the hbase shell.
>
> When we examined the mob files, we saw that all the data was there.
>
>   *   Has anyone encountered a similar situation?
>   *   Is it possible to manually add the missing mob reference tags to the
> "bad" rows?
>
> No changes where made to the table in recently. We never encountered this
> issue before.
>
> We would appreciate any help on this issue.
> 
> From: Omri Cohen
> Sent: Saturday, August 17, 2019 4:28 PM
> To: hbase-u...@hadoop.apache.org 
> Subject: Mob reference tags go missing
>
> Hello,
>
> We recently encountered a problem in our production hbase cluster.
> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
> the threshold is the default 102400 bytes). The cluster has been active for
> about a year. Recently, our clients started to receive the mob reference
> instead of the mob data when trying to access certain rows (in most tables
> about 10% of the rows are bad, in one table it is about 50% of rows).
>
> When we investigated the HFiles, we saw that the "bad" cells are missing
> two tags that exist in the "good" rows, it looks something like that:
>

Re: Speedup region transition

[Sean Busbey]

Depending on your version of HBase, HBASE-22263 has a workaround
section that describes some tuning options for making assignments
happen faster. (if opening at the new RS is the part of a transition
that's going slow for you.)


On Thu, Aug 8, 2019 at 8:54 AM Alexander Batyrshin <0x62...@gmail.com> wrote:
>
> Hello all,
> Is there any way to make regions transition faster by configuration options?

Re: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC

[Sean Busbey]

Thanks for taking on these added responsibilities Zheng!

On Sun, Aug 4, 2019 at 9:08 PM Duo Zhang  wrote:
>
> On behalf of the Apache HBase PMC I am pleased to announce that Zheng Hu
> has accepted our invitation to become a PMC member on the Apache HBase
> project. We appreciate Zheng Hu stepping up to take more responsibility in
> the HBase project.
>
> Please join me in welcoming Zheng Hu to the HBase PMC!

[ANNOUNCE] New HBase committer Tak-Lon (Stephen) Wu

[Sean Busbey]

On behalf of the Apache HBase PMC I am super pleased to announce that
Tak-Lon (Stephen) Wu has accepted the PMC's invitation to become a
commiter on the project.

Thanks so much for the work you've been contributing. We look forward
to your continued involvement.

Congratulations and welcome!

-busbey

[ANNOUNCE] new HBase committer Sakthi

[Sean Busbey]

On behalf of the HBase PMC, I'm pleased to announce that Sakthi has
accepted our invitation to become an HBase committer.

We'd like to thank Sakthi for all of his diligent contributions to the
project thus far. We look forward to his continued participation in our
community.

Congrats and welcome Sakthi!

Re: HBase Build Failed (License errors detected)

[Sean Busbey]

To get the error details you have to open the file mentioned:

/ws/build/hbase2/rpm/BUILD/hbase-2.1.5-src/hbase-shaded/hbas
e-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE

And then search for the details around the text "ERROR"

On Fri, Jul 12, 2019, 00:46 Kang Minwoo  wrote:

> I try to build HBase 2.1.5
> error is..
>
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell
> failed with message:
> License errors detected, for more detail find ERROR in
>
> /ws/build/hbase2/rpm/BUILD/hbase-2.1.5-src/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
> [INFO]
> 
> [INFO] Reactor Summary for Apache HBase 2.1.5:
> [INFO]
> [INFO] Apache HBase ... FAILURE [02:29
> min]
> [INFO] Apache HBase - Checkstyle .. SKIPPED
> [INFO] Apache HBase - Annotations . SKIPPED
> [INFO] Apache HBase - Build Configuration . SKIPPED
> [INFO] Apache HBase - Shaded Protocol . SKIPPED
> [INFO] Apache HBase - Common .. SKIPPED
> [INFO] Apache HBase - Metrics API . SKIPPED
> [INFO] Apache HBase - Hadoop Compatibility  SKIPPED
> [INFO] Apache HBase - Metrics Implementation .. SKIPPED
> [INFO] Apache HBase - Hadoop Two Compatibility  SKIPPED
> [INFO] Apache HBase - Protocol  SKIPPED
> [INFO] Apache HBase - Client .. SKIPPED
> [INFO] Apache HBase - Zookeeper ... SKIPPED
> [INFO] Apache HBase - Replication . SKIPPED
> [INFO] Apache HBase - Resource Bundle . SKIPPED
> [INFO] Apache HBase - HTTP  SKIPPED
> [INFO] Apache HBase - Procedure ... SKIPPED
> [INFO] Apache HBase - Server .. SKIPPED
> [INFO] Apache HBase - MapReduce ... SKIPPED
> [INFO] Apache HBase - Testing Util  SKIPPED
> [INFO] Apache HBase - Thrift .. SKIPPED
> [INFO] Apache HBase - RSGroup . SKIPPED
> [INFO] Apache HBase - Shell ... SKIPPED
> [INFO] Apache HBase - Coprocessor Endpoint  SKIPPED
> [INFO] Apache HBase - Integration Tests ... SKIPPED
> [INFO] Apache HBase - Rest  SKIPPED
> [INFO] Apache HBase - Examples  SKIPPED
> [INFO] Apache HBase - Shaded .. SKIPPED
> [INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SKIPPED
> [INFO] Apache HBase - Shaded - Client . SKIPPED
> [INFO] Apache HBase - Shaded - MapReduce .. SKIPPED
> [INFO] Apache HBase - External Block Cache  SKIPPED
> [INFO] Apache HBase - Assembly  SKIPPED
> [INFO] Apache HBase Shaded Packaging Invariants ... SKIPPED
> [INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled)
> SKIPPED
> [INFO] Apache HBase - Archetypes .. SKIPPED
> [INFO] Apache HBase - Exemplar for hbase-client archetype . SKIPPED
> [INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SKIPPED
> [INFO] Apache HBase - Archetype builder ... SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time:  02:31 min
> [INFO] Finished at: 2019-07-12T05:31:20Z
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on
> project hbase: failed to get report for
> org.apache.maven.plugins:maven-javadoc-plugin: Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce
> (check-aggregate-license) on project hbase-shaded-client: Some Enforcer
> rules have failed. Look above for specific messages explaining why the rule
> failed. -> [Help 1]
>
> 
> 보낸 사람: Sean Busbey 
> 보낸 날짜: 2019년 7월 12일 금요일 14:14
> 받는 사람: user@hbase.apache.org
> 제목: Re: HBase Build Failed (License errors detected)
>
> As a project we can't speak on what your risk tolerance is for open
> source licensing.
>
> what HBase version are you trying to build from source? what's the
> specific error reported in the LICENSE

Re: HBase Build Failed (License errors detected)

[Sean Busbey]

As a project we can't speak on what your risk tolerance is for open
source licensing.

what HBase version are you trying to build from source? what's the
specific error reported in the LICENSE file?

probably the particular Hadoop version you are building added some new
dependency that we haven't accounted for (or that wasn't accounted for
in the release you are using)

On Thu, Jul 11, 2019 at 11:38 PM Kang Minwoo  wrote:
>
> Hello, User.
>
> While I build HBase from the source. I got an error that is License errors 
> detected, for more detail find ERROR in 
> /hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE.
>
> My build environment is CentOS 6.3
> Command is mvn -DskipTests -Dslf4j.version=1.7.25 -Dhadoop.profile=3.0 
> -Dhadoop-three.version=3.1.2 -Dzookeeper.version=3.4.14 
> -Dcheckstyle.skip=true clean install site assembly:single
>
> I think I have an option that does not check the license.
> Is it okay that do not check license?
>
> Best regards,
> Minwoo Kang

[ANNOUNCE] end of maintenance (EOM) for Apache HBase 1.2 releases

[Sean Busbey]

hi folks!

As previously mentioned during ANNOUNCE emails for Apache HBase 1.2
releases, there are no further releases planned from branch-1.2.
Apache HBase 1.2.12 was the final release, and it is still available
from archive.apache.org.

Users of 1.2 based releases should upgrade to the current stable
release (currently 1.4.9) as soon as possible.

Thank you again to all the contributors who made branch-1.2 successful
over its 3+ year lifespan.


-busbey

Re: [DISCUSS] Publishing hbase binaries with different hadoop release lines

[Sean Busbey]

What about moving back to having a per-major-version-of-hadoop
compatibility module again that builds against one needed major version all
the time? (Presumably with some shell script magic to pick the right one?)
that would be preferable imho to e.g. producing main project binary
tarballs per Hadoop version.

Or! We could move stuff that relies on brittle Hadoop internals into its
own repo (or one of our existing repos) and build _that_ with binaries for
our supported Hadoop versions. Then in the main project we can include the
appropriate artifact for the version of Hadoop we happen to build with
(essentially leaving the main repo how it is) and update our "replace the
version of Hadoop!" note to including replacing this "HBase stuff that's
closely tied to Hadoop internals" jar as well.

On Wed, May 29, 2019, 08:41 张铎(Duo Zhang)  wrote:

> See the comments here
>
> https://issues.apache.org/jira/browse/HBASE-22394
>
> Although we claim that hbase 2.1.x can work together with hadoop 3.1.x,
> actually we require users to build the hbase binary with hadoop 3.1.x by
> their own if they really want to use hbase together with hadoop 3.1.x
> clients.
>
> The problem for HBASE-22394 is that our asyncfswal references some
> IA.Private classes and they have been changed between minor releases. It
> can be solved by using reflection. But in general, since hadoop also
> follows the semantic versioning, I do not think it is safe to do drop-in
> replacement between minor releases. So maybe a better way is to publish a
> hbase binary for each hadoop minor release line?
>
> Thanks.
>

[ANNOUNCE] Please welcome Jan Hentschel to the Apache HBase PMC

[Sean Busbey]

On behalf of the Apache HBase PMC I am pleased to announce that Jan
Hentschel has accepted our invitation to become a PMC member on the
HBase project. We appreciate Jan stepping up to take more
responsibility in the HBase project.

Please join me in welcoming Jan to the HBase PMC!



As a reminder, if anyone would like to nominate another person as a
committer or PMC member, even if you are not currently a committer or
PMC member, you can always drop a note to priv...@hbase.apache.org to
let us know.

-busbey

[ANNOUNCE] Apache HBase 1.2.12 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.2.12

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.2.12 is the twelfth maintenance release in the HBase 1.2 line,
continuing on the theme of bringing a stable, reliable database to the
Hadoop and NoSQL communities. This release includes one bug fix and
an operational improvement done in the month and a half since 1.2.11.

All users of previous 1.2.z releases are encouraged to upgrade to either
this release or the latest in our stable release line, which is currently
1.4.9. This is expected to be the last release in the 1.2 line. Users
wishing to get further fixes will need to upgrade to an active release line.

Critical changes include:

* HBASE-22058 - upgrade thrift dependency to 0.9.3.1 (fixes a CVE)
* HBASE-21926 - Profiler servlet

The full list of fixes included in this release is available at:

https://s.apache.org/hbase-1.2.12-jira-release-notes

and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/hbase-1.2.12

The relevant checksums files are available at:

https://www.apache.org/dist/hbase/hbase-1.2.12/hbase-1.2.12-src.tar.gz.sha512
https://www.apache.org/dist/hbase/hbase-1.2.12/hbase-1.2.12-bin.tar.gz.sha512

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/hbase-1.2.12/hbase-1.2.12-src.tar.gz.asc
https://www.apache.org/dist/hbase/hbase-1.2.12/hbase-1.2.12-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at:
d...@hbase.apache.org.

Cheers,
The HBase Dev Team

Re: Spark HBase Partitionner

[Sean Busbey]

Spark integration stuff goes into the HBase connectors repo:

https://github.com/apache/hbase-connectors

Don't know if there's a partitioner already or not.

On Thu, Apr 11, 2019, 09:14 Jean-Marc Spaggiari 
wrote:

> Hi,
>
> Do we have an HBase Partitionner for Spark? It's pretty straight forward
> but I'm not able to find one. If we don't, I will do it and contribute it
> back, but then where should it be pushed?
>
> Thanks,
>
> JMS
>

Re: illegal reflective access

[Sean Busbey]

also being tracked in HBASE-22172 and HBASE-21110

On Wed, Apr 10, 2019 at 8:39 AM Jean-Marc Spaggiari
 wrote:
>
> Didn't even realized I was running JDK11 ;) Side effect of a recent apt-get
> dist-upgrade...
>
> jmspaggi@node8:~/p$ java -version
> openjdk version "11.0.2" 2019-01-15
> OpenJDK Runtime Environment (build 11.0.2+9-Debian-3)
> OpenJDK 64-Bit Server VM (build 11.0.2+9-Debian-3, mixed mode, sharing)
>
> So for now I will ignore that.
>
> Thanks,
>
> JMS
>
>
> Le mer. 10 avr. 2019 à 09:30, 张铎(Duo Zhang)  a
> écrit :
>
> > You're using Java11? HBase has not added the support for Java11 yet...
> >
> > Jean-Marc Spaggiari  于2019年4月10日周三 下午9:25写道：
> >
> > > Hi,
> > >
> > > Does this need to be reported or should I just ignore it?
> > >
> > > Thanks,
> > >
> > > JMS
> > >
> > >
> > > WARNING: An illegal reflective access operation has occurred
> > > WARNING: Illegal reflective access by
> > > org.apache.hadoop.hbase.util.UnsafeAvailChecker
> > > (file:/tmp/p-0.0.1-SNAPSHOT-jar-with-dependencies.jar) to method
> > > java.nio.Bits.unaligned()
> > > WARNING: Please consider reporting this to the maintainers of
> > > org.apache.hadoop.hbase.util.UnsafeAvailChecker
> > > WARNING: Use --illegal-access=warn to enable warnings of further illegal
> > > reflective access operations
> > > WARNING: All illegal access operations will be denied in a future release
> > >
> >

Re: Bits getting flipped in record value

[Sean Busbey]

So you're saying no records should ever be updated, right?


Do you have any coprocessors loaded?


On Wed, Mar 20, 2019, 20:32 aheyne  wrote:

> I don't have the WALs but due to the nature of the data each record/key
> is unique. The keys for the data are generated using spatial-temporal
> dimensions of the observation.
>
> -Austin
>
> On 2019-03-20 21:25, Sean Busbey wrote:
> > Have you examined the wals for writes to the impacted cells to verify
> > an
> > update wasn't written with the change to the value?
> >
> > On Wed, Mar 20, 2019, 17:47 Austin Heyne  wrote:
> >
> >> Hey all,
> >>
> >> We're running HBase 1.4.8 on EMR 5.20 backed by S3 and we're seeing a
> >> bit get flipped in some record values.
> >>
> >> We've preformed a bulk ingest and bulk load of a large chunk of data
> >> and
> >> then pointed a live ingest feed to that table. After a period of time
> >> we
> >> found that a few records in the table had been corrupted and were one
> >> bit different from their original value. Since we saved the output of
> >> the bulk ingest we re-loaded those files and verified that at the time
> >> of bulk load the record was correct. This seems to us to indicate that
> >> at some point during the live ingest writes the record was corrupted.
> >>
> >> I've verified that the region that the record is in has never been
> >> split
> >> but it has received over 2 million write requests so there very likely
> >> could have been some minor compactions there.
> >>
> >> Has anyone seen anything like this before?
> >>
> >> Thanks,
> >> Austin
> >>
> >> --
> >> Austin L. Heyne
> >>
> >>
>

Re: Bits getting flipped in record value

[Sean Busbey]

Have you examined the wals for writes to the impacted cells to verify an
update wasn't written with the change to the value?

On Wed, Mar 20, 2019, 17:47 Austin Heyne  wrote:

> Hey all,
>
> We're running HBase 1.4.8 on EMR 5.20 backed by S3 and we're seeing a
> bit get flipped in some record values.
>
> We've preformed a bulk ingest and bulk load of a large chunk of data and
> then pointed a live ingest feed to that table. After a period of time we
> found that a few records in the table had been corrupted and were one
> bit different from their original value. Since we saved the output of
> the bulk ingest we re-loaded those files and verified that at the time
> of bulk load the record was correct. This seems to us to indicate that
> at some point during the live ingest writes the record was corrupted.
>
> I've verified that the region that the record is in has never been split
> but it has received over 2 million write requests so there very likely
> could have been some minor compactions there.
>
> Has anyone seen anything like this before?
>
> Thanks,
> Austin
>
> --
> Austin L. Heyne
>
>

[ANNOUNCE] Apache HBase 1.2.11 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.2.11

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.2.11 is the eleventh maintenance release in the HBase 1.2 line,
continuing on the theme of bringing a stable, reliable database to the
Hadoop and NoSQL communities. This release includes two bug fixes done
in the month since 1.2.10.

All users of previous 1.2.z releases are encouraged to upgrade to either
this release or the latest in our stable release line, which is currently
1.4.9. Releases in the 1.2.z line are expected to stop in late spring 2019.

Critical fixes include:

* HBASE-21196 - HTableMultiplexer clears the meta cache after every put
operation
* HBASE-21374 - FileSystem in use may get closed by other bulk load call
in secure bulkLoad

The full list of fixes included in this release is available at:

https://s.apache.org/hbase-1.2.11-jira-release-notes

and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/hbase-1.2.11

The relevant checksums files are available at:

https://www.apache.org/dist/hbase/hbase-1.2.11/hbase-1.2.11-src.tar.gz.sha512
https://www.apache.org/dist/hbase/hbase-1.2.11/hbase-1.2.11-bin.tar.gz.sha512

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/hbase-1.2.11/hbase-1.2.11-src.tar.gz.asc
https://www.apache.org/dist/hbase/hbase-1.2.11/hbase-1.2.11-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at:
d...@hbase.apache.org.

Cheers,
The HBase Dev Team

[ANNOUNCE] Apache HBase 1.2.10 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.2.10

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.2.10 is the tenth maintenance release in the HBase 1.2 line,
continuing on the theme of bringing a stable, reliable database to
the Hadoop and NoSQL communities. This release includes about a half
dozen bug fixes done in the two months since 1.2.10.

All users of previous 1.2.z releases are encouraged to upgrade to either
this release or the latest in our stable release line, which is currently
1.4.9. Releases in the 1.2.z line are expected to stop in late spring 2019.

Critical fixes include:

* HBASE-21387 - Race condition surrounding in progress snapshot handling in
snapshot cache leads to loss of snapshot files
* HBASE-21492 - CellCodec Written To WAL Before It's Verified
* HBASE-21504 - If enable FIFOCompactionPolicy, a compaction may write a
"empty" hfile whose maxTimeStamp is long max. This kind of
hfile will never be archived.
* HBASE-21582 - If call HBaseAdmin#snapshotAsync but forget call
isSnapshotFinished, then SnapshotHFileCleaner will skip to
run every time

The full list of fixes included in this release is available at:

 https://s.apache.org/hbase-1.2.10-jira-release-notes

 and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/hbase-1.2.10

The relevant checksums files are available at:


https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-src.tar.gz.sha512

https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-bin.tar.gz.sha512

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-src.tar.gz.asc
https://www.apache.org/dist/hbase/hbase-1.2.10/hbase-1.2.10-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at:
d...@hbase.apache.org.

Cheers,
The HBase Dev Team

Re: How should I be getting the set of regions in transition?

[Sean Busbey]

I fixed the fix version on HBASE-21410. it'll be new in 2.1.2 if the
current RC passes.
On Mon, Dec 10, 2018 at 5:49 PM Sean Busbey  wrote:
>
> On Mon, Dec 10, 2018 at 11:32 AM Stack  wrote:
> >
> > On Thu, Dec 6, 2018 at 7:45 AM Sean Busbey  wrote:
> >
> > > ...
> >
> > > Once I confirmed the given RS was not currently doing anything for any
> > > of those regions I figured I'd use HBCK2 to run an assigns to get
> > > things fixed. However, since there were like 900 RITs, the Master UI
> > > was unusable for getting a complete list.
> >
> >
> >
> > How unusable Sean? Was it up?
> >
>
> It was up. but we paginate the results so there's only 5 at a time.
> I'm not going to click through ~200 pages to get the list of things to
> copy/paste.
>
> >
> > > Also with that many all in
> > > the same state I want to be able to automate running against each of
> > > them.
> > >
> > > I ended up greping the master log file and pulling out the WARN
> > > messages about RIT to tease out the list of regions, then passed those
> > > to hbck2.
> > >
> > >
> >
> > Yeah. You saw the doc over on hbck2?
> > https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> >
>
> indeed. super helpful and why I knew how to use the assigns to bypass things.
>
> > Did you have:
> >
> > commit fa6373660f622e7520a9f2639485cc386f18ede0
> > Author: jingyuntian 
> > Date:   Thu Nov 8 15:30:30 2018 +0800
> >
> > HBASE-21410 A helper page that help find all problematic regions and
> > procedures
> >
> > It dumps the problematic on the UI so can save on messing in logs.
> >
>
> Hurm. the fix version on HBASE-21410 suggest I should have had it, but
> I don't think that page is present? I must be missing something.
>
> Between that and HBASE-21283 I should have plenty to put into a
> troubleshooting blurb. :)

Re: How should I be getting the set of regions in transition?

[Sean Busbey]

On Mon, Dec 10, 2018 at 11:32 AM Stack  wrote:
>
> On Thu, Dec 6, 2018 at 7:45 AM Sean Busbey  wrote:
>
> > ...
>
> > Once I confirmed the given RS was not currently doing anything for any
> > of those regions I figured I'd use HBCK2 to run an assigns to get
> > things fixed. However, since there were like 900 RITs, the Master UI
> > was unusable for getting a complete list.
>
>
>
> How unusable Sean? Was it up?
>

It was up. but we paginate the results so there's only 5 at a time.
I'm not going to click through ~200 pages to get the list of things to
copy/paste.

>
> > Also with that many all in
> > the same state I want to be able to automate running against each of
> > them.
> >
> > I ended up greping the master log file and pulling out the WARN
> > messages about RIT to tease out the list of regions, then passed those
> > to hbck2.
> >
> >
>
> Yeah. You saw the doc over on hbck2?
> https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
>

indeed. super helpful and why I knew how to use the assigns to bypass things.

> Did you have:
>
> commit fa6373660f622e7520a9f2639485cc386f18ede0
> Author: jingyuntian 
> Date:   Thu Nov 8 15:30:30 2018 +0800
>
> HBASE-21410 A helper page that help find all problematic regions and
> procedures
>
> It dumps the problematic on the UI so can save on messing in logs.
>

Hurm. the fix version on HBASE-21410 suggest I should have had it, but
I don't think that page is present? I must be missing something.

Between that and HBASE-21283 I should have plenty to put into a
troubleshooting blurb. :)

Re: [DISCUSS] EOL branch-1.3

[Sean Busbey]

correct, branch-1.2 was the stable release line prior to 1.4 becoming
it in August.

at the time I said I'd keep making monthly 1.2.z releases for ~6 months:

https://s.apache.org/EYkB

I wanted to give folks who heed our "this is the stable line" advice a
cushion for planning migration once we updated it to a different
release line.
On Fri, Dec 7, 2018 at 7:57 PM Guanghao Zhang  wrote:
>
> +1. But branch-1.2 is not EOL now?
>
> 张铎(Duo Zhang)  于2018年12月8日周六 上午9:28写道：
>
> > +1.
> >
> > Andrew Purtell  于2018年12月8日周六 上午5:45写道：
> >
> > > I'm good with doing one more 1.3 release. It would be my pleasure to
> > offer
> > > that service to the community. I like RM-ing.
> > >
> > >
> > > On Fri, Dec 7, 2018 at 12:29 PM Stack  wrote:
> > >
> > > > +1
> > > >
> > > > (Pity you have to make a release to EOL it).
> > > >
> > > > S
> > > >
> > > > On Fri, Dec 7, 2018 at 11:25 AM Andrew Purtell 
> > > > wrote:
> > > >
> > > > > We haven't had a release from branch-1.3 for a long time and do not
> > > > appear
> > > > > to have an active RM for it. Unless a RM for 1.3 steps forward and
> > > > promises
> > > > > to make a release in the very near future, I propose we make one more
> > > > > release of 1.3, from the head of branch-1.3, and then retire the
> > > branch.
> > > > If
> > > > > this is acceptable I can RM the final 1.3 release.
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew
> > > > >
> > > > > Words like orphans lost among the crosstalk, meaning torn from
> > truth's
> > > > > decrepit hands
> > > > >- A23, Crosstalk
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >

Re: [DISCUSS] EOL branch-1.3

[Sean Busbey]

big +1 from me.
On Fri, Dec 7, 2018 at 1:25 PM Andrew Purtell  wrote:
>
> We haven't had a release from branch-1.3 for a long time and do not appear
> to have an active RM for it. Unless a RM for 1.3 steps forward and promises
> to make a release in the very near future, I propose we make one more
> release of 1.3, from the head of branch-1.3, and then retire the branch. If
> this is acceptable I can RM the final 1.3 release.
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk

Re: [DISCUSS] No more release branches for 1.x, release from branch-1 directly

[Sean Busbey]

yes please this would be great!

I have a DISCUSS thread I've been drafting about doing the same thing
for branch-2 releases. I think in general we need to get back in the
habit of only making maintenance releases when someone steps forward
with a specific need.

I plan to keep making branch-1.2 releases until April 2019.
On Fri, Dec 7, 2018 at 1:36 PM Andrew Purtell  wrote:
>
> Please be advised I plan to RM the next minor release from branch-1, 1.5.0,
> in January of 2019. Once this is done we can continue making maintenance
> releases from branch-1.4 as needed but expect that not to be necessary
> after a couple of months (or perhaps even immediately).
>
> I see no need to make a branch-1.5. As community resources continue to
> shift away from branch-1 we need to conserve available attention. I don't
> see why we cannot release directly from branch-1. Certainly in the
> beginning any branch-1.5 would be lock step with branch-1. No distinction
> in branch curation means no need for a new branch, at least initially.
> Also, should a commit land in branch-1 that requires a new minor per our
> compatibility guidelines then I don't see why the next release from
> branch-1 cannot a new minor (1.6.0, etc.) right there and then. We have
> expressed intent to make more frequent minor releases anyhow.
>
> Related, I started a DISCUSS thread about EOL of branch-1.3.
>
> In my opinion the optimal future for branch-1, until all attention moves
> away from it, is continuing releases directly from branch-1 and perhaps
> branch-1.2 (depends on Busbey's plans for it).
>
> If you would prefer we continue to make new branches for minor code lines,
> I can do that for 1.5, no problem, but perhaps you will agree it is no
> longer necessary.
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk

Re: How should I be getting the set of regions in transition?

[Sean Busbey]

Excellent! that definitely would have been perfect for the first case.
In the second one the master was still initializing so it'd depend on
if master will respond to cluster status requests in that state.

I'll go grab the patch and get ready to use it the next time I hit
this. If it works I'm going to add some docs to the ref guide and a
release note.
On Thu, Dec 6, 2018 at 11:05 AM Andrew Purtell  wrote:
>
> HBASE-21283
>
> > On Dec 6, 2018, at 8:55 AM, Andrew Purtell  wrote:
> >
> > I recently added a shell command "rit" that displays the list of current 
> > RIT. Would that have worked? It does require that the master is responsive 
> > to a GetClusterStatus request.
> >
> >
> >> On Dec 6, 2018, at 7:45 AM, Sean Busbey  wrote:
> >>
> >> This week I've run into two cases where I needed the set of regions in
> >> transition so I could recover them and I ran into what I think is a
> >> gap in our operator tooling. I'm hoping folks will have some ideas
> >> I've missed.
> >>
> >> Depending on how this thread goes, I'll make some follow-on on the
> >> dev@hbase list for implementing changes and documentation.
> >>
> >> Case 1: HBase 1.2-ish RIT following RS crash
> >>
> >> Cluster had a handful of region servers fail and for whatever reason a
> >> few regions were stuck in transition. The operator I was helping
> >> already is used to dealing with the occasional manual recovery. Their
> >> normal process looks like this:
> >>
> >> 1) Got to Master UI website
> >> 2) Scroll down to Regions in Transition list
> >> 3) Find a RIT in FAILED_CLOSE / FAILED_OPEN / PENDING_OPEN
> >> 4) confirm on RS logs that the RS associated in the above is now in
> >> good health and doesn't expect to do anything with said region
> >> 5) run "assign" in the hbase shell for the region
> >>
> >> Unfortunately, the cluster's HDFS was under duress and so listing
> >> snapshot information was super slow. This caused the Master UI website
> >> to hang prior to displaying the RIT list.
> >>
> >> We ended up looking at the master log file.
> >>
> >> Case 2: HBase 2.1-ish RIT following cluster wide crash
> >>
> >> AFAICT cluster had experienced a failure of all RS and masters. Upon
> >> coming back up Master was left with ~10% of ~10K regions in a state of
> >> PENDING_OPEN or OPENING all with a RS that had no idea it was involved
> >> with those regions. I'm pretty sure this is a bug;  I'm still triaging
> >> it and I don't think it's relevant to the current question.
> >>
> >> Once I confirmed the given RS was not currently doing anything for any
> >> of those regions I figured I'd use HBCK2 to run an assigns to get
> >> things fixed. However, since there were like 900 RITs, the Master UI
> >> was unusable for getting a complete list. Also with that many all in
> >> the same state I want to be able to automate running against each of
> >> them.
> >>
> >> I ended up greping the master log file and pulling out the WARN
> >> messages about RIT to tease out the list of regions, then passed those
> >> to hbck2.
> >>
> >> 
> >>
> >> Am I missing some obvious place where I can use a CLI tool to get a
> >> list of RIT? I don't see anything in the ref guide. I looked through
> >> the help of HBCK 1 and the shell and couldn't find anything.
> >>
> >> I think I can use Admin.getClusterStatus() and getClusterMetrics() to
> >> get this info from the Java API. That means there's some way to get it
> >> in the hbase shell, but it'll probably be ugly. If there's not already
> >> an easier way I'll want to wrap that so it's a simple command.

How should I be getting the set of regions in transition?

[Sean Busbey]

This week I've run into two cases where I needed the set of regions in
 transition so I could recover them and I ran into what I think is a
gap in our operator tooling. I'm hoping folks will have some ideas
I've missed.

Depending on how this thread goes, I'll make some follow-on on the
dev@hbase list for implementing changes and documentation.

Case 1: HBase 1.2-ish RIT following RS crash

Cluster had a handful of region servers fail and for whatever reason a
few regions were stuck in transition. The operator I was helping
already is used to dealing with the occasional manual recovery. Their
normal process looks like this:

1) Got to Master UI website
2) Scroll down to Regions in Transition list
3) Find a RIT in FAILED_CLOSE / FAILED_OPEN / PENDING_OPEN
4) confirm on RS logs that the RS associated in the above is now in
good health and doesn't expect to do anything with said region
5) run "assign" in the hbase shell for the region

Unfortunately, the cluster's HDFS was under duress and so listing
snapshot information was super slow. This caused the Master UI website
to hang prior to displaying the RIT list.

We ended up looking at the master log file.

Case 2: HBase 2.1-ish RIT following cluster wide crash

AFAICT cluster had experienced a failure of all RS and masters. Upon
coming back up Master was left with ~10% of ~10K regions in a state of
PENDING_OPEN or OPENING all with a RS that had no idea it was involved
with those regions. I'm pretty sure this is a bug;  I'm still triaging
it and I don't think it's relevant to the current question.

Once I confirmed the given RS was not currently doing anything for any
of those regions I figured I'd use HBCK2 to run an assigns to get
things fixed. However, since there were like 900 RITs, the Master UI
was unusable for getting a complete list. Also with that many all in
the same state I want to be able to automate running against each of
them.

I ended up greping the master log file and pulling out the WARN
messages about RIT to tease out the list of regions, then passed those
to hbck2.



Am I missing some obvious place where I can use a CLI tool to get a
list of RIT? I don't see anything in the ref guide. I looked through
the help of HBCK 1 and the shell and couldn't find anything.

I think I can use Admin.getClusterStatus() and getClusterMetrics() to
get this info from the Java API. That means there's some way to get it
in the hbase shell, but it'll probably be ugly. If there's not already
an easier way I'll want to wrap that so it's a simple command.

[ANNOUNCE] Apache HBase 1.2.9 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.2.9

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.2.9 is the ninth maintenance release in the HBase 1.2 line,
 continuing on the theme of bringing a stable, reliable database to
 the Hadoop and NoSQL communities. This release includes about a half
 dozen bug fixes done in the month since 1.2.8.

All users of previous 1.2.z releases are encouraged to upgrade to either
this release or the latest in our stable release line, which is currently
1.4.8. Releases in the 1.2.z line are expected to stop in late spring 2019.

Critical fixes include:

 * HBASE-21347 Backport HBASE-21200 "Memstore flush doesn't finish
   because of seekToPreviousRow() in memstore scanner." to
   branch-1
 * HBASE-21357 RS should abort if OOM in Reader thread
 * HBASE-20604 ProtobufLogReader#readNext can incorrectly loop to the
   same position in the stream until the the WAL is rolled

The full list of fixes included in this release is available at:

 https://s.apache.org/hbase-1.2.9-jira-release-notes

 and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/hbase-1.2.9

The relevant checksums files are available at:

https://www.apache.org/dist/hbase/hbase-1.2.9/hbase-1.2.9-src.tar.gz.sha512
https://www.apache.org/dist/hbase/hbase-1.2.9/hbase-1.2.9-bin.tar.gz.sha512

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/hbase-1.2.9/hbase-1.2.9-src.tar.gz.asc
https://www.apache.org/dist/hbase/hbase-1.2.9/hbase-1.2.9-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at:
d...@hbase.apache.org.

Cheers,
The HBase Dev Team

[ANNOUNCE] Apache HBase 1.2.8 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.2.8.

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.2.8 is the eighth maintenance release in the HBase 1.2 line,
continuing on the theme of bringing a stable, reliable database to
the Hadoop and NoSQL communities. This release includes about a half
dozen bug fixes done in the month since 1.2.7.

All users of previous 1.2.z releases are encouraged to upgrade to either
this release or the latest in our stable release line, which is currently
1.4.8. Releases in the 1.2.z line are expected to stop in late spring 2019.

Critical fixes include:

* HBASE-21158 Empty qualifier cell should not be returned if it does not
  match QualifierFilter
* HBASE-21228 Memory leak since AbstractFSWAL caches Thread object and
  never clean later

This release candidate contains the following incompatible changes,
details in the release notes for the specific issue:

* HBASE-21158 Empty qualifier cell should not be returned if it does not
  match QualifierFilter

The full list of fixes included in this release is available at:
https://s.apache.org/hbase-1.2.8-jira-release-notes
and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/hbase-1.2.8

The relevant checksums files are available at:

https://www.apache.org/dist/hbase/hbase-1.2.8/hbase-1.2.8-src.tar.gz.sha512
https://www.apache.org/dist/hbase/hbase-1.2.8/hbase-1.2.8-bin.tar.gz.sha512

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/hbase-1.2.8/hbase-1.2.8-src.tar.gz.asc
https://www.apache.org/dist/hbase/hbase-1.2.8/hbase-1.2.8-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at:
d...@hbase.apache.org.

Cheers,
The HBase Dev Team

[ANNOUNCE] New Committer: Balazs Meszaros

[Sean Busbey]

On behalf of the HBase PMC, I'm pleased to announce that Balazs
Meszaros has accepted our invitation to become an HBase committer.

Thanks for all your hard work Balazs; we look forward to more contributions!

Please join me in extending congratulations to Balazs!

Re: HBase REST Client class appears not to be Thread Safe

[Sean Busbey]

Yep! Raise a JIRA describing the issue.

If you're up for fixing it leave a comment stating so and one of the JIRA
admins will make it so you can assign the ticket to yourself.

We fix things in the most recent branch impacted first; for this it'd be
master.

Thanks for pointing this issue out!



On Tue, Sep 25, 2018, 01:57 Jon Poulton  wrote:

> Hi there,
> Not that hard to make it thread safe to be honest. And given the older
> version of the client was thread safe, "principal of least surprise"
> suggests that it continue to be so. Apache Http Components is a nice
> library that is well understood, and it looks like its being used correctly
> elsewhere in the class. The instance variables are only used in a couple of
> places, and can be replaced with variables at the method level, which seems
> to have been done correctly in most methods. You could then explicitly
> state in the Javadoc that the client is intended to be thread safe.
>
> Would I raise a raise a ticket with your issue tracking system first? And
> which branch would this be fixed in? Apologies, I'm not really a regular
> open source contributor.
>
> Jon
>
> On Tue, 25 Sep 2018 at 04:50, Stack  wrote:
>
> > Yeah, looks like it is not written thread-safe.
> >
> > As you suggest, we should note that at the least. You think it possible
> to
> > make it thread-safe Jon? Is it onerous running multiple clients or a pool
> > of clients?
> >
> > Thanks,
> > M
> >
> > On Mon, Sep 24, 2018 at 11:04 AM Jon Poulton  wrote:
> >
> > > I was looking at moving up to the 2.1 version of the HBase REST API and
> > was
> > > poking around the code base wondering how I set headers on a per
> request
> > > basis, when I noticed the the "Client" class at:
> > >
> > >
> >
> https://github.com/apache/hbase/blob/branch-2.1/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/client/Client.java
> > > appears to have HttpGet and HttpResponse instance variables (line 68
> and
> > > 69) that are not synchronised, nor do they have any kind of associated
> > > lock. The thread safety of the Client is undocumented, but usually
> Client
> > > classes can assumed to be thread safe.
> > >
> > > These instance variables are used further down in the class in
> > > unsynchronized blocks, meaning that were more than one thread to access
> > > methods that accessed the variables there is a potential race
> condition.
> > I
> > > was wondering if this is a known issue, a deliberate choice for some
> > > reason, or if the class is not supposed to be used in a multithreaded
> > > manner, and is intended to be used only within ThreadLocal?
> > >
> > > I checked in the master branch of the project and the issue appears to
> be
> > > present there also.
> > >
> > > Thanks
> > >
> > > Jon
> > >
> >
>

[ANNOUNCE] Apache HBase 1.2.7 is now available for download

[Sean Busbey]

The HBase team is happy to announce the immediate availability of Apache
HBase 1.2.7.

Apache HBase is an open-source, distributed, versioned, non-relational
database. Apache HBase gives you low latency random access to billions of
rows with millions of columns atop non-specialized hardware. To learn more
about HBase, see https://hbase.apache.org/.

HBase 1.2.7 is the latest maintenance release in the HBase 1.2 line,
continuing on the theme of bringing a stable, reliable database to the
Hadoop and NoSQL communities. This release includes over 250
bug fixes done in the 15 months since 1.2.6.

All users of previous 1.2.z releases are encouraged to upgrade to either
this release or the latest in our stable release line, which is currently
1.4.7. Releases in the 1.2.z line are expected to stop in late spring 2019.

Critical fixes include:

* HBASE-18036 Data locality is not maintained after cluster restart
* HBASE-18233 We shouldn't wait for readlock in doMiniBatchMutation in
  case of deadlock
* HBASE-9393  Region Server fails to properly close socket resulting in
  many CLOSE_WAIT to Data Nodes
* HBASE-19924 RPC throttling does not work for multi() with request
  count rater.
* HBASE-18192 Replication drops recovered queues on Region Server
  shutdown
* HBASE-18282 ReplicationLogCleaner can delete WALs not yet replicated
  in case of a KeeperException
* HBASE-19796 ReplicationSynUp tool is not replicating data if a WAL is
  moved to splitting directory
* HBASE-17648 HBase table-level synchronization fails between two
  secured(kerberized) clusters
* HBASE-18137 Replication gets stuck for empty WALs
* HBASE-18577 shaded client should not include non-relocated third party
  dependencies
* HBASE-19900 Region level exceptions destroy the result of batch client
  operations
* HBASE-21007 Memory leak in HBase REST server

This release contains the following incompatible changes, details in the
release notes for the specific issue:

* HBASE-20884 Replace usage of our Base64 implementation with java's
* HBASE-18577 shaded client should not include non-relocated third party
  dependencies
* HBASE-18142 delete in HBase shell should not delete previous versions
  of a cell
* HBASE-18731 some protected methods of QuotaSettings have been marked
  IA.Private and deprecated
* HBASE-16459 HBase shell no longer recognizes the --format option

The full list of fixes included in this release is available at:

https://s.apache.org/hbase-1.2.7-jira-release-notes
and in the CHANGES.txt file included in the distribution.

Download through an ASF mirror near you:

https://www.apache.org/dyn/closer.lua/hbase/1.2.7

The relevant checksums files are available at:

https://www.apache.org/dist/hbase/1.2.7/hbase-1.2.7-src.tar.gz.sha512
https://www.apache.org/dist/hbase/1.2.7/hbase-1.2.7-bin.tar.gz.sha512

Project member signature keys can be found at

https://www.apache.org/dist/hbase/KEYS

PGP signatures are available at:

https://www.apache.org/dist/hbase/1.2.7/hbase-1.2.7-src.tar.gz.asc
https://www.apache.org/dist/hbase/1.2.7/hbase-1.2.7-bin.tar.gz.asc

For instructions on verifying ASF release downloads, please see

https://www.apache.org/dyn/closer.cgi#verify

Question, comments, and problems are always welcome at:
d...@hbase.apache.org.

Cheers,
The HBase Dev Team

Re: build hbase2.1 without htrace-core-3.1.0-incubating.jar

[Sean Busbey]

That does not look like an Apache HBase version number on
hbase-server.  陈叶超  even said "HBase 2.1" in the original email.




On Mon, Aug 20, 2018 at 2:17 PM, Ted Yu  wrote:
> Looking at the dependency tree output, I see the following:
>
> [INFO] org.apache.hbase:hbase-server:jar:2.0.0.3.0.0.0-SNAPSHOT
> ...
> [INFO] +- org.apache.htrace:htrace-core:jar:3.2.0-incubating:compile
>
> FYI
>
> On Mon, Aug 20, 2018 at 8:10 AM Sean Busbey  wrote:
>
>> neither Hadoop 3.1 nor HBase 2.1 use that version of HTrace. what are
>> you trying to do?
>>
>> On Sun, Aug 19, 2018 at 11:24 PM, 陈叶超 
>> wrote:
>> > hi:
>> >
>> >
>> > I build hbase 2.1 with hadoop 3.1,and will lost
>> htrace-core-3.1.0-incubating.jar in \lib\client-facing-thirdparty\
>> >
>> > but i found in the apache site download tar.gz ,
>> >
>> > my build command is : "mvn  -P build-with-jdk8,hadoop-3.0
>> -DskipTests=true clean package install  assembly:single"
>> >
>> > what am i missing ?
>> >
>> >
>> >
>> >
>> > 陈叶超 Yechao Chen
>> > 中移（苏州）软件技术有限公司|中国移动苏州研发中心 | 大数据产品部
>> > China Mobile (Suzhou) Software Technology Co., Ltd.
>> > Mobile: (+86) 18896724791
>> > Email: chenyec...@cmss.chinamobile.com
>>

Re: build hbase2.1 without htrace-core-3.1.0-incubating.jar

[Sean Busbey]

neither Hadoop 3.1 nor HBase 2.1 use that version of HTrace. what are
you trying to do?

On Sun, Aug 19, 2018 at 11:24 PM, 陈叶超  wrote:
> hi:
>
>
> I build hbase 2.1 with hadoop 3.1,and will lost 
> htrace-core-3.1.0-incubating.jar in \lib\client-facing-thirdparty\
>
> but i found in the apache site download tar.gz ,
>
> my build command is : "mvn  -P build-with-jdk8,hadoop-3.0 -DskipTests=true 
> clean package install  assembly:single"
>
> what am i missing ?
>
>
>
>
> 陈叶超 Yechao Chen
> 中移（苏州）软件技术有限公司|中国移动苏州研发中心 | 大数据产品部
> China Mobile (Suzhou) Software Technology Co., Ltd.
> Mobile: (+86) 18896724791
> Email: chenyec...@cmss.chinamobile.com

[ANNOUNCE] Apache HBase "stable" release line is now 1.4.6+

[Sean Busbey]

Hi HBase Community!

The HBase developers would like you to know that we have updated the
"stable" release pointer to the 1.4.6 release. As future 1.4.z
releases are made the stable pointer will track them. All users of
releases prior to 1.4.6 are advised to upgrade to the new stable
release line.

The prior stable release line was 1.2.z. I'm in the process of getting
releases on that branch going again. Users of that release line are
advised that once releases start again they will continue monthly for
a period of about 6 months before the branch moves to end-of-life.

If you are currently a user of a release line older than 1.4 and
there's anything that would make your transition to the stable release
line easier, please let us know either via the mailing list or JIRAs.

Re: Ingest performance after upgrade

[Sean Busbey]

It sounds like it needs to get moved under the "what's changed" for 1.4 though.

On Tue, Jul 31, 2018 at 2:31 PM, Stack  wrote:
> Thanks for coming back to the list with your finding Austin.
>
> Pleased to say we'd called this out in "Changed Metrics" under
> "Changes of Note" for hbase-2.0.0 [1], not that anyone reads the
> manual (smile).
>
> St.Ack
>
> 1. http://hbase.apache.org/book.html#_changes_of_note
> On Mon, Jul 30, 2018 at 8:58 AM Austin Heyne  wrote:
>>
>> I ran some benchmarks comparing the two and actual performance is the
>> same, the perception came from HBASE 18469 [1] where the request/s
>> numbers were changed for 1.4.0 and 2.0.0.
>>
>> Thanks,
>> Austin
>>
>> [1] https://issues.apache.org/jira/browse/HBASE-18469
>>
>>
>> On 07/26/2018 06:41 PM, Stack wrote:
>> > On Mon, Jul 23, 2018 at 6:57 AM Austin Heyne  wrote:
>> >
>> >> Hey all, hopefully this is a simple one.
>> >>
>> >> We've migrated from HBase (on s3) 1.3.2 to 1.4.4, creating a fresh
>> >> instance with all our configuration from the previous one with few
>> >> changes. However, we're seeing ingest run ~100x slower than we saw on
>> >> 1.3.2 and the requests per second seem really low at around 3k total for
>> >> a 10 node cluster. We're not seeing any atypical errors but it does seem
>> >> we're seeing a ton of flushes, 1 about every 3 or 4 seconds but the logs
>> >> indicate the flushes are full size ~128MB. I've included our
>> >> configuration below.
>> >>
>> >>
>> > Check old logs to see what rate you used to flush at?
>> >
>> > Is it possible that you are just writing way more data now or the data
>> > character is different now? Larger values?
>> >
>> > S
>> >
>> >
>> >
>> >> Thanks for the help,
>> >> Austin
>> >>
>> >> [
>> >> {
>> >>   "classification": "hbase-site",
>> >>   "properties": {
>> >> "fs.s3.consistent.retryPeriodSeconds": "10",
>> >> "hbase.regionserver.thread.compaction.large": "3",
>> >> "fs.s3.consistent.retryPolicyType": "fixed",
>> >> "hbase.hstore.blockingStoreFiles": "1000",
>> >> "fs.s3.consistent.throwExceptionOnInconsistency": "false",
>> >> "hbase.bucketcache.size": "27000",
>> >> "hbase.ipc.server.callqueue.read.ratio": "0.25",
>> >> "hbase.bucketcache.combinedcache.enabled": "true",
>> >> "fs.s3a.threads.max": "50",
>> >> "hbase.regionserver.thread.compaction.small": "2",
>> >> "hbase.hregion.memstore.flush.size": "134217728",
>> >> "hbase.hregion.max.filesize": "21474836480",
>> >> "hbase.regionserver.regionSplitLimit": "1",
>> >> "fs.s3.consistent.metadata.tableName": "redacted",
>> >> "hbase.hstore.compaction.max": "1000",
>> >> "hbase.regionserver.global.memstore.size": "0.4",
>> >> "hbase.ipc.server.callqueue.handler.factor": "0.5",
>> >> "hbase.regionserver.logroll.period": "10",
>> >> "hbase.hregion.majorcompaction": "0",
>> >> "hbase.hstore.compactionThreshold": "1000",
>> >> "hbase.hregion.memstore.mslab.enabled": "false",
>> >> "hbase.regionserver.handler.count": "50",
>> >> "fs.s3a.connection.maximum": "100",
>> >> "hbase.hstore.flusher.count": "10",
>> >> "hbase.hstore.blockingWaitTime": "0",
>> >> "hbase.hregion.memstore.block.multiplier": "10",
>> >> "hbase.bucketcache.ioengine": "offheap"
>> >>   }
>> >> },
>> >> {
>> >>   "configurations": [
>> >> {
>> >>   "classification": "export",
>> >>   "properties": {
>> >> "HBASE_REGIONSERVER_OPTS": "\"-Dcom.sun.management.jmxremote
>> >> -Dcom.sun.management.jmxremote.authenticate\u003dfalse
>> >> -Dcom.sun.management.jmxremote.port\u003d10102
>> >> -Dcom.sun.management.jmxremote.ssl\u003dfalse -Xmx28G
>> >> -XX:MaxDirectMemorySize\u003d28G\"",
>> >> "HBASE_MASTER_OPTS": "\"-Dcom.sun.management.jmxremote
>> >> -Dcom.sun.management.jmxremote.authenticate\u003dfalse
>> >> -Dcom.sun.management.jmxremote.port\u003d10101
>> >> -Dcom.sun.management.jmxremote.ssl\u003dfalse -Xmx28G
>> >> -XX:MaxDirectMemorySize\u003d28G\""
>> >>   }
>> >> }
>> >>   ],
>> >>   "classification": "hbase-env",
>> >>   "properties": {
>> >>
>> >>   }
>> >> },
>> >> {
>> >>   "classification": "hbase-metrics",
>> >>   "properties": {
>> >> "rpc.period": "60",
>> >> "hbase.period": "60",
>> >> "rpc.class":
>> >> "org.apache.hadoop.metrics.spi.NullContextWithUpdateThread",
>> >> "hbase.class":
>> >> "org.apache.hadoop.metrics.spi.NullContextWithUpdateThread",
>> >> "jvm.class":
>> >> "org.apache.hadoop.metrics.spi.NullContextWithUpdateThread",
>> >> "jvm.period": "60"
>> >>   }
>> >> },
>> >> {
>> >>   "classification": "emrfs-site",
>> >>   "properties": {
>> >> "fs.s3.consistent": "true"
>> >>   }
>> >> }
>> >> ]
>> >>
>> >> --
>> >> 

Re: Query for OldWals and use of WAl for Hbase indexer

[Sean Busbey]

On Wed, Jul 11, 2018 at 12:49 PM, Manjeet Singh
 wrote:
> Thanks Sean for your reply
>
> I still have some question un answered like
> Q1: How Hbase syncronized with Hbase indexer.

The Lily Indexer for HBase (sometimes referred to as the "Lily HBase
Indexer") is an independent project, linked to in my previous email.
They would be best positioned to answer questions you have about how
it works. I suggest you talk with that project.

My understanding is that the Lily Indexer leverages the HBase
replication system to get observe edits as they come into the system
and then applies a corresponding change to the index it maintains.
That means it has all the same configurable options and warnings that
come with any HBase replication set up:

http://hbase.apache.org/1.2/book.html#_cluster_replication


> Q2 What optimization I can apply.

You need to dig into the logs to find out if there is a problem
talking to the indexers or if they are just lagging. If they're
lagging, then I believe you can add more indexer nodes to scale up
effective throughput. The Lily Indexer for HBase project would be
better suited to answer that though.

> Q3 As it's clear from my stats, data in OldWals is quite huge so it's not
> getting clear my HMaster., how can I improve my HDFS space issue due to
> this?

The only way to safely decrease the size of the retained wals is to
make it so they are no longer needed. That means either getting the
Lily Indexer for HBase to catch up or reseting things and using a
batch indexing method to fill in the gap.

Re: Query for OldWals and use of WAl for Hbase indexer

[Sean Busbey]

On Wed, Jul 11, 2018 at 7:19 PM, Manjeet Singh
 wrote:
> I have one more question
>
> If solr is having its own data mean its maintaining data in their shards
> and hbase is maintaining in data folder... Why still oldWals need?


Once the HBase replication system is active and there's a peer (even a
disabled peer) it starts retaining WALs until they have been
successfully acknowledged by all peer clusters. Since the Lily Indexer
for HBase makes use of the replication system to get an stream of
edits, the WALs will stick around until the indexer has said it saw
the edits in them.

Re: Query for OldWals and use of WAl for Hbase indexer

[Sean Busbey]

Presuming you're using the Lily indexer[1], yes it relies on hbase's
built in cross-cluster replication.

The replication system stores WALs until it can successfully send them
for replication. If you look in ZK you should be able to see which
regionserver(s) are waiting to send those WALs over. The easiest way
to do this is probably to look at the "zk dump" web page on the
Master's web ui[2].

Once you have the particular region server(s), take a look at their
logs for messages about difficulty sending edits to the replication
peer you have set up for the destination solr collection.

If you remove the WALs then the solr collection will have a hole in
it. Depending on how far behind you are, it might be quicker to 1)
remove the replication peer, 2) wait for old wals to clear, 3)
reenable replication, 4) use a batch indexing tool to index data
already in the table.

[1]:

http://ngdata.github.io/hbase-indexer/

[2]:

The specifics will vary depending on your installation, but the page
is essentially at a URL like
https://active-master-host.example.com:22002/zk.jsp

the link is on the master UI landing page, near the bottom, in the
description of the "ZooKeeper Quorum" row. it's the end of "Addresses
of all registered ZK servers. For more, see zk dump."

On Wed, Jul 11, 2018 at 10:16 AM, Manjeet Singh
 wrote:
> Hi All
>
> I have a query regarding Hbase replication and OldWals
>
> Hbase version 1.2.1
>
> To enable Hbase indexing we use below command on table
>
> alter '', {NAME => 'CF1', REPLICATION_SCOPE => 1}
>
> By Doing this actually replication get enabled as hbase-indexer required
> it, as per my understanding indexer use hbase WAL (Please correct me if I
> am wrong).
>
> so question is How Hbase syncronize with Solr Indexer? What is the role of
> replication? what optimization we can apply in order to reduce data size?
>
>
> I can see that our OldWals are getting filled , if Hmaster it self taking
> care why it's reached to 7.2 TB? what if I delete it, does it impact solr
> indexing?
>
> 7.2 K   21.5 K  /hbase/.hbase-snapshot
> 0   0   /hbase/.tmp
> 0   0   /hbase/MasterProcWALs
> 18.3 G  60.2 G  /hbase/WALs
> 28.7 G  86.1 G  /hbase/archive
> 0   0   /hbase/corrupt
> 1.7 T   5.2 T   /hbase/data
> 42  126 /hbase/hbase.id
> 7   21  /hbase/hbase.version
> 7.2 T   21.6 T  /hbase/oldWALs
>
>
>
>
> Thanks
> Manjeet Singh

Re: [question] what's the Hbase-spark module different with other two spark on Hbase

[Sean Busbey]

The hbase-spark module in the HBase project (which hasn't yet made it
into a release) is FWICT the eventual replacement for both the
Cloudera Labs SparkOnHBase and the Hortonworks SHC.

The code in the hbase-spark module started as an update of the
SparkOnHBase code and then quickly expanded via contributions from the
SHC folks to incorporate the features they wanted to provide. So while
it is safe to say the current code "comes from" both of those efforts,
it no longer looks like either of them.

On Wed, Jul 11, 2018 at 6:23 AM, nurseryboy  wrote:
> Dear All
>
>  I saw there is one Hbase-spark module in Hbase code and saw there is one
> jira for this: https://issues.apache.org/jira/browse/HBASE-13992
> In this jira it's told the Hbase-spark module code initially from
> https://github.com/cloudera-labs/SparkOnHBase
> And in anther discuss list it's mentioned SHC shared all the code to Hbase (
> discussion link is:
> https://lists.apache.org/thread.html/fd74ef9b9da77abf794664f06ea19c839fb3d543647fb29115081683@%3Cdev.hbase.apache.org%3E
> SHC link is: https://github.com/hortonworks-spark/shc)
>
> I download the code Hbase-spark, SHC and cloudera-las SparkOnHbase, then
> found the code is not same.
>
> So I am little confusion: what's the Hbase-spark relationship with SHC and
> cloudera-las SparkOnHbase?
> inherit from them or just get the idea from them ?
>
> thanks for community can help me clear about this confusion.  thanks
>
> Regards
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html