Re: High CPU Utilization by meta region

[Stack]

On Mon, Nov 28, 2016 at 10:25 AM, Timothy Brown  wrote:

> Responses inlined.
>
> ...

> > >
> > > What is the difference when you compare servers? More requests? More
> i/o?
> > Thread dump the metadata server and let us see a link in here? (What you
> > attached below is cut-off... just as it is getting to the good part).
> >
> >
> > There are more requests to the server containing meta. The network in
> bytes are greater for the meta regionserver than the others but the network
> out bytes are less.
>
> Here's a dropbox link to the output https://dl.dropboxusercontent.com/u/
> 54494127/thread_dump.txt. I apologize for the cliffhanger.
>
>
The in bytes are < the out bytes on the hbase:meta server? Or compared to
other servers? Queries are usually smaller than response and in hbase:meta
case, I'd think that we'd be mostly querying/reading with out much bigger
than in.

Anything else running on this machine besides Master?

If you turn on RPC-level TRACE logging for a minute or so, anything about
the client addresses that seems interesting?

Looking at the thread dump (thanks), you have 1k handlers running?

Thread 1037 (B.defaultRpcServer.handler=999,queue=99,port=60020):

They are all idle in this thread dump (Same for the readers).

I've found that having handlers == # of cpus seems to do the best when
mostly a random read workload If lots of writes, good to have a few
extras in case one gets occupied but 1k is a little OTT. Any particular
reason for this many handlers? Would suggest trying way less. Might help w/
CPU. 1k is a lot.

GCG1? (See HBASE-17072 CPU usage starts to climb up to 90-100% when using
G1GC; purge ThreadLocal usage)


>
> >
> > > Here's some more info about our cluster:
> > > HBase version 1.2
> > >
> >
> > Which 1.2?
> >
> > 1.2.0 which is bundled with CDH 5.8.0
>
> >
> >
> > > Number of regions: 72
> > > Number of tables: 97
> > >
> >
> > On whole cluster? (Can't have more tables than regions...)
> >
> >
> > An error on my part, I meant to put 72 region servers.
>
>
> >
> > > Approx. requests per second to meta region server: 3k
> > >
>

That is not much. If all cached should be able to do way more than that.



> >
> > Can you see who is hitting he meta region most? (Enable rpc-level TRACE
> > logging on the server hosting meta for a minute or so and see where the
> > requests are coming in from).
> >
> > What is your cache hit rate? Can you get it higher?
> >
> > Cache hit rate is above 99%. We see very little disk reads.
>
>
> > Is there much writing going on against meta? Or is cluster stable regards
> > region movement/creation?
> >
> > Writing is very infrequent. The cluster is stable with regards to region
> movement and creation.
>
> >
> >
> > > Approx. requests per second to entire HBase cluster: 90k
> > >
> > > Additional info:
> > >
> > >
> > > From Storefile Metrics:
> > > Stores Num: 1
> > > Storefiles: 1
> > > Storefile Size: 30m
> > > Uncompressed Storefile Size: 30m
>

Super small.

St.Ack




> > > Index Size: 459k
> > >
> > >
> > This from meta table? That is very small.
> >
> > Yes this is from the meta table.
>
>
> >
> > >
> > > I/O for the region server with only meta on it:
> > > 48M bytes in
> > >
> >
> >
> > Whats all the writing about?
> >
> > I'm not sure. According to the AWS dashboard there are no disk writes at
> that time.
>
> >
> >
> > > 5.9B bytes out
> > >
> > >
> > This is disk or network? If network, is that 5.9 bytes?
> >
> > This is network and thats 5.9 billion byes. (I'm using the AWS dashboard
> for this)
>
>
> > Thanks Tim,
> > S
> >
> >
> >
> > > I used the debug dump on the region server's UI but it was too large
> > > for paste bin so here's a portion of it: http://pastebin.com/nkYhEceE
> > >
> > >
> > > Thanks for the help,
> > >
> > > Tim
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Phil Yang

[Phil Yang]

Thanks all.
It is a great honor to be a committer and I will try my best to contribute
more to HBase.

Thanks,
Phil


2016-11-30 1:07 GMT+08:00 Stack :

> Welcome Phil!
> St.Ack
>
> On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Phil
> Yang
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Phil's generous contributions thus far and look forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Phil!
> >
>

Re: [ANNOUNCE] New HBase committer Phil Yang

[Heng Chen]

Congratulations!!

2016-11-30 8:32 GMT+08:00 Stephen Jiang :
> Congratulations, Phil!
>
> On Tue, Nov 29, 2016 at 2:42 PM, Andrew Purtell  wrote:
>
>> Congratulations and welcome, Phil!
>>
>>
>> On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang  wrote:
>>
>> > On behalf of the Apache HBase PMC, I am pleased to announce that Phil
>> Yang
>> > has accepted the PMC's invitation to become a committer on the project.
>> We
>> > appreciate all of Phil's generous contributions thus far and look forward
>> > to his continued involvement.
>> >
>> > Congratulations and welcome, Phil!
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>- Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>

Re: [ANNOUNCE] New HBase committer Phil Yang

[Stephen Jiang]

Congratulations, Phil!

On Tue, Nov 29, 2016 at 2:42 PM, Andrew Purtell  wrote:

> Congratulations and welcome, Phil!
>
>
> On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Phil
> Yang
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Phil's generous contributions thus far and look forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Phil!
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: [ANNOUNCE] New HBase committer Phil Yang

[Andrew Purtell]

Congratulations and welcome, Phil!


On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang  wrote:

> On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang
> has accepted the PMC's invitation to become a committer on the project. We
> appreciate all of Phil's generous contributions thus far and look forward
> to his continued involvement.
>
> Congratulations and welcome, Phil!
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Rolling upgrade with coprocessors

[iain wright]

@Albert,

Just a heads up -- in my experience, when replacing a coprocessor jar
stored in HDFS (using an identical jar name), the new jar will not be
loaded (even if you disable/enable the table). Tracked here:
https://issues.apache.org/jira/browse/HBASE-9046

-- 
Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Tue, Nov 29, 2016 at 1:27 PM, Albert Shau 
wrote:

> Thanks very much for the help. I'm currently using a RegionObserver (to
> implement readless increments) on a subset of my tables, and placing the
> jar on hdfs. I will try what Clay mentioned and see if it works.
> Best,Albert
>
> On Wednesday, November 23, 2016 4:42 AM, Clay Baenziger (BLOOMBERG/
> 731 LEX)  wrote:
>
>
>  Hi Albert,
>
> The way I've seen it working -- if you need to leave the table online --
> is specifying the coprocessor in the hbase-site.xml of each region server
> and deploying the class to the table without a path (just the classname).
> Then, ensure your hbase-site.xml has the correct version for the region
> server running on that machine.
>
> Esther and I briefly talked on this type of operation in
> http://www.slideshare.net/EstherKundin/coprocessors-uses-abuses-solutions-
> presented-at-hbasecon-east-2016 or https://www.youtube.com/watch?
> v=dYeJd9AAVlw
>
> -Clay
> From: user@hbase.apache.org At: 11/18/16 19:33:54To: albert.s...@yahoo.com,
> user@hbase.apache.org
> Subject: Re: Rolling upgrade with coprocessors
>
> Can you tell us the version of hbase you are using and the new version
> which you plan to upgrade to ?
>
> A bit more detail on your coprocessor would also help narrow the scope of
> search.
>
> Cheers
>
> On Fri, Nov 18, 2016 at 4:28 PM, Albert Shau  >
> wrote:
>
> > Hi all,
> > I'm using coprocessors with my tables and am wondering how I would
> perform
> > an HBase rolling upgrade, since it seems like there are no compatibility
> > guarantees for the coprocessor APIs. I'm guessing I would have to disable
> > the table, alter the table to use a coprocessor compatible with the next
> > HBase version, perform the rolling upgrade, then enable the table? Is
> there
> > a way to keep the table enabled for this?
> > Thanks,Albert
>
>
>
>
>
>

Re: Rolling upgrade with coprocessors

[Albert Shau]

Thanks very much for the help. I'm currently using a RegionObserver (to 
implement readless increments) on a subset of my tables, and placing the jar on 
hdfs. I will try what Clay mentioned and see if it works.
Best,Albert  

On Wednesday, November 23, 2016 4:42 AM, Clay Baenziger (BLOOMBERG/ 731 
LEX)  wrote:
 

 Hi Albert,

The way I've seen it working -- if you need to leave the table online -- is 
specifying the coprocessor in the hbase-site.xml of each region server and 
deploying the class to the table without a path (just the classname). Then, 
ensure your hbase-site.xml has the correct version for the region server 
running on that machine.

Esther and I briefly talked on this type of operation in 
http://www.slideshare.net/EstherKundin/coprocessors-uses-abuses-solutions-presented-at-hbasecon-east-2016
 or https://www.youtube.com/watch?v=dYeJd9AAVlw

-Clay
From: user@hbase.apache.org At: 11/18/16 19:33:54To: albert.s...@yahoo.com, 
user@hbase.apache.org
Subject: Re: Rolling upgrade with coprocessors

Can you tell us the version of hbase you are using and the new version
which you plan to upgrade to ?

A bit more detail on your coprocessor would also help narrow the scope of
search.

Cheers

On Fri, Nov 18, 2016 at 4:28 PM, Albert Shau 
wrote:

> Hi all,
> I'm using coprocessors with my tables and am wondering how I would perform
> an HBase rolling upgrade, since it seems like there are no compatibility
> guarantees for the coprocessor APIs. I'm guessing I would have to disable
> the table, alter the table to use a coprocessor compatible with the next
> HBase version, perform the rolling upgrade, then enable the table? Is there
> a way to keep the table enabled for this?
> Thanks,Albert




   

Re: [ANNOUNCE] New HBase committer Lijin Bin

[Esteban Gutierrez]

Congratulations and welcome, Lijin!

--
Cloudera, Inc.


On Tue, Nov 29, 2016 at 5:32 AM, Yu Li  wrote:

> Congratulations and welcome Lijin!
>
> Best Regards,
> Yu
>
> On 29 November 2016 at 17:48, Duo Zhang  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
> > Bin(binlijin) has accepted the PMC's invitation to become a committer on
> > the project. We appreciate all of Lijin's generous contributions thus far
> > and look forward to his continued involvement.
> >
> > Congratulations and welcome, Lijin!
> >
>

Re: [ANNOUNCE] New HBase committer Phil Yang

[Stack]

Welcome Phil!
St.Ack

On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang  wrote:

> On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang
> has accepted the PMC's invitation to become a committer on the project. We
> appreciate all of Phil's generous contributions thus far and look forward
> to his continued involvement.
>
> Congratulations and welcome, Phil!
>

Re: [ANNOUNCE] New HBase committer Lijin Bin

[Stack]

Welcome Lijin!

On Tue, Nov 29, 2016 at 1:48 AM, Duo Zhang  wrote:

> On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
> Bin(binlijin) has accepted the PMC's invitation to become a committer on
> the project. We appreciate all of Lijin's generous contributions thus far
> and look forward to his continued involvement.
>
> Congratulations and welcome, Lijin!
>

Multi-homed hosts with Hbase

[Sumit Nigam]

Hi,

I am using multi-homed kerberized hosts with hbase 0.98.

The /etc/hosts files are uniformly containing FQDNs for cluster nodes. However, 
region server fails to register with master server with the error:

2016-11-17 15:05:47,700 WARN  [regionserver54723] regionserver.HRegionServer: 
error telling master we are up

com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup 
connection for 
INFACSDEVKERB/dvterhpdw02.devadmin.nbsdev.co...@devadmin.nbsdev.co.uk to 
INFACSDEVKERB/dvterhpdw03.devadmin.nbsdev.co...@devadmin.nbsdev.co.uk
 at 
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1679)
 at 
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1720)
 at 
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
 at 
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2190)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Server not found 
in Kerberos database (7))]
 at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
 at 
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)

The error indicates some DNS lookup problem. So, my question is do I need to 
enable some config in Hbase to allow multi-homed hosts? Also, I assume that if 
/etc/hosts is setup correctly, then entries in DNS server should not matter 
given the order of lookup is /etc/hosts and then DNS in my setup. I looked at 
some documentation and bugs (https://issues.apache.org/jira/browse/HBASE-12954 
or https://issues.apache.org/jira/browse/HADOOP-8198) which seem to indicate 
multi-home support is not straight forward?

Zookeeper is not managed by Hbase. As a next step, I will dump zookeeper data 
from hbase shell to see what hostnames/ fqdns are being reported by master and 
region server. Any other suggestions can I try?

Thanks,

Sumit

Re: deleteAll not working for regions with startkey

[Ted Yu]

Did you copy the start key verbatim ?
Please take a look at ./hbase-shell/src/main/ruby/shell.rb to see example of 
proper escaping.
Cheers 

On Tuesday, November 29, 2016 1:58 AM, Ravi Kumar Bommada 
 wrote:
 

 Hi,

I'm trying to delete a row from 'hbase:meta' by providing region name as below:

deleteall 'hbase:meta', 
'EDGE,acc_sr\x00344739220_768a74f5-cf1b-11e5-9e5b-005056a93f00\x00sr\x00-->\x00\x00\x7F\xFF\xFE\xB4\x8A\xABto2015-0128-T-0573,1467107859752.c596a0695736e4b18babdb8fe458d7bd.'

I'm getting output as ERROR: Row Not Found, but I could see the row in web ui 
and dump of meta table.

When I'm trying to delete one region without startkey, I could delete that row, 
for example:

deleteall 'hbase:meta', 'EDGE,,1455098754904.39683a213541a2a45cc9509c82c6c5ba.'

This delete is happening, but deleting the region with startkey is not 
happening.

Please provide me clues in case of any.

Thank's

Regard's

Ravi Kumar B
Mob: +91 9591144511




   

Re: [ANNOUNCE] New HBase committer Phil Yang

[Ted Yu]

Congratulations, Phil. 

On Tuesday, November 29, 2016 1:49 AM, Duo Zhang  
wrote:
 

 On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang
has accepted the PMC's invitation to become a committer on the project. We
appreciate all of Phil's generous contributions thus far and look forward
to his continued involvement.

Congratulations and welcome, Phil!


   

Re: [ANNOUNCE] New HBase committer Lijin Bin

[Ted Yu]

Congratulations, Lijin. 

On Tuesday, November 29, 2016 3:01 AM, Anoop John  
wrote:
 

 Congrats and welcome Binlijin.

-Anoop-

On Tue, Nov 29, 2016 at 3:18 PM, Duo Zhang  wrote:
> On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
> Bin(binlijin) has accepted the PMC's invitation to become a committer on
> the project. We appreciate all of Lijin's generous contributions thus far
> and look forward to his continued involvement.
>
> Congratulations and welcome, Lijin!


   

Re: [ANNOUNCE] New HBase committer Lijin Bin

[Anoop John]

Congrats and welcome Binlijin.

-Anoop-

On Tue, Nov 29, 2016 at 3:18 PM, Duo Zhang  wrote:
> On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
> Bin(binlijin) has accepted the PMC's invitation to become a committer on
> the project. We appreciate all of Lijin's generous contributions thus far
> and look forward to his continued involvement.
>
> Congratulations and welcome, Lijin!

Downsides of having large number of versions in hbase

[Sachin Jain]

Hi,

I am curious to understand the impact of having large number of versions in
HBase. Suppose I want to maintain previous 100 versions for a row/cell.

My thoughts are:-

Having large number of versions means more number of HFiles
More number of HFiles can increase lookup time of a rowKey.

  Hypothesis 1 : Region server has to check each HFile for the presence of
that rowKey and then based on timestamp it will accumulate the latest
version.

  Hypothesis 2 : Region server may not scan each HFile. Based on last
creation date of HFile,as soon as it gets rowKey in the last created HFile
it will not scan HFiles further. Because we are interested in latest
version only and we have got in the file recently created.

Want to confirm what is true among 1 and 2.

Similarly, large number of versions can also degrade the performance of
full scan for joins etc.

Thanks
-Sachin

deleteAll not working for regions with startkey

[Ravi Kumar Bommada]

Hi,

I'm trying to delete a row from 'hbase:meta' by providing region name as below:

deleteall 'hbase:meta', 
'EDGE,acc_sr\x00344739220_768a74f5-cf1b-11e5-9e5b-005056a93f00\x00sr\x00-->\x00\x00\x7F\xFF\xFE\xB4\x8A\xABto2015-0128-T-0573,1467107859752.c596a0695736e4b18babdb8fe458d7bd.'

I'm getting output as ERROR: Row Not Found, but I could see the row in web ui 
and dump of meta table.

When I'm trying to delete one region without startkey, I could delete that row, 
for example:

deleteall 'hbase:meta', 'EDGE,,1455098754904.39683a213541a2a45cc9509c82c6c5ba.'

This delete is happening, but deleting the region with startkey is not 
happening.

Please provide me clues in case of any.

Thank's

Regard's

Ravi Kumar B
Mob: +91 9591144511

[ANNOUNCE] New HBase committer Phil Yang

[Duo Zhang]

On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang
has accepted the PMC's invitation to become a committer on the project. We
appreciate all of Phil's generous contributions thus far and look forward
to his continued involvement.

Congratulations and welcome, Phil!

[ANNOUNCE] New HBase committer Lijin Bin

[Duo Zhang]

On behalf of the Apache HBase PMC, I am pleased to announce that Lijin
Bin(binlijin) has accepted the PMC's invitation to become a committer on
the project. We appreciate all of Lijin's generous contributions thus far
and look forward to his continued involvement.

Congratulations and welcome, Lijin!

Re: Creating HBase table with presplits

[Sachin Jain]

Thanks Dave for your suggestions!
Will let you know if I find some approach to tackle this situation.

Regards

On Mon, Nov 28, 2016 at 9:05 PM, Dave Latham  wrote:

> If you truly have no way to predict anything about the distribution of your
> data across the row key space, then you are correct that there is no way to
> presplit your regions in an effective way.  Either you need to make some
> starting guess, such as a small number of uniform splits, or wait until you
> have some information about what the data will look like.
>
> Dave
>
> On Mon, Nov 28, 2016 at 12:42 AM, Sachin Jain 
> wrote:
>
> > Hi,
> >
> > I was going though pre-splitting a table article [0] and it is mentioned
> > that it is generally best practice to presplit your table. But don't we
> > need to know the data in advance in order to presplit it.
> >
> > Question: What should be the best practice when we don't know what data
> is
> > going to be inserted into HBase. Essentially I don't know the key range
> so
> > if I specify wrong splits, then either first or last split can be a hot
> > region in my system.
> >
> > [0]: https://hbase.apache.org/book.html#rowkey.regionsplits
> >
> > Thanks
> > -Sachin
> >
>