Re: High CPU Utilization by meta region
On Mon, Nov 28, 2016 at 10:25 AM, Timothy Brownwrote: > Responses inlined. > > ... > > > > > > What is the difference when you compare servers? More requests? More > i/o? > > Thread dump the metadata server and let us see a link in here? (What you > > attached below is cut-off... just as it is getting to the good part). > > > > > > There are more requests to the server containing meta. The network in > bytes are greater for the meta regionserver than the others but the network > out bytes are less. > > Here's a dropbox link to the output https://dl.dropboxusercontent.com/u/ > 54494127/thread_dump.txt. I apologize for the cliffhanger. > > The in bytes are < the out bytes on the hbase:meta server? Or compared to other servers? Queries are usually smaller than response and in hbase:meta case, I'd think that we'd be mostly querying/reading with out much bigger than in. Anything else running on this machine besides Master? If you turn on RPC-level TRACE logging for a minute or so, anything about the client addresses that seems interesting? Looking at the thread dump (thanks), you have 1k handlers running? Thread 1037 (B.defaultRpcServer.handler=999,queue=99,port=60020): They are all idle in this thread dump (Same for the readers). I've found that having handlers == # of cpus seems to do the best when mostly a random read workload If lots of writes, good to have a few extras in case one gets occupied but 1k is a little OTT. Any particular reason for this many handlers? Would suggest trying way less. Might help w/ CPU. 1k is a lot. GCG1? (See HBASE-17072 CPU usage starts to climb up to 90-100% when using G1GC; purge ThreadLocal usage) > > > > > > Here's some more info about our cluster: > > > HBase version 1.2 > > > > > > > Which 1.2? > > > > 1.2.0 which is bundled with CDH 5.8.0 > > > > > > > > Number of regions: 72 > > > Number of tables: 97 > > > > > > > On whole cluster? (Can't have more tables than regions...) > > > > > > An error on my part, I meant to put 72 region servers. > > > > > > > Approx. requests per second to meta region server: 3k > > > > That is not much. If all cached should be able to do way more than that. > > > > Can you see who is hitting he meta region most? (Enable rpc-level TRACE > > logging on the server hosting meta for a minute or so and see where the > > requests are coming in from). > > > > What is your cache hit rate? Can you get it higher? > > > > Cache hit rate is above 99%. We see very little disk reads. > > > > Is there much writing going on against meta? Or is cluster stable regards > > region movement/creation? > > > > Writing is very infrequent. The cluster is stable with regards to region > movement and creation. > > > > > > > > Approx. requests per second to entire HBase cluster: 90k > > > > > > Additional info: > > > > > > > > > From Storefile Metrics: > > > Stores Num: 1 > > > Storefiles: 1 > > > Storefile Size: 30m > > > Uncompressed Storefile Size: 30m > Super small. St.Ack > > > Index Size: 459k > > > > > > > > This from meta table? That is very small. > > > > Yes this is from the meta table. > > > > > > > > > > I/O for the region server with only meta on it: > > > 48M bytes in > > > > > > > > > Whats all the writing about? > > > > I'm not sure. According to the AWS dashboard there are no disk writes at > that time. > > > > > > > > 5.9B bytes out > > > > > > > > This is disk or network? If network, is that 5.9 bytes? > > > > This is network and thats 5.9 billion byes. (I'm using the AWS dashboard > for this) > > > > Thanks Tim, > > S > > > > > > > > > I used the debug dump on the region server's UI but it was too large > > > for paste bin so here's a portion of it: http://pastebin.com/nkYhEceE > > > > > > > > > Thanks for the help, > > > > > > Tim > > > > > >
Re: [ANNOUNCE] New HBase committer Phil Yang
Thanks all. It is a great honor to be a committer and I will try my best to contribute more to HBase. Thanks, Phil 2016-11-30 1:07 GMT+08:00 Stack: > Welcome Phil! > St.Ack > > On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang wrote: > > > On behalf of the Apache HBase PMC, I am pleased to announce that Phil > Yang > > has accepted the PMC's invitation to become a committer on the project. > We > > appreciate all of Phil's generous contributions thus far and look forward > > to his continued involvement. > > > > Congratulations and welcome, Phil! > > >
Re: [ANNOUNCE] New HBase committer Phil Yang
Congratulations!! 2016-11-30 8:32 GMT+08:00 Stephen Jiang: > Congratulations, Phil! > > On Tue, Nov 29, 2016 at 2:42 PM, Andrew Purtell wrote: > >> Congratulations and welcome, Phil! >> >> >> On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang wrote: >> >> > On behalf of the Apache HBase PMC, I am pleased to announce that Phil >> Yang >> > has accepted the PMC's invitation to become a committer on the project. >> We >> > appreciate all of Phil's generous contributions thus far and look forward >> > to his continued involvement. >> > >> > Congratulations and welcome, Phil! >> > >> >> >> >> -- >> Best regards, >> >>- Andy >> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein >> (via Tom White) >>
Re: [ANNOUNCE] New HBase committer Phil Yang
Congratulations, Phil! On Tue, Nov 29, 2016 at 2:42 PM, Andrew Purtellwrote: > Congratulations and welcome, Phil! > > > On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhang wrote: > > > On behalf of the Apache HBase PMC, I am pleased to announce that Phil > Yang > > has accepted the PMC's invitation to become a committer on the project. > We > > appreciate all of Phil's generous contributions thus far and look forward > > to his continued involvement. > > > > Congratulations and welcome, Phil! > > > > > > -- > Best regards, > >- Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White) >
Re: [ANNOUNCE] New HBase committer Phil Yang
Congratulations and welcome, Phil! On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhangwrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang > has accepted the PMC's invitation to become a committer on the project. We > appreciate all of Phil's generous contributions thus far and look forward > to his continued involvement. > > Congratulations and welcome, Phil! > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Rolling upgrade with coprocessors
@Albert, Just a heads up -- in my experience, when replacing a coprocessor jar stored in HDFS (using an identical jar name), the new jar will not be loaded (even if you disable/enable the table). Tracked here: https://issues.apache.org/jira/browse/HBASE-9046 -- Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message. On Tue, Nov 29, 2016 at 1:27 PM, Albert Shauwrote: > Thanks very much for the help. I'm currently using a RegionObserver (to > implement readless increments) on a subset of my tables, and placing the > jar on hdfs. I will try what Clay mentioned and see if it works. > Best,Albert > > On Wednesday, November 23, 2016 4:42 AM, Clay Baenziger (BLOOMBERG/ > 731 LEX) wrote: > > > Hi Albert, > > The way I've seen it working -- if you need to leave the table online -- > is specifying the coprocessor in the hbase-site.xml of each region server > and deploying the class to the table without a path (just the classname). > Then, ensure your hbase-site.xml has the correct version for the region > server running on that machine. > > Esther and I briefly talked on this type of operation in > http://www.slideshare.net/EstherKundin/coprocessors-uses-abuses-solutions- > presented-at-hbasecon-east-2016 or https://www.youtube.com/watch? > v=dYeJd9AAVlw > > -Clay > From: user@hbase.apache.org At: 11/18/16 19:33:54To: albert.s...@yahoo.com, > user@hbase.apache.org > Subject: Re: Rolling upgrade with coprocessors > > Can you tell us the version of hbase you are using and the new version > which you plan to upgrade to ? > > A bit more detail on your coprocessor would also help narrow the scope of > search. > > Cheers > > On Fri, Nov 18, 2016 at 4:28 PM, Albert Shau > > wrote: > > > Hi all, > > I'm using coprocessors with my tables and am wondering how I would > perform > > an HBase rolling upgrade, since it seems like there are no compatibility > > guarantees for the coprocessor APIs. I'm guessing I would have to disable > > the table, alter the table to use a coprocessor compatible with the next > > HBase version, perform the rolling upgrade, then enable the table? Is > there > > a way to keep the table enabled for this? > > Thanks,Albert > > > > > >
Re: Rolling upgrade with coprocessors
Thanks very much for the help. I'm currently using a RegionObserver (to implement readless increments) on a subset of my tables, and placing the jar on hdfs. I will try what Clay mentioned and see if it works. Best,Albert On Wednesday, November 23, 2016 4:42 AM, Clay Baenziger (BLOOMBERG/ 731 LEX)wrote: Hi Albert, The way I've seen it working -- if you need to leave the table online -- is specifying the coprocessor in the hbase-site.xml of each region server and deploying the class to the table without a path (just the classname). Then, ensure your hbase-site.xml has the correct version for the region server running on that machine. Esther and I briefly talked on this type of operation in http://www.slideshare.net/EstherKundin/coprocessors-uses-abuses-solutions-presented-at-hbasecon-east-2016 or https://www.youtube.com/watch?v=dYeJd9AAVlw -Clay From: user@hbase.apache.org At: 11/18/16 19:33:54To: albert.s...@yahoo.com, user@hbase.apache.org Subject: Re: Rolling upgrade with coprocessors Can you tell us the version of hbase you are using and the new version which you plan to upgrade to ? A bit more detail on your coprocessor would also help narrow the scope of search. Cheers On Fri, Nov 18, 2016 at 4:28 PM, Albert Shau wrote: > Hi all, > I'm using coprocessors with my tables and am wondering how I would perform > an HBase rolling upgrade, since it seems like there are no compatibility > guarantees for the coprocessor APIs. I'm guessing I would have to disable > the table, alter the table to use a coprocessor compatible with the next > HBase version, perform the rolling upgrade, then enable the table? Is there > a way to keep the table enabled for this? > Thanks,Albert
Re: [ANNOUNCE] New HBase committer Lijin Bin
Congratulations and welcome, Lijin! -- Cloudera, Inc. On Tue, Nov 29, 2016 at 5:32 AM, Yu Liwrote: > Congratulations and welcome Lijin! > > Best Regards, > Yu > > On 29 November 2016 at 17:48, Duo Zhang wrote: > > > On behalf of the Apache HBase PMC, I am pleased to announce that Lijin > > Bin(binlijin) has accepted the PMC's invitation to become a committer on > > the project. We appreciate all of Lijin's generous contributions thus far > > and look forward to his continued involvement. > > > > Congratulations and welcome, Lijin! > > >
Re: [ANNOUNCE] New HBase committer Phil Yang
Welcome Phil! St.Ack On Tue, Nov 29, 2016 at 1:49 AM, Duo Zhangwrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang > has accepted the PMC's invitation to become a committer on the project. We > appreciate all of Phil's generous contributions thus far and look forward > to his continued involvement. > > Congratulations and welcome, Phil! >
Re: [ANNOUNCE] New HBase committer Lijin Bin
Welcome Lijin! On Tue, Nov 29, 2016 at 1:48 AM, Duo Zhangwrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Lijin > Bin(binlijin) has accepted the PMC's invitation to become a committer on > the project. We appreciate all of Lijin's generous contributions thus far > and look forward to his continued involvement. > > Congratulations and welcome, Lijin! >
Multi-homed hosts with Hbase
Hi, I am using multi-homed kerberized hosts with hbase 0.98. The /etc/hosts files are uniformly containing FQDNs for cluster nodes. However, region server fails to register with master server with the error: 2016-11-17 15:05:47,700 WARN [regionserver54723] regionserver.HRegionServer: error telling master we are up com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for INFACSDEVKERB/dvterhpdw02.devadmin.nbsdev.co...@devadmin.nbsdev.co.uk to INFACSDEVKERB/dvterhpdw03.devadmin.nbsdev.co...@devadmin.nbsdev.co.uk at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1679) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1720) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277) at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2190) Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) The error indicates some DNS lookup problem. So, my question is do I need to enable some config in Hbase to allow multi-homed hosts? Also, I assume that if /etc/hosts is setup correctly, then entries in DNS server should not matter given the order of lookup is /etc/hosts and then DNS in my setup. I looked at some documentation and bugs (https://issues.apache.org/jira/browse/HBASE-12954 or https://issues.apache.org/jira/browse/HADOOP-8198) which seem to indicate multi-home support is not straight forward? Zookeeper is not managed by Hbase. As a next step, I will dump zookeeper data from hbase shell to see what hostnames/ fqdns are being reported by master and region server. Any other suggestions can I try? Thanks, Sumit
Re: deleteAll not working for regions with startkey
Did you copy the start key verbatim ? Please take a look at ./hbase-shell/src/main/ruby/shell.rb to see example of proper escaping. Cheers On Tuesday, November 29, 2016 1:58 AM, Ravi Kumar Bommadawrote: Hi, I'm trying to delete a row from 'hbase:meta' by providing region name as below: deleteall 'hbase:meta', 'EDGE,acc_sr\x00344739220_768a74f5-cf1b-11e5-9e5b-005056a93f00\x00sr\x00-->\x00\x00\x7F\xFF\xFE\xB4\x8A\xABto2015-0128-T-0573,1467107859752.c596a0695736e4b18babdb8fe458d7bd.' I'm getting output as ERROR: Row Not Found, but I could see the row in web ui and dump of meta table. When I'm trying to delete one region without startkey, I could delete that row, for example: deleteall 'hbase:meta', 'EDGE,,1455098754904.39683a213541a2a45cc9509c82c6c5ba.' This delete is happening, but deleting the region with startkey is not happening. Please provide me clues in case of any. Thank's Regard's Ravi Kumar B Mob: +91 9591144511
Re: [ANNOUNCE] New HBase committer Phil Yang
Congratulations, Phil. On Tuesday, November 29, 2016 1:49 AM, Duo Zhangwrote: On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang has accepted the PMC's invitation to become a committer on the project. We appreciate all of Phil's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Phil!
Re: [ANNOUNCE] New HBase committer Lijin Bin
Congratulations, Lijin. On Tuesday, November 29, 2016 3:01 AM, Anoop Johnwrote: Congrats and welcome Binlijin. -Anoop- On Tue, Nov 29, 2016 at 3:18 PM, Duo Zhang wrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Lijin > Bin(binlijin) has accepted the PMC's invitation to become a committer on > the project. We appreciate all of Lijin's generous contributions thus far > and look forward to his continued involvement. > > Congratulations and welcome, Lijin!
Re: [ANNOUNCE] New HBase committer Lijin Bin
Congrats and welcome Binlijin. -Anoop- On Tue, Nov 29, 2016 at 3:18 PM, Duo Zhangwrote: > On behalf of the Apache HBase PMC, I am pleased to announce that Lijin > Bin(binlijin) has accepted the PMC's invitation to become a committer on > the project. We appreciate all of Lijin's generous contributions thus far > and look forward to his continued involvement. > > Congratulations and welcome, Lijin!
Downsides of having large number of versions in hbase
Hi, I am curious to understand the impact of having large number of versions in HBase. Suppose I want to maintain previous 100 versions for a row/cell. My thoughts are:- Having large number of versions means more number of HFiles More number of HFiles can increase lookup time of a rowKey. Hypothesis 1 : Region server has to check each HFile for the presence of that rowKey and then based on timestamp it will accumulate the latest version. Hypothesis 2 : Region server may not scan each HFile. Based on last creation date of HFile,as soon as it gets rowKey in the last created HFile it will not scan HFiles further. Because we are interested in latest version only and we have got in the file recently created. Want to confirm what is true among 1 and 2. Similarly, large number of versions can also degrade the performance of full scan for joins etc. Thanks -Sachin
deleteAll not working for regions with startkey
Hi, I'm trying to delete a row from 'hbase:meta' by providing region name as below: deleteall 'hbase:meta', 'EDGE,acc_sr\x00344739220_768a74f5-cf1b-11e5-9e5b-005056a93f00\x00sr\x00-->\x00\x00\x7F\xFF\xFE\xB4\x8A\xABto2015-0128-T-0573,1467107859752.c596a0695736e4b18babdb8fe458d7bd.' I'm getting output as ERROR: Row Not Found, but I could see the row in web ui and dump of meta table. When I'm trying to delete one region without startkey, I could delete that row, for example: deleteall 'hbase:meta', 'EDGE,,1455098754904.39683a213541a2a45cc9509c82c6c5ba.' This delete is happening, but deleting the region with startkey is not happening. Please provide me clues in case of any. Thank's Regard's Ravi Kumar B Mob: +91 9591144511
[ANNOUNCE] New HBase committer Phil Yang
On behalf of the Apache HBase PMC, I am pleased to announce that Phil Yang has accepted the PMC's invitation to become a committer on the project. We appreciate all of Phil's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Phil!
[ANNOUNCE] New HBase committer Lijin Bin
On behalf of the Apache HBase PMC, I am pleased to announce that Lijin Bin(binlijin) has accepted the PMC's invitation to become a committer on the project. We appreciate all of Lijin's generous contributions thus far and look forward to his continued involvement. Congratulations and welcome, Lijin!
Re: Creating HBase table with presplits
Thanks Dave for your suggestions! Will let you know if I find some approach to tackle this situation. Regards On Mon, Nov 28, 2016 at 9:05 PM, Dave Lathamwrote: > If you truly have no way to predict anything about the distribution of your > data across the row key space, then you are correct that there is no way to > presplit your regions in an effective way. Either you need to make some > starting guess, such as a small number of uniform splits, or wait until you > have some information about what the data will look like. > > Dave > > On Mon, Nov 28, 2016 at 12:42 AM, Sachin Jain > wrote: > > > Hi, > > > > I was going though pre-splitting a table article [0] and it is mentioned > > that it is generally best practice to presplit your table. But don't we > > need to know the data in advance in order to presplit it. > > > > Question: What should be the best practice when we don't know what data > is > > going to be inserted into HBase. Essentially I don't know the key range > so > > if I specify wrong splits, then either first or last split can be a hot > > region in my system. > > > > [0]: https://hbase.apache.org/book.html#rowkey.regionsplits > > > > Thanks > > -Sachin > > >