Re: Extremely high CPU usage after upgrading to Hbase 1.4.4
Hi Srinidhi, We are also facing a similar problem in HBase-1.4.8. We're curious to know what you did after this to resolve the issue. Did you revert back to a previous version? I've seen a Jira regarding this issue. ( https://issues.apache.org/jira/browse/HBASE-21620) @Ted Yu any update on this? On Tue, Sep 11, 2018 at 4:11 AM Srinidhi Muppalla wrote: > It is during a period when the number of client operations was relatively > low. It wasn’t zero, but it was definitely off peak hours. > > On 9/10/18, 12:16 PM, "Ted Yu" wrote: > > In the previous stack trace you sent, shortCompactions and > longCompactions > threads were not active. > > Was the stack trace captured during period when the number of client > operations was low ? > > If not, can you capture stack trace during off peak hours ? > > Cheers > > On Mon, Sep 10, 2018 at 12:08 PM Srinidhi Muppalla < > srinid...@trulia.com> > wrote: > > > Hi Ted, > > > > The highest number of filters used is 10, but the average is > generally > > close to 1. Is it possible the CPU usage spike has to do with Hbase > > internal maintenance operations? It looks like post-upgrade the > spike isn’t > > correlated with the frequency of reads/writes we are making, because > the > > high CPU usage persisted when the number of operations went down. > > > > Thank you, > > Srinidhi > > > > On 9/8/18, 9:44 AM, "Ted Yu" wrote: > > > > Srinidhi : > > Do you know the average / highest number of ColumnPrefixFilter's > in the > > FilterList ? > > > > Thanks > > > > On Fri, Sep 7, 2018 at 10:00 PM Ted Yu > wrote: > > > > > Thanks for detailed background information. > > > > > > I assume your code has done de-dup for the filters contained in > > > FilterListWithOR. > > > > > > I took a look at JIRAs which > > > touched > hbase-client/src/main/java/org/apache/hadoop/hbase/filter in > > > branch-1.4 > > > There were a few patches (some were very big) since the > release of > > 1.3.0 > > > So it is not obvious at first glance which one(s) might be > related. > > > > > > I noticed ColumnPrefixFilter.getNextCellHint (and > > > KeyValueUtil.createFirstOnRow) appearing many times in the > stack > > trace. > > > > > > I plan to dig more in this area. > > > > > > Cheers > > > > > > On Fri, Sep 7, 2018 at 11:30 AM Srinidhi Muppalla < > > srinid...@trulia.com> > > > wrote: > > > > > >> Sure thing. For our table schema, each row represents one > user and > > the > > >> row key is that user’s unique id in our system. We currently > only > > use one > > >> column family in the table. The column qualifiers represent > an item > > that > > >> has been surfaced to that user as well as additional > information to > > >> differentiate the way the item has been surfaced to the user. > > Without > > >> getting into too many specifics, the qualifier follows the > rough > > format of: > > >> > > >> “Channel-itemId-distinguisher”. > > >> > > >> The channel here is the channel through the item was > previously > > surfaced > > >> to the user. The itemid is the unique id of the item that has > been > > surfaced > > >> to the user. A distinguisher is some attribute about how that > item > > was > > >> surfaced to the user. > > >> > > >> When we run a scan, we currently only ever run it on one row > at a > > time. > > >> It was chosen over ‘get’ because (from our understanding) the > > performance > > >> difference is negligible, and down the road using scan would > allow > > us some > > >> more flexibility. > > >> > > >> The filter list that is constructed with scan works by using a > > >> ColumnPrefixFilter as you mentioned. When a user is being > > communicated to > > >> on a particular channel, we have a list of items that we want > to > > >> potentially surface for that user. So, we construct a prefix > list > > with the > > >> channel and each of the item ids in the form of: > “channel-itemId”. > > Then we > > >> run a scan on that row with that filter list using “WithOr” > to get > > all of > > >> the matching channel-itemId combinations currently in that > > row/column > > >> family in the table. This way we can then know which of the > items > > we want > > >> to surface to that user on that channel have already been > surfaced > > on that > > >> channel. The reason we query using a prefix filter is so that > we > > don’t need > > >> to know the ‘distinguisher’ part of the
Re: Uninitialized Message Exception thrown while getting values.
We have been using this setup for months now and this issue never occurred before. And we are not able to reproduce the same. On Fri, Jan 19, 2018 at 4:29 PM, Ted Yu <yuzhih...@gmail.com> wrote: > From the exception message, it seems some GetRequest was empty. > > How often did this happen ? > If you can describe characteristics of the get request, that may give some > clue. > > Can you come up with a unit test that reproduces the issue ? > > On Thu, Jan 18, 2018 at 11:40 PM, Karthick Ram <karthik.28...@gmail.com> > wrote: > > > ------ Forwarded message -- > > From: Karthick Ram <karthik.28...@gmail.com> > > Date: Thu, Jan 18, 2018 at 12:56 PM > > Subject: Re: Uninitialized Message Exception thrown while getting values. > > To: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> > > > > > > Hbase version 1.2.5 and both client and server have pb version 2.5.0 > only. > > > > On Thu, Jan 18, 2018 at 12:46 PM, ramkrishna vasudevan < > > ramkrishna.s.vasude...@gmail.com> wrote: > > > > > Hi > > > > > > Which version of HBase you get this problem? Do you have any pb > classpath > > > issues? > > > > > > Regards > > > Ram > > > > > > On Thu, Jan 18, 2018 at 12:40 PM, Karthick Ram < > karthik.28...@gmail.com> > > > wrote: > > > > > >> "UninitializedMessageException : Message missing required fields : > > >> region, get", is thrown while performing Get. Due to this all the Get > > >> requests to the same Region Server are getting stalled. > > >> > > >> com.google.protobuf.UninitializedMessageException: Message missing > > >> required fields : region, get > > >> at com.google.protobuf.AbstractMessage$Build.newUninitializedMe > > >> ssageException(AbstractMessage.java:770) > > >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetR > > >> equest$Builder.build(ClientProtos.java:6377) > > >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetR > > >> equest$Builder.build(ClientProtos.java:6309) > > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequ > > >> est(RpcServer.java:1840) > > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneR > > >> pc(RpcServer.java:1775) > > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RPc > > >> Server.java:1623) > > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProc > > >> ess(RpcServer.java:1603) > > >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcSer > > >> ver.java:861) > > >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunL > > >> oop(RpcServer.java:643) > > >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(Rp > > >> cServer.java:619) > > >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > > >> Executor.java:1145) > > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > > >> lExecutor.java:615) > > >> at java.lang.Thread.run(Thread.java:745) > > >> > > > > > > > > >
Fwd: Uninitialized Message Exception thrown while getting values.
-- Forwarded message -- From: Karthick Ram <karthik.28...@gmail.com> Date: Thu, Jan 18, 2018 at 4:35 PM Subject: Re: Uninitialized Message Exception thrown while getting values. To: Anoop John <anoop.hb...@gmail.com> Hbase version(client & server) 1.2.5, Protobuf version 2.5.0 and JDK 1.8.0 On Thu, Jan 18, 2018 at 3:19 PM, Anoop John <anoop.hb...@gmail.com> wrote: > Karthick > What is ur server and client versions? U using Java > client or some thing else? > > -Anoop- > > On Thu, Jan 18, 2018 at 12:46 PM, ramkrishna vasudevan > <ramkrishna.s.vasude...@gmail.com> wrote: > > Hi > > > > Which version of HBase you get this problem? Do you have any pb classpath > > issues? > > > > Regards > > Ram > > > > On Thu, Jan 18, 2018 at 12:40 PM, Karthick Ram <karthik.28...@gmail.com> > > wrote: > > > >> "UninitializedMessageException : Message missing required fields : > region, > >> get", is thrown while performing Get. Due to this all the Get requests > to > >> the same Region Server are getting stalled. > >> > >> com.google.protobuf.UninitializedMessageException: Message missing > >> required fields : region, get > >> at com.google.protobuf.AbstractMessage$Build. > >> newUninitializedMessageException(AbstractMessage.java:770) > >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetRequest$ > >> Builder.build(ClientProtos.java:6377) > >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetRequest$ > >> Builder.build(ClientProtos.java:6309) > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection. > >> processRequest(RpcServer.java:1840) > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection. > >> processOneRpc(RpcServer.java:1775) > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process( > >> RPcServer.java:1623) > >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection. > >> readAndProcess(RpcServer.java:1603) > >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead( > >> RpcServer.java:861) > >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader. > >> doRunLoop(RpcServer.java:643) > >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run( > >> RpcServer.java:619) > >> at java.util.concurrent.ThreadPoolExecutor.runWorker( > >> ThreadPoolExecutor.java:1145) > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run( > >> ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:745) > >> >
Fwd: Uninitialized Message Exception thrown while getting values.
-- Forwarded message -- From: Karthick Ram <karthik.28...@gmail.com> Date: Thu, Jan 18, 2018 at 12:56 PM Subject: Re: Uninitialized Message Exception thrown while getting values. To: ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> Hbase version 1.2.5 and both client and server have pb version 2.5.0 only. On Thu, Jan 18, 2018 at 12:46 PM, ramkrishna vasudevan < ramkrishna.s.vasude...@gmail.com> wrote: > Hi > > Which version of HBase you get this problem? Do you have any pb classpath > issues? > > Regards > Ram > > On Thu, Jan 18, 2018 at 12:40 PM, Karthick Ram <karthik.28...@gmail.com> > wrote: > >> "UninitializedMessageException : Message missing required fields : >> region, get", is thrown while performing Get. Due to this all the Get >> requests to the same Region Server are getting stalled. >> >> com.google.protobuf.UninitializedMessageException: Message missing >> required fields : region, get >> at com.google.protobuf.AbstractMessage$Build.newUninitializedMe >> ssageException(AbstractMessage.java:770) >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetR >> equest$Builder.build(ClientProtos.java:6377) >> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetR >> equest$Builder.build(ClientProtos.java:6309) >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequ >> est(RpcServer.java:1840) >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneR >> pc(RpcServer.java:1775) >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RPc >> Server.java:1623) >> at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProc >> ess(RpcServer.java:1603) >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcSer >> ver.java:861) >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunL >> oop(RpcServer.java:643) >> at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(Rp >> cServer.java:619) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1145) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> > >
Uninitialized Message Exception thrown while getting values.
"UninitializedMessageException : Message missing required fields : region, get", is thrown while performing Get. Due to this all the Get requests to the same Region Server are getting stalled. com.google.protobuf.UninitializedMessageException: Message missing required fields : region, get at com.google.protobuf.AbstractMessage$Build.newUninitializedMessageException(AbstractMessage.java:770) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetRequest$Builder.build(ClientProtos.java:6377) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetRequest$Builder.build(ClientProtos.java:6309) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1840) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1775) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RPcServer.java:1623) at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1603) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:861) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:643) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:619) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Fwd: Reply for 'Empty byte array values for cells which doesn't exist'
-- Forwarded message --
From: Karthick Ram <karthik.28...@gmail.com>
Date: Wed, Sep 27, 2017 at 5:21 PM
Subject: Reply for 'Empty byte array values for cells which doesn't exist'
To: yuzhih...@gmail.com, us...@hbase.apache.org, iss...@hbase.apache.org
Hi we use HBase 1.2.5 version and we can't reproduce the scenario in other
tables. And here is the description of the table,
Table namespace:tablename is ENABLED
namespace:tablename
COLUMN FAMILIES DESCRIPTION
{NAME => 'dataCF', DATA_BLOCK_ENCODING => 'NONE', ENCRYPTION => 'AES',
BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '1', VERSIONS => '1',
COMPRESSION => 'SNAPPY', MIN_VERSIONS => '1', TTL => '86400 SECOND
S (1 DAY)', KEEP_DELETED_CELLS => 'TTL', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}
{NAME => 'pointerCF', DATA_BLOCK_ENCODING => 'NONE', ENCRYPTION => 'AES',
BLOOMFILTER => 'ROWCOL', REPLICATION_SCOPE => '1', VERSIONS => '1',
COMPRESSION => 'SNAPPY', MIN_VERSIONS => '0', TTL => 'FOREVER',
KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false',
BLOCKCACHE => 'true'}
Fwd: Empty byte array values for cells which doesn't exist
-- Forwarded message -- From: Karthick Ram <karthik.28...@gmail.com> Date: Mon, Sep 25, 2017 at 10:00 AM Subject: Empty byte array values for cells which doesn't exist To: iss...@hbase.apache.org, d...@hbase.apache.org, us...@hbase.apache.org, ramkrishna vasudevan <ramkrishna.s.vasude...@gmail.com> Hi, we have a table which when queried with more than one columkey for a row key (using addcolumn(colfamily, columkey)), returns cells which are not present. It returns a empty byte array as value for those cells. Using debugger we found the timestamp of those cells to *'OLDEST_TIMESTAMP'* and the type to be *'Minimum'*. These cells turns out to be fake cells, however when queried with only columnkey it doesn't return any cells. Please look into the following files and suggest some ways to rectify this problem. 1. HConstant.java 2. KeyValue.java 3. ScanQueryMatcher.java 4. StoreFileScanner.java NOTE: We are not able to reproduce the same in other tables.
