Re: parallel processing of regions in coprocessor

[Daniel Połaczański]

Yes. I am using batch variant. I will try your variant

02.03.2017 7:53 AM "Jingcheng Du"  napisał(a):

What APIs are you using to execute the coprocessor?
HTable#batchCoprocessorService? It executes the coprocessors regions by
regions in each region server.
How about using coprocessorService? This API should run coprocessors in
parallel for different regions.

2017-03-02 14:32 GMT+08:00 Daniel Połaczański :

> I invoke my business logic which is similar to map reduce paradigm.
> Everything works. Only performance is problem that regions are not
> processed parallely.
> I'm thinking about some hack like invoking 4 times rpc from client when i
> have 4 regions
>
> 02.03.2017 00:43 "Ted Yu"  napisał(a):
>
> > To my knowledge, there is no support for this type of combination of map
> > reduce and coprocessor.
> >
> > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański <
> dpolaczan...@gmail.com
> > >
> > wrote:
> >
> > > It is something like map reduce processing.
> > > I want to run map and combine phase inside in a coprocessor for every
> > > region separately, return partial results to the client. On the client
> > > invoke reduce phase and store result in another hbase table.
> > >
> > > 2017-03-01 23:26 GMT+01:00 Ted Yu :
> > >
> > > > Can you describe your use case in more detail ?
> > > >
> > > > What type of custom coprocessor are you loading to the region server
> ?
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > > dpolaczan...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Let's assume that we have cluster consisting from one RegionServer
> > and
> > > > the
> > > > > RegionsServer contains one table consisting from 3 regions.
> > > > >
> > > > > I would like to process regions  in coprocessor parallely. Is it
> > > > possible?
> > > > >
> > > > > I observed that currenlty it invokes coproprocessor with all the
> > > regions
> > > > > one by one. So it process at the beginig region1, when finished
> > process
> > > > > regions2 and ...
> > > > >
> > > > > Is it possible to change this behaviour?
> > > > >
> > > > > Regards
> > > > >
> > > >
> > >
> >
>

client side configurations

[Rajeshkumar J]

I am trying to scan a table from a class in java. So this is hbase client.
I have created hbase configuration using HBaseCOnfiguration.create() in the
class and also I have placed my hbase-site.xml in the classpath. My hbase
services are up and running. Initially I dont have these properties
  1.hbase.client.scanner.timeout.period
  2.hbase.rpc.timeout

I have added this property and i tried to run the java class and it has
hbase-site.xml in classpath so whether these properties are added to both
hbase client and server side? or only client side and do server side needs
restart?

Re: parallel processing of regions in coprocessor

[Jingcheng Du]

What APIs are you using to execute the coprocessor?
HTable#batchCoprocessorService? It executes the coprocessors regions by
regions in each region server.
How about using coprocessorService? This API should run coprocessors in
parallel for different regions.

2017-03-02 14:32 GMT+08:00 Daniel Połaczański :

> I invoke my business logic which is similar to map reduce paradigm.
> Everything works. Only performance is problem that regions are not
> processed parallely.
> I'm thinking about some hack like invoking 4 times rpc from client when i
> have 4 regions
>
> 02.03.2017 00:43 "Ted Yu"  napisał(a):
>
> > To my knowledge, there is no support for this type of combination of map
> > reduce and coprocessor.
> >
> > On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański <
> dpolaczan...@gmail.com
> > >
> > wrote:
> >
> > > It is something like map reduce processing.
> > > I want to run map and combine phase inside in a coprocessor for every
> > > region separately, return partial results to the client. On the client
> > > invoke reduce phase and store result in another hbase table.
> > >
> > > 2017-03-01 23:26 GMT+01:00 Ted Yu :
> > >
> > > > Can you describe your use case in more detail ?
> > > >
> > > > What type of custom coprocessor are you loading to the region server
> ?
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > > dpolaczan...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > Let's assume that we have cluster consisting from one RegionServer
> > and
> > > > the
> > > > > RegionsServer contains one table consisting from 3 regions.
> > > > >
> > > > > I would like to process regions  in coprocessor parallely. Is it
> > > > possible?
> > > > >
> > > > > I observed that currenlty it invokes coproprocessor with all the
> > > regions
> > > > > one by one. So it process at the beginig region1, when finished
> > process
> > > > > regions2 and ...
> > > > >
> > > > > Is it possible to change this behaviour?
> > > > >
> > > > > Regards
> > > > >
> > > >
> > >
> >
>

Re: parallel processing of regions in coprocessor

[Daniel Połaczański]

I invoke my business logic which is similar to map reduce paradigm.
Everything works. Only performance is problem that regions are not
processed parallely.
I'm thinking about some hack like invoking 4 times rpc from client when i
have 4 regions

02.03.2017 00:43 "Ted Yu"  napisał(a):

> To my knowledge, there is no support for this type of combination of map
> reduce and coprocessor.
>
> On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański  >
> wrote:
>
> > It is something like map reduce processing.
> > I want to run map and combine phase inside in a coprocessor for every
> > region separately, return partial results to the client. On the client
> > invoke reduce phase and store result in another hbase table.
> >
> > 2017-03-01 23:26 GMT+01:00 Ted Yu :
> >
> > > Can you describe your use case in more detail ?
> > >
> > > What type of custom coprocessor are you loading to the region server ?
> > >
> > > Thanks
> > >
> > > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> > dpolaczan...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > > Let's assume that we have cluster consisting from one RegionServer
> and
> > > the
> > > > RegionsServer contains one table consisting from 3 regions.
> > > >
> > > > I would like to process regions  in coprocessor parallely. Is it
> > > possible?
> > > >
> > > > I observed that currenlty it invokes coproprocessor with all the
> > regions
> > > > one by one. So it process at the beginig region1, when finished
> process
> > > > regions2 and ...
> > > >
> > > > Is it possible to change this behaviour?
> > > >
> > > > Regards
> > > >
> > >
> >
>

Re: Need guidance on getting detailed elapsed times in every stage of processing a request

[jeff saremi]

Where would i seek help for issues revolving around HTrace and zipkin? Here? 
Because I have configured everything the way documentation said but i see 
nothing in the zipkin server or in the logs. nothing at all


From: jeff saremi 
Sent: Tuesday, February 28, 2017 12:52:32 PM
To: user@hbase.apache.org
Subject: Re: Need guidance on getting detailed elapsed times in every stage of 
processing a request

No I had not. but it looks like what i needed. Thanks Ted.

I'll see if I have any more questions after reading this.


From: Ted Yu 
Sent: Tuesday, February 28, 2017 12:47:08 PM
To: user@hbase.apache.org
Subject: Re: Need guidance on getting detailed elapsed times in every stage of 
processing a request

Have you looked at:
http://hbase.apache.org/book.html#tracing

On Tue, Feb 28, 2017 at 12:37 PM, jeff saremi 
wrote:

> I think we need to get detailed information from HBase RegionServer logs
> on how a request (read or write) is processed. Specifically speaking, i
> need to know of say 100 ms time spent in processing a write, how much of it
> was spent waiting for the HDFS?
> What is the most efficient way of enabling this in log4j properties? Are
> there better mechanisms to get this information?
>
> If I can get this in the log, then I can process the logs offline or in
> neartime and mount some dashboards on the top.
>
> thanks
>
>
> Jeff
>

Re: parallel processing of regions in coprocessor

[Ted Yu]

To my knowledge, there is no support for this type of combination of map
reduce and coprocessor.

On Wed, Mar 1, 2017 at 2:55 PM, Daniel Połaczański 
wrote:

> It is something like map reduce processing.
> I want to run map and combine phase inside in a coprocessor for every
> region separately, return partial results to the client. On the client
> invoke reduce phase and store result in another hbase table.
>
> 2017-03-01 23:26 GMT+01:00 Ted Yu :
>
> > Can you describe your use case in more detail ?
> >
> > What type of custom coprocessor are you loading to the region server ?
> >
> > Thanks
> >
> > On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański <
> dpolaczan...@gmail.com
> > >
> > wrote:
> >
> > > Hi,
> > > Let's assume that we have cluster consisting from one RegionServer and
> > the
> > > RegionsServer contains one table consisting from 3 regions.
> > >
> > > I would like to process regions  in coprocessor parallely. Is it
> > possible?
> > >
> > > I observed that currenlty it invokes coproprocessor with all the
> regions
> > > one by one. So it process at the beginig region1, when finished process
> > > regions2 and ...
> > >
> > > Is it possible to change this behaviour?
> > >
> > > Regards
> > >
> >
>

Re: parallel processing of regions in coprocessor

[Daniel Połaczański]

It is something like map reduce processing.
I want to run map and combine phase inside in a coprocessor for every
region separately, return partial results to the client. On the client
invoke reduce phase and store result in another hbase table.

2017-03-01 23:26 GMT+01:00 Ted Yu :

> Can you describe your use case in more detail ?
>
> What type of custom coprocessor are you loading to the region server ?
>
> Thanks
>
> On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański  >
> wrote:
>
> > Hi,
> > Let's assume that we have cluster consisting from one RegionServer and
> the
> > RegionsServer contains one table consisting from 3 regions.
> >
> > I would like to process regions  in coprocessor parallely. Is it
> possible?
> >
> > I observed that currenlty it invokes coproprocessor with all the regions
> > one by one. So it process at the beginig region1, when finished process
> > regions2 and ...
> >
> > Is it possible to change this behaviour?
> >
> > Regards
> >
>

Re: parallel processing of regions in coprocessor

[Ted Yu]

Can you describe your use case in more detail ?

What type of custom coprocessor are you loading to the region server ?

Thanks

On Wed, Mar 1, 2017 at 2:24 PM, Daniel Połaczański 
wrote:

> Hi,
> Let's assume that we have cluster consisting from one RegionServer and the
> RegionsServer contains one table consisting from 3 regions.
>
> I would like to process regions  in coprocessor parallely. Is it possible?
>
> I observed that currenlty it invokes coproprocessor with all the regions
> one by one. So it process at the beginig region1, when finished process
> regions2 and ...
>
> Is it possible to change this behaviour?
>
> Regards
>

parallel processing of regions in coprocessor

[Daniel Połaczański]

Hi,
Let's assume that we have cluster consisting from one RegionServer and the
RegionsServer contains one table consisting from 3 regions.

I would like to process regions  in coprocessor parallely. Is it possible?

I observed that currenlty it invokes coproprocessor with all the regions
one by one. So it process at the beginig region1, when finished process
regions2 and ...

Is it possible to change this behaviour?

Regards

Re: On HBase Read Replicas

[Biju N]

>From the table definition. For e.g.
https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HTableDescriptor.html#getRegionReplication--

On Tue, Feb 28, 2017 at 3:30 PM, jeff saremi  wrote:

> Enis
>
> just one more question. How would i go about getting the count of the
> replica's for a table or columngroup? thanks
>
> 
> From: Enis Söztutar 
> Sent: Wednesday, February 22, 2017 1:38:41 PM
> To: hbase-user
> Subject: Re: On HBase Read Replicas
>
> If you are doing a get to a specific replica, it will execute as a read
> with retries to a single "copy". There will not be any backup / fallback
> RPCs to any other replica.
>
> Only in timeline consistency mode there will be fallback RPCs.
>
> Enis
>
> On Sun, Feb 19, 2017 at 9:43 PM, Anoop John  wrote:
>
> > Thanks Enis.. I was not knowing the way of setting replica id
> > specifically..  So what will happen if that said replica is down at
> > the read time?  Will that go to another replica?
> >
> > -Anoop-
> >
> > On Sat, Feb 18, 2017 at 3:34 AM, Enis Söztutar 
> wrote:
> > > You can do gets using two different "modes":
> > >  - Do a read with backup RPCs. In case, the algorithm that I have above
> > > will be used. 1 RPC to primary, and 2 more RPCs after primary timeouts.
> > >  - Do a read to a single replica. In this case, there is only 1 RPC
> that
> > > will happen to that given replica.
> > >
> > > Enis
> > >
> > > On Fri, Feb 17, 2017 at 12:03 PM, jeff saremi 
> > > wrote:
> > >
> > >> Enis
> > >>
> > >> Thanks for taking the time to reply
> > >>
> > >> So i thought that a read request is sent to all Replicas regardless.
> If
> > we
> > >> have the option of Sending to one, analyzing response, and then
> sending
> > to
> > >> another, this bodes well with our scenarios.
> > >>
> > >> Please confirm
> > >>
> > >> thanks
> > >>
> > >> 
> > >> From: Enis Söztutar 
> > >> Sent: Friday, February 17, 2017 11:38:42 AM
> > >> To: hbase-user
> > >> Subject: Re: On HBase Read Replicas
> > >>
> > >> You can use read-replicas to distribute the read-load if you are fine
> > with
> > >> stale reads. The read replicas normally have a "backup rpc" path,
> which
> > >> implements a logic like this:
> > >>  - Send the RPC to the primary replica
> > >>  - if no response for 100ms (or configured timeout), send RPCs to the
> > other
> > >> replicas
> > >>  - return the first non-exception response.
> > >>
> > >> However, there is also another feature for read replicas, where you
> can
> > >> indicate which exact replica_id you want to read from when you are
> > doing a
> > >> get. If you do this:
> > >> Get get = new Get(row);
> > >> get.setReplicaId(2);
> > >>
> > >> the Get RPC will only go to the replica_id=2. Note that if you have
> > region
> > >> replication = 3, then you will have regions with replica ids: {0, 1,
> 2}
> > >> where replica_id=0 is the primary.
> > >>
> > >> So you can do load-balancing with a get.setReplicaId(random() %
> > >> num_replicas) kind of pattern.
> > >>
> > >> Enis
> > >>
> > >>
> > >>
> > >> On Thu, Feb 16, 2017 at 9:41 AM, Anoop John 
> > wrote:
> > >>
> > >> > Never saw this kind of discussion.
> > >> >
> > >> > -Anoop-
> > >> >
> > >> > On Thu, Feb 16, 2017 at 10:13 PM, jeff saremi <
> jeffsar...@hotmail.com
> > >
> > >> > wrote:
> > >> > > Thanks Anoop.
> > >> > >
> > >> > > Understood.
> > >> > >
> > >> > > Have there been enhancement requests or discussions on load
> > balancing
> > >> by
> > >> > providing additional replicas in the past? Has anyone else come up
> > with
> > >> > anything on this?
> > >> > > thanks
> > >> > >
> > >> > > 
> > >> > > From: Anoop John 
> > >> > > Sent: Thursday, February 16, 2017 2:35:48 AM
> > >> > > To: user@hbase.apache.org
> > >> > > Subject: Re: On HBase Read Replicas
> > >> > >
> > >> > > The region replica feature came in so as to reduce the MTTR and so
> > >> > > increase the data availability.  When the master region containing
> > RS
> > >> > > dies, the clients can read from the secondary regions.  But to
> keep
> > >> > > one thing in mind that this data from secondary regions will be
> bit
> > >> > > out of sync as the replica is eventual consistent.   Because of
> this
> > >> > > said reason,  change client so as to share the load across diff
> RSs
> > >> > > might be tough.
> > >> > >
> > >> > > -Anoop-
> > >> > >
> > >> > > On Sun, Feb 12, 2017 at 8:13 AM, jeff saremi <
> > jeffsar...@hotmail.com>
> > >> > wrote:
> > >> > >> Yes indeed. thank you very much Ted
> > >> > >>
> > >> > >> 
> > >> > >> From: Ted Yu 
> > >> > >> Sent: Saturday, February 11, 2017 3:40:50 PM
> > >> > >> To: user@hbase.apache.org
> > >> > >> Subject: Re: On HBase Read Replicas
> > >> > 

Re: hbase dynamic configuration

[Jerry He]

IIRC Updating configuration dynamically works by re-loading from disk the
conf files on the target server by the server.
But again, only a few properties are actually being effectively refreshed
by the components.

The shell command accepts a server name.

Thanks.

Jerry


On Wed, Mar 1, 2017 at 12:39 AM, Rajeshkumar J 
wrote:

> ​Thanks Jerry for the reply. Also is there any sample command for this
> dynamic configuration. As I have tried like
>
> update_config 'hbase.hstore.blockingStoreFiles','101'​
> but it is throwing error "ERROR: wrong number of arguments (2 for 1)"
>
> On Wed, Mar 1, 2017 at 12:55 PM, Jerry He  wrote:
>
> > These properties can be used on the client side and the server side.
> > I assume you are asking about them on the server side.
> >
> > Unfortunately, these two are not supported yet for dynamic configuration.
> >
> > Thanks.
> >
> > Jerry
> >
> > On Tue, Feb 28, 2017 at 10:00 PM, Rajeshkumar J <
> > rajeshkumarit8...@gmail.com
> > > wrote:
> >
> > > Hi,
> > >
> > >Can any one tell me whether we can add the below properties
> > dynamically
> > >
> > >   1.hbase.client.scanner.timeout.period
> > >   2.hbase.rpc.timeout
> > >
> > > Thanks,
> > > Rajeshkumar
> > >
> >
>

Re: HBase scan returns inconsistent results on multiple runs for same dataset

[Sean Busbey]

The place to check for include JIRAs on top of those in the ASF release is here:

http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.9.1.releasenotes.html

HBASE-15378 is not in CDH5.9.1.

On Wed, Mar 1, 2017 at 9:58 AM, Ted Yu  wrote:
> I don't see it here:
>
> http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.9.1.CHANGES.txt?_ga=1.10311413.1914112506.1454459553
>
> On Wed, Mar 1, 2017 at 5:46 AM, Hef  wrote:
>
>> I'm using CDH 5.9, the document show its HBase version is
>> hbase-1.2.0+cdh5.9.1+222.  (
>> https://www.cloudera.com/documentation/enterprise/
>> release-notes/topics/cdh_vd_cdh_package_tarball_59.html
>> )
>> I have no idea if  HBASE-15378  is included.
>>
>> On Wed, Mar 1, 2017 at 9:33 PM, Ted Yu  wrote:
>>
>> > Which hbase version are you using ?
>> >
>> > Does it include HBASE-15378 ?
>> >
>> > > On Mar 1, 2017, at 5:02 AM, Hef  wrote:
>> > >
>> > > Hi,
>> > > I'm encountering a strange behavior on MapReduce when using HBase as
>> > input
>> > > format. I run my MR tasks on a same table, same dataset, with a same
>> > > pattern of Fuzzy Row Filter, multiple times. The Input Records counters
>> > > shown are not consistent, the smallest number can be 40% less than the
>> > > largest one.
>> > >
>> > > More specifically,
>> > > - the table is split into 18 regions, distributed on 3 region server.
>> The
>> > > TTL is set to 10 days for the record, though the dataset for MR only
>> > > includes those inserted in 7days.
>> > >
>> > > - The row key is defined as:
>> > > sault(1byte) + time_of_hour(4bytes) + uuid(36bytes)
>> > >
>> > >
>> > > - The scan is created as below:
>> > >
>> > > Scan scan = new Scan();
>> > > scan.setBatch(100);
>> > > scan.setCaching(1);
>> > > scan.setCacheBlocks(false);
>> > > scan.setMaxVersions(1);
>> > >
>> > >
>> > > And the row filter for the scan is a FuzzyRowFilter that filters only
>> > > events of a given time_of_hour.
>> > >
>> > > Everything looks fine while the result is out of expect.
>> > > A same task runs 10 times, the Input Records counters  show 6 different
>> > > numbers, and the final output shows 6 different results.
>> > >
>> > > Does anyone has every faced this problem before?
>> > > What could be the cause of this inconsistency of HBase scan result?
>> > >
>> > > Thanks
>> >
>>

Re: HBase scan returns inconsistent results on multiple runs for same dataset

[Ted Yu]

I don't see it here:

http://archive.cloudera.com/cdh5/cdh/5/hbase-1.2.0-cdh5.9.1.CHANGES.txt?_ga=1.10311413.1914112506.1454459553

On Wed, Mar 1, 2017 at 5:46 AM, Hef  wrote:

> I'm using CDH 5.9, the document show its HBase version is
> hbase-1.2.0+cdh5.9.1+222.  (
> https://www.cloudera.com/documentation/enterprise/
> release-notes/topics/cdh_vd_cdh_package_tarball_59.html
> )
> I have no idea if  HBASE-15378  is included.
>
> On Wed, Mar 1, 2017 at 9:33 PM, Ted Yu  wrote:
>
> > Which hbase version are you using ?
> >
> > Does it include HBASE-15378 ?
> >
> > > On Mar 1, 2017, at 5:02 AM, Hef  wrote:
> > >
> > > Hi,
> > > I'm encountering a strange behavior on MapReduce when using HBase as
> > input
> > > format. I run my MR tasks on a same table, same dataset, with a same
> > > pattern of Fuzzy Row Filter, multiple times. The Input Records counters
> > > shown are not consistent, the smallest number can be 40% less than the
> > > largest one.
> > >
> > > More specifically,
> > > - the table is split into 18 regions, distributed on 3 region server.
> The
> > > TTL is set to 10 days for the record, though the dataset for MR only
> > > includes those inserted in 7days.
> > >
> > > - The row key is defined as:
> > > sault(1byte) + time_of_hour(4bytes) + uuid(36bytes)
> > >
> > >
> > > - The scan is created as below:
> > >
> > > Scan scan = new Scan();
> > > scan.setBatch(100);
> > > scan.setCaching(1);
> > > scan.setCacheBlocks(false);
> > > scan.setMaxVersions(1);
> > >
> > >
> > > And the row filter for the scan is a FuzzyRowFilter that filters only
> > > events of a given time_of_hour.
> > >
> > > Everything looks fine while the result is out of expect.
> > > A same task runs 10 times, the Input Records counters  show 6 different
> > > numbers, and the final output shows 6 different results.
> > >
> > > Does anyone has every faced this problem before?
> > > What could be the cause of this inconsistency of HBase scan result?
> > >
> > > Thanks
> >
>

Re: Region Server Hotspot/CPU Problem

[Sean Busbey]

Several of those jiras are fixed in later versions of CDH. Since the
inclusion of jiras in packaging by particular vendors is a vendor
specific issue, please seek out help from the vendor (e.g. on the
community forum you've just mentioned).

On Wed, Mar 1, 2017 at 8:49 AM, Saad Mufti  wrote:
> Someone in our team found this:
>
> http://community.cloudera.com/t5/Storage-Random-Access-HDFS/CPU-Usage-high-when-using-G1GC/td-p/48101
>
> Looks like we're bitten by this bug. Unfortunately this is only fixed in
> HBase 1.4.0 so we'll have to undertake a version upgrade which is not
> trivial.
>
> -
> Saad
>
>
> On Wed, Mar 1, 2017 at 9:38 AM, Sudhir Babu Pothineni > wrote:
>
>> First obvious thing to check is "major compaction" happening at the same
>> time when it goes to 100% CPU?
>> See this helps:
>> https://community.hortonworks.com/articles/52616/hbase-
>> compaction-tuning-tips.html
>>
>>
>>
>> Sent from my iPhone
>>
>> > On Mar 1, 2017, at 6:06 AM, Saad Mufti  wrote:
>> >
>> > Hi,
>> >
>> > We are using HBase 1.0.0-cdh5.5.2 on AWS EC2 instances. The load on HBase
>> > is heavy and a mix of reads and writes. For a few months we have had a
>> > problem where occasionally (once a day or more) one of the region servers
>> > starts consuming close to 100% CPU. This causes all the client thread
>> pool
>> > to get filled up serving the slow region server, causing overall response
>> > times to slow to a crawl and many calls either start timing out right in
>> > the client, or at a higher level.
>> >
>> > We have done lots of analysis and looked at various metrics but could
>> never
>> > pin it down to any particular kind of traffic or specific "hot keys".
>> > Looking at region server logs has not resulted in any findings. The only
>> > sort of vague evidence we have is that from the reported metrics, reads
>> per
>> > second on the hot server looks more than the other but not in a steady
>> > state but in a spiky but steady fashion, but gets per second looks no
>> > different than any other server.
>> >
>> > Until now our hacky way that we discovered to get around this was to just
>> > restart the region server. This works because while some calls error out
>> > while the regions are in transition, this is a batch oriented system
>> with a
>> > retry strategy built in.
>> >
>> > But just yesterday we discovered something interesting, if we connect to
>> > the region server in VisualVM and press the "Perform GC" button, there
>> > seems to be a brief pause and then CPU settles down back to normal. This
>> is
>> > despite the fact that memory appears to be under no pressure and before
>> we
>> > do this, VisualVM indicates very low percentage of CPU time spent in GC,
>> so
>> > we're baffled, and hoping someone with deeper insight into the HBase code
>> > could explain this behavior.
>> >
>> > Our region server processes are configured with 32GB of RAM and the
>> > following GC related JVM settings :
>> >
>> > HBASE_REGIONSERVER_OPTS=-Xms34359738368 -Xmx34359738368 -XX:+UseG1GC
>> > -XX:MaxGCPauseMillis=100
>> > -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=14
>> > -XX:InitiatingHeapOccupancyPercent=70
>> >
>> > Any insight anyone can provide would be most appreciated.
>> >
>> > 
>> > Saad
>>

Re: Region Server Hotspot/CPU Problem

[Saad Mufti]

Someone in our team found this:

http://community.cloudera.com/t5/Storage-Random-Access-HDFS/CPU-Usage-high-when-using-G1GC/td-p/48101

Looks like we're bitten by this bug. Unfortunately this is only fixed in
HBase 1.4.0 so we'll have to undertake a version upgrade which is not
trivial.

-
Saad


On Wed, Mar 1, 2017 at 9:38 AM, Sudhir Babu Pothineni  wrote:

> First obvious thing to check is "major compaction" happening at the same
> time when it goes to 100% CPU?
> See this helps:
> https://community.hortonworks.com/articles/52616/hbase-
> compaction-tuning-tips.html
>
>
>
> Sent from my iPhone
>
> > On Mar 1, 2017, at 6:06 AM, Saad Mufti  wrote:
> >
> > Hi,
> >
> > We are using HBase 1.0.0-cdh5.5.2 on AWS EC2 instances. The load on HBase
> > is heavy and a mix of reads and writes. For a few months we have had a
> > problem where occasionally (once a day or more) one of the region servers
> > starts consuming close to 100% CPU. This causes all the client thread
> pool
> > to get filled up serving the slow region server, causing overall response
> > times to slow to a crawl and many calls either start timing out right in
> > the client, or at a higher level.
> >
> > We have done lots of analysis and looked at various metrics but could
> never
> > pin it down to any particular kind of traffic or specific "hot keys".
> > Looking at region server logs has not resulted in any findings. The only
> > sort of vague evidence we have is that from the reported metrics, reads
> per
> > second on the hot server looks more than the other but not in a steady
> > state but in a spiky but steady fashion, but gets per second looks no
> > different than any other server.
> >
> > Until now our hacky way that we discovered to get around this was to just
> > restart the region server. This works because while some calls error out
> > while the regions are in transition, this is a batch oriented system
> with a
> > retry strategy built in.
> >
> > But just yesterday we discovered something interesting, if we connect to
> > the region server in VisualVM and press the "Perform GC" button, there
> > seems to be a brief pause and then CPU settles down back to normal. This
> is
> > despite the fact that memory appears to be under no pressure and before
> we
> > do this, VisualVM indicates very low percentage of CPU time spent in GC,
> so
> > we're baffled, and hoping someone with deeper insight into the HBase code
> > could explain this behavior.
> >
> > Our region server processes are configured with 32GB of RAM and the
> > following GC related JVM settings :
> >
> > HBASE_REGIONSERVER_OPTS=-Xms34359738368 -Xmx34359738368 -XX:+UseG1GC
> > -XX:MaxGCPauseMillis=100
> > -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=14
> > -XX:InitiatingHeapOccupancyPercent=70
> >
> > Any insight anyone can provide would be most appreciated.
> >
> > 
> > Saad
>

Re: Region Server Hotspot/CPU Problem

[Sudhir Babu Pothineni]

First obvious thing to check is "major compaction" happening at the same time 
when it goes to 100% CPU?
See this helps:
https://community.hortonworks.com/articles/52616/hbase-compaction-tuning-tips.html



Sent from my iPhone

> On Mar 1, 2017, at 6:06 AM, Saad Mufti  wrote:
> 
> Hi,
> 
> We are using HBase 1.0.0-cdh5.5.2 on AWS EC2 instances. The load on HBase
> is heavy and a mix of reads and writes. For a few months we have had a
> problem where occasionally (once a day or more) one of the region servers
> starts consuming close to 100% CPU. This causes all the client thread pool
> to get filled up serving the slow region server, causing overall response
> times to slow to a crawl and many calls either start timing out right in
> the client, or at a higher level.
> 
> We have done lots of analysis and looked at various metrics but could never
> pin it down to any particular kind of traffic or specific "hot keys".
> Looking at region server logs has not resulted in any findings. The only
> sort of vague evidence we have is that from the reported metrics, reads per
> second on the hot server looks more than the other but not in a steady
> state but in a spiky but steady fashion, but gets per second looks no
> different than any other server.
> 
> Until now our hacky way that we discovered to get around this was to just
> restart the region server. This works because while some calls error out
> while the regions are in transition, this is a batch oriented system with a
> retry strategy built in.
> 
> But just yesterday we discovered something interesting, if we connect to
> the region server in VisualVM and press the "Perform GC" button, there
> seems to be a brief pause and then CPU settles down back to normal. This is
> despite the fact that memory appears to be under no pressure and before we
> do this, VisualVM indicates very low percentage of CPU time spent in GC, so
> we're baffled, and hoping someone with deeper insight into the HBase code
> could explain this behavior.
> 
> Our region server processes are configured with 32GB of RAM and the
> following GC related JVM settings :
> 
> HBASE_REGIONSERVER_OPTS=-Xms34359738368 -Xmx34359738368 -XX:+UseG1GC
> -XX:MaxGCPauseMillis=100
> -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=14
> -XX:InitiatingHeapOccupancyPercent=70
> 
> Any insight anyone can provide would be most appreciated.
> 
> 
> Saad

Re: HBase scan returns inconsistent results on multiple runs for same dataset

[Hef]

I'm using CDH 5.9, the document show its HBase version is
hbase-1.2.0+cdh5.9.1+222.  (
https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_59.html
)
I have no idea if  HBASE-15378  is included.

On Wed, Mar 1, 2017 at 9:33 PM, Ted Yu  wrote:

> Which hbase version are you using ?
>
> Does it include HBASE-15378 ?
>
> > On Mar 1, 2017, at 5:02 AM, Hef  wrote:
> >
> > Hi,
> > I'm encountering a strange behavior on MapReduce when using HBase as
> input
> > format. I run my MR tasks on a same table, same dataset, with a same
> > pattern of Fuzzy Row Filter, multiple times. The Input Records counters
> > shown are not consistent, the smallest number can be 40% less than the
> > largest one.
> >
> > More specifically,
> > - the table is split into 18 regions, distributed on 3 region server. The
> > TTL is set to 10 days for the record, though the dataset for MR only
> > includes those inserted in 7days.
> >
> > - The row key is defined as:
> > sault(1byte) + time_of_hour(4bytes) + uuid(36bytes)
> >
> >
> > - The scan is created as below:
> >
> > Scan scan = new Scan();
> > scan.setBatch(100);
> > scan.setCaching(1);
> > scan.setCacheBlocks(false);
> > scan.setMaxVersions(1);
> >
> >
> > And the row filter for the scan is a FuzzyRowFilter that filters only
> > events of a given time_of_hour.
> >
> > Everything looks fine while the result is out of expect.
> > A same task runs 10 times, the Input Records counters  show 6 different
> > numbers, and the final output shows 6 different results.
> >
> > Does anyone has every faced this problem before?
> > What could be the cause of this inconsistency of HBase scan result?
> >
> > Thanks
>

Re: HBase scan returns inconsistent results on multiple runs for same dataset

[Ted Yu]

Which hbase version are you using ?

Does it include HBASE-15378 ?

> On Mar 1, 2017, at 5:02 AM, Hef  wrote:
> 
> Hi,
> I'm encountering a strange behavior on MapReduce when using HBase as input
> format. I run my MR tasks on a same table, same dataset, with a same
> pattern of Fuzzy Row Filter, multiple times. The Input Records counters
> shown are not consistent, the smallest number can be 40% less than the
> largest one.
> 
> More specifically,
> - the table is split into 18 regions, distributed on 3 region server. The
> TTL is set to 10 days for the record, though the dataset for MR only
> includes those inserted in 7days.
> 
> - The row key is defined as:
> sault(1byte) + time_of_hour(4bytes) + uuid(36bytes)
> 
> 
> - The scan is created as below:
> 
> Scan scan = new Scan();
> scan.setBatch(100);
> scan.setCaching(1);
> scan.setCacheBlocks(false);
> scan.setMaxVersions(1);
> 
> 
> And the row filter for the scan is a FuzzyRowFilter that filters only
> events of a given time_of_hour.
> 
> Everything looks fine while the result is out of expect.
> A same task runs 10 times, the Input Records counters  show 6 different
> numbers, and the final output shows 6 different results.
> 
> Does anyone has every faced this problem before?
> What could be the cause of this inconsistency of HBase scan result?
> 
> Thanks

HBase scan returns inconsistent results on multiple runs for same dataset

[Hef]

Hi,
I'm encountering a strange behavior on MapReduce when using HBase as input
format. I run my MR tasks on a same table, same dataset, with a same
pattern of Fuzzy Row Filter, multiple times. The Input Records counters
shown are not consistent, the smallest number can be 40% less than the
largest one.

More specifically,
- the table is split into 18 regions, distributed on 3 region server. The
TTL is set to 10 days for the record, though the dataset for MR only
includes those inserted in 7days.

- The row key is defined as:
sault(1byte) + time_of_hour(4bytes) + uuid(36bytes)


- The scan is created as below:

Scan scan = new Scan();
scan.setBatch(100);
scan.setCaching(1);
scan.setCacheBlocks(false);
scan.setMaxVersions(1);


And the row filter for the scan is a FuzzyRowFilter that filters only
events of a given time_of_hour.

Everything looks fine while the result is out of expect.
A same task runs 10 times, the Input Records counters  show 6 different
numbers, and the final output shows 6 different results.

Does anyone has every faced this problem before?
What could be the cause of this inconsistency of HBase scan result?

Thanks

Re:Whether turning off assignSeqIds using hbase bulk load will cause adverse impact or not

[Allan Yang]

Please refer to https://issues.apache.org/jira/browse/HBASE-10958
Assigning seqids to bulk loaded files is to avoid data loss when replay log.
If you are 1. all data are imported by bulkload other than insert or
 2. seqids in the loaded file are guaranteed small than the data in the memory
you can get rid of the assignSeqId process.


Allan Yang







At 2017-03-01 16:18:02, "沈国权"  wrote:
>Hi community:
>
>Currently our environment will import about 200G into HBase each day.
>These datas are cross six tables which table has about 100 Regions. We use
>HBase bulk tools completed these import
>tasks. We investigate HBase source code found when bulk loading task
>happen, HRegionServer will at firstly flush memstore according to each
>HRegion, We want to turn off assignSeqIds avoid each HRegion to force flush
>memstore. But We don't know these adverse impact if we do. Please help me
>to point out these impacts. Thanks

Re: hbase dynamic configuration

[Rajeshkumar J]

​Thanks Jerry for the reply. Also is there any sample command for this
dynamic configuration. As I have tried like

update_config 'hbase.hstore.blockingStoreFiles','101'​
but it is throwing error "ERROR: wrong number of arguments (2 for 1)"

On Wed, Mar 1, 2017 at 12:55 PM, Jerry He  wrote:

> These properties can be used on the client side and the server side.
> I assume you are asking about them on the server side.
>
> Unfortunately, these two are not supported yet for dynamic configuration.
>
> Thanks.
>
> Jerry
>
> On Tue, Feb 28, 2017 at 10:00 PM, Rajeshkumar J <
> rajeshkumarit8...@gmail.com
> > wrote:
>
> > Hi,
> >
> >Can any one tell me whether we can add the below properties
> dynamically
> >
> >   1.hbase.client.scanner.timeout.period
> >   2.hbase.rpc.timeout
> >
> > Thanks,
> > Rajeshkumar
> >
>

Whether turning off assignSeqIds using hbase bulk load will cause adverse impact or not

[沈国权]

Hi community:

Currently our environment will import about 200G into HBase each day.
These datas are cross six tables which table has about 100 Regions. We use
HBase bulk tools completed these import
tasks. We investigate HBase source code found when bulk loading task
happen, HRegionServer will at firstly flush memstore according to each
HRegion, We want to turn off assignSeqIds avoid each HRegion to force flush
memstore. But We don't know these adverse impact if we do. Please help me
to point out these impacts. Thanks