Re: Re: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC

[kevin su]

Congratulations, Zheng

Anoop John 於 2019年8月6日 週二，下午5:31寫道：

> Congrats Zheng.
>
> Anoop
>
> On Tue, Aug 6, 2019 at 8:52 AM OpenInx  wrote:
>
> > I'm so glad to join the PMC, Apache HBase is a great open source project
> > and the
> > community is also very nice and friendly.  In the comming days,   will do
> > more to make
> > the project & community forward, also we need more work to attract people
> > to enhance
> > and expand the community: blog/post/book/talk to share hbase experience
> or
> > ecosystem,
> > answer the question from our user list in time, help the beginner to
> > contribute the code, quick
> > feedback to our release vote / patch reviewing, etc.  all those things
> will
> > be helpful I think and
> > I will try to do more things like the above :-)
> >
> > Thanks all.
> >
> > On Tue, Aug 6, 2019 at 10:27 AM Chunhui Shen  wrote:
> >
> > > Congratulations, Zheng Hu!
> > >
> > >
> > > 在 2019-08-06 00:54:51，"Xu Cang"  写道：
> > > >Congratulations, Zheng Hu!
> > > >
> > > >On Mon, Aug 5, 2019 at 6:07 AM Guangxu Cheng 
> > > wrote:
> > > >
> > > >> Congratulations, Zheng!
> > > >>
> > > >> Allan Yang  于2019年8月5日周一 下午5:13写道：
> > > >>
> > > >> > Congratulations, Hu！
> > > >> > Best Regards
> > > >> > Allan Yang
> > > >> >
> > > >> >
> > > >> > Peter Somogyi  于2019年8月5日周一 下午4:47写道：
> > > >> >
> > > >> > > Congratulations!
> > > >> > >
> > > >> > > On Mon, Aug 5, 2019 at 8:57 AM Pankaj kr 
> > > wrote:
> > > >> > >
> > > >> > > > Congratulations Zheng..!!
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > > Pankaj
> > > >> > > >
> > > >> > > > -Original Message-
> > > >> > > > From: Duo Zhang [mailto:zhang...@apache.org]
> > > >> > > > Sent: 05 August 2019 07:38
> > > >> > > > To: HBase Dev List ; hbase-user <
> > > >> > > > user@hbase.apache.org>
> > > >> > > > Subject: [ANNOUNCE] Please welcome Zheng Hu to the HBase PMC
> > > >> > > >
> > > >> > > > On behalf of the Apache HBase PMC I am pleased to announce
> that
> > > Zheng
> > > >> > Hu
> > > >> > > > has accepted our invitation to become a PMC member on the
> Apache
> > > >> HBase
> > > >> > > > project. We appreciate Zheng Hu stepping up to take more
> > > >> responsibility
> > > >> > > in
> > > >> > > > the HBase project.
> > > >> > > >
> > > >> > > > Please join me in welcoming Zheng Hu to the HBase PMC!
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>

Re: [ANNOUNCE] new HBase committer Sakthi

[kevin su]

Congratulations 

OpenInx 於 2019年8月1日 週四，下午3:17寫道：

> Congratulations, Sakthi.
>
> On Thu, Aug 1, 2019 at 3:09 PM Jan Hentschel <
> jan.hentsc...@ultratendency.com> wrote:
>
> > Congrats Sakthi
> >
> > From: Reid Chan 
> > Reply-To: "user@hbase.apache.org" 
> > Date: Thursday, August 1, 2019 at 9:04 AM
> > To: "user@hbase.apache.org" , dev <
> > d...@hbase.apache.org>
> > Subject: Re: [ANNOUNCE] new HBase committer Sakthi
> >
> >
> > Congratulations and welcome, Sakthi!
> >
> >
> >
> > --
> >
> > Best regards,
> > R.C
> >
> >
> >
> > 
> > From: Sean Busbey mailto:bus...@apache.org>>
> > Sent: 01 August 2019 08:04
> > To: user@hbase.apache.org; dev
> > Subject: [ANNOUNCE] new HBase committer Sakthi
> >
> > On behalf of the HBase PMC, I'm pleased to announce that Sakthi has
> > accepted our invitation to become an HBase committer.
> >
> > We'd like to thank Sakthi for all of his diligent contributions to the
> > project thus far. We look forward to his continued participation in our
> > community.
> >
> > Congrats and welcome Sakthi!
> >
> >
>

Build error in latest two patch

[kevin su]

I tried to build project(clone from githb), use "mvn package -DskipTests"
i got this error

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
(default-compile) on project hbase-server: Compilation
failure[ERROR*]*
/C:/git/hbase/hbase-server/target/generated-sources/java/org/apache/hadoop/hbase/generated/master/rits_jsp.java:[104,87]
incompatible types:
org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure
cannot be converted to
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure

But when throw back two commit at HBASE-22123

It could build successfully.

Do I miss something when i build?

Re: splitPoint parameter in splitRegionAsync

[kevin su]

ok, I got you
Thanks stack.

Stack  於 2019年3月29日 週五 上午5:45寫道：

> The end key of one region is the start key of the next so cecking the
> startkey is sufficient?
> Thanks Kevin,
> S
>
>
> On Tue, Mar 26, 2019 at 1:47 PM kevin su  wrote:
>
> > Hi Users,
> >
> > I found that when we start to split region, used splitRegionAsync in
> > HbaseAdmin.
> > it only check whether the splitPoint is startkey or not.
> >
> > should we also check splitPoint is endkey ?
> >
> >
> > Thanks.
> >
> > Kevin
> >
>

Re: splitPoint parameter in splitRegionAsync

[kevin su]

hi stack,

In the hbase/client/HBaseAdmin.java : https://goo.gl/hojSM3
<https://goo.gl/hojSM3>

 Future splitRegionAsync(RegionInfo hri, byte[] splitPoint)
throws IOException {
TableName tableName = hri.getTable();
if (hri.getStartKey() != null && splitPoint != null &&
Bytes.compareTo(hri.getStartKey(), splitPoint) == 0) {
  throw new IOException("should not give a splitkey which equals
to startkey!");
}

...
...

Thanks,
Kevin


Stack  於 2019年3月27日 週三 下午11:07寫道：

> That sounds right Kevin. Mind adding pointer to where in the code you are
> referring to?
> Thanks,
> S
>
> On Tue, Mar 26, 2019 at 1:47 PM kevin su  wrote:
>
> > Hi Users,
> >
> > I found that when we start to split region, used splitRegionAsync in
> > HbaseAdmin.
> > it only check whether the splitPoint is startkey or not.
> >
> > should we also check splitPoint is endkey ?
> >
> >
> > Thanks.
> >
> > Kevin
> >
>

splitPoint parameter in splitRegionAsync

[kevin su]

Hi Users,

I found that when we start to split region, used splitRegionAsync in
HbaseAdmin.
it only check whether the splitPoint is startkey or not.

should we also check splitPoint is endkey ?


Thanks.

Kevin

Re: A little typo in ClusterConnection

[kevin su]

Ok, i got you
Thanks for you reply.

Stack  於 2019年3月15日 週五 下午1:10寫道：

> File an issue Kevin? Maybe attach a patch?
> Thank you,
> S
>
> On Wed, Mar 13, 2019 at 1:51 AM kevin su  wrote:
>
> > Hi,
> >
> > I clone the latest the hbase from github.
> > i found the comment on the top in ClusterConnection from
> > org.apache.hadoop.hbase.client.
> >
> > /**
> >  *
> >
> >  * Licensed to the Apache Software Foundation (ASF) under one
> >  * or more contributor license agreements.  See the NOTICE file
> >  * distributed with this work for additional information
> >  * regarding copyright ownership.  The ASF licenses this file
> >  * to you under the Apache License, Version 2.0 (the
> >
> >  There are addition blank line, I thinks this should be fixed.
> >
> > Kevin Su,
> > Thanks
> >
>

A little typo in ClusterConnection

[kevin su]

Hi,

I clone the latest the hbase from github.
i found the comment on the top in ClusterConnection from
org.apache.hadoop.hbase.client.

/**
 *

 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the

 There are addition blank line, I thinks this should be fixed.

Kevin Su,
Thanks

Re: How to upgrade JDK in Hbase and Hadoop server

[Kevin Bowling]

I can’t envision any incompatibilities and nodes running the different JDKs
shouldn’t have any issues communicating but depending on the stakes you may
wish to build out either a simple lab or a complex staging environment with
a snapshot of all the data to develop a playbook for doing the rollout and
also testing a revert if something goes wrong.

On Thu, Feb 21, 2019 at 2:24 PM Roshan  wrote:

> Hi
>
> I need to update the JDK from oracleJDK7 to OpenJDK7 for the Hbase(Ver
> 1.2.6) and Hadoop(Ver 2.7.3). Is the OpenJDK is compatible for the above 2
> versions?
>
> Also during rollover upgrade, will any problem occur as some of them
> refers OracleJDK and some refers OpenJDK?
>
> What are the issues/challenges in it?
>
> Any help is appreciated.
>
> Regards,
> Roshan
>

Re: [ANNOUNCE] Please welcome Peter Somogyi to the HBase PMC

[kevin su]

Congrats Peter~~

Regards,
Kevin Su

Duo Zhang  於 2019年1月22日 週二 上午9:36寫道：

> On behalf of the Apache HBase PMC I am pleased to announce that Peter
> Somogyi
> has accepted our invitation to become a PMC member on the Apache HBase
> project.
> We appreciate Peter stepping up to take more responsibility in the HBase
> project.
>
> Please join me in welcoming Peter to the HBase PMC!
>

hbase maven build error

[kevin su]

HI :

I try to use maven to build newest hbase from github,but i got this error.

[WARNING] The requested profile "assembly:single" could not be activated
because it does not exist.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce
(check-aggregate-license) on project hbase-shaded: Some Enforcer rules have
failed. Look above for specific messages explaining why the rule failed. ->
[Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR]   mvn  -rf :hbase-shaded

OS : windows
maven version : 3.5.4
hbase branch : master

I didn't edit anything in repository.
*i use mvn clean package,do i use wrnog command ?*

Best Regards,
Kevin

Re: Asked to modify this region's memstoreSize to a negative value which is incorrect

[Kevin GEORGES]

Yes, we load org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint

No idea about how we led to this situation :/
We have had some trouble with some RS last week, which lead us to perform 
multiple hbase hack -repair - not sure it’s related


On May 15, 2018 at 7:35:11 PM, rahul gidwani (rahul.gidw...@gmail.com) wrote:

Are you using coprocessors? Can you tell us any more about what led to  
this.  

Thanks  


On Tue, May 15, 2018 at 4:27 AM Kevin GEORGES <ke...@d33d33.fr> wrote:  

> We are running HBASE 1.4.0  
>  
>  
> On May 15, 2018 at 1:10:15 PM, Kevin GEORGES (ke...@d33d33.fr) wrote:  
>  
> Hello,  
>  
> We find region server abort with the following exception:  
>  
> 2018-05-15 08:23:23,920 ERROR  
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=7,port=16020]  
> regionserver.HRegion: Asked to modify this region's  
> (continuum,R\x0C\xF6\xF2\xBD\xD4L"\xB5\xFC\  
> xC6b\x8D\xD7\xC8x$\x7F\xFA\x9F\xA4\x92,1524491062878.117704beb050dcd3920335b4b290a898.)
>   
> memstoreSize to a negative value which is incorrect. Current  
> memstoreSize=-1533230, delta=480  
> java.lang.Exception  
> at  
> org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1205)
>   
> at  
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3534)
>   
> at  
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3102)  
> at  
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3044)  
> at  
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:894)
>   
> at  
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:822)
>   
> at  
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2376)
>   
> at  
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36621)
>   
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352)  
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)  
> at  
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)  
> at  
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)  
> 2018-05-15 08:23:23,922 FATAL [regionserver/  
> dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
>   
> regionserver.HRegionServer: ABORTING region server dn-35.hadoo  
> p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while  
> closing store  
> continuum,RH\xFD\xA6\x88\xD7\xFB5\xBBq\xD9\xE8|\xF2I_r\x7F\xFA\xB0Y\x88,1509095154547.51aea042f53
>   
> 655350c0d098fd378ab9b. v. flushableSize expected=0, actual= 23429. Current  
> memstoreSize=-34925. Maybe a coprocessor operation failed and left the  
> memstore in a partially updated state.  
> 2018-05-15 08:23:23,922 FATAL [regionserver/  
> dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
>   
> regionserver.HRegionServer: RegionServer abort: loaded coproce  
> ssors are: [org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint  
>  
>  
> The error about memstoreSize becoming negative appear at a steady rate  
> before abort (hundreds/sec)  
>  
> Any ideas?  
>  
> Thanks,  
>  
> Kevin  

Re: Asked to modify this region's memstoreSize to a negative value which is incorrect

[Kevin GEORGES]

We are running HBASE 1.4.0


On May 15, 2018 at 1:10:15 PM, Kevin GEORGES (ke...@d33d33.fr) wrote:

Hello,

We find region server abort with the following exception:

2018-05-15 08:23:23,920 ERROR 
[RpcServer.default.FPBQ.Fifo.handler=27,queue=7,port=16020] 
regionserver.HRegion: Asked to modify this region's 
(continuum,R\x0C\xF6\xF2\xBD\xD4L"\xB5\xFC\
xC6b\x8D\xD7\xC8x$\x7F\xFA\x9F\xA4\x92,1524491062878.117704beb050dcd3920335b4b290a898.)
 memstoreSize to a negative value which is incorrect. Current 
memstoreSize=-1533230, delta=480
java.lang.Exception
        at 
org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1205)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3534)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3102)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3044)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:894)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:822)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2376)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36621)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
2018-05-15 08:23:23,922 FATAL 
[regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
 regionserver.HRegionServer: ABORTING region server dn-35.hadoo
p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while 
closing store 
continuum,RH\xFD\xA6\x88\xD7\xFB5\xBBq\xD9\xE8|\xF2I_r\x7F\xFA\xB0Y\x88,1509095154547.51aea042f53
655350c0d098fd378ab9b. v. flushableSize expected=0, actual= 23429. Current 
memstoreSize=-34925. Maybe a coprocessor operation failed and left the memstore 
in a partially updated state.
2018-05-15 08:23:23,922 FATAL 
[regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
 regionserver.HRegionServer: RegionServer abort: loaded coproce
ssors are: [org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint


The error about memstoreSize becoming negative appear at a steady rate before 
abort (hundreds/sec)

Any ideas?

Thanks,

Kevin

Asked to modify this region's memstoreSize to a negative value which is incorrect

[Kevin GEORGES]

Hello,

We find region server abort with the following exception:

2018-05-15 08:23:23,920 ERROR 
[RpcServer.default.FPBQ.Fifo.handler=27,queue=7,port=16020] 
regionserver.HRegion: Asked to modify this region's 
(continuum,R\x0C\xF6\xF2\xBD\xD4L"\xB5\xFC\
xC6b\x8D\xD7\xC8x$\x7F\xFA\x9F\xA4\x92,1524491062878.117704beb050dcd3920335b4b290a898.)
 memstoreSize to a negative value which is incorrect. Current 
memstoreSize=-1533230, delta=480
java.lang.Exception
        at 
org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1205)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3534)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3102)
        at 
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3044)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:894)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:822)
        at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2376)
        at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36621)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
        at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
2018-05-15 08:23:23,922 FATAL 
[regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
 regionserver.HRegionServer: ABORTING region server dn-35.hadoo
p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while 
closing store 
continuum,RH\xFD\xA6\x88\xD7\xFB5\xBBq\xD9\xE8|\xF2I_r\x7F\xFA\xB0Y\x88,1509095154547.51aea042f53
655350c0d098fd378ab9b. v. flushableSize expected=0, actual= 23429. Current 
memstoreSize=-34925. Maybe a coprocessor operation failed and left the memstore 
in a partially updated state.
2018-05-15 08:23:23,922 FATAL 
[regionserver/dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]
 regionserver.HRegionServer: RegionServer abort: loaded coproce
ssors are: [org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint


The error about memstoreSize becoming negative appear at a steady rate before 
abort (hundreds/sec)

Any ideas?

Thanks,

Kevin

Re: HBase - REST API - Table Schema PUT vs POST

[Kevin Risden]

Looks like this might have triggered
https://issues.apache.org/jira/browse/HBASE-20581

Kevin Risden

On Mon, May 14, 2018 at 8:46 AM, Kevin Risden <kris...@apache.org> wrote:

> We are using HDP 2.5 with HBase 1.2.x. We think we found that the PUT vs
> POST documentation on the HBase book [1] website is incorrect.
>
> POST - Create a new table, or replace an existing table’s schema
>> PUT - Update an existing table with the provided schema fragment
>
>
> This contradicts what is in the original HBase 1.2 API javadocs [2].
>
> PUT //schema
>> POST //schema
>> Uploads table schema. PUT or POST creates table as necessary. PUT fully
>> replaces schema. POST modifies schema (add or modify column family). Supply
>> the full table schema for PUT or a well formed schema fragment for POST in
>> the desired encoding. Set Content-Type header to text/xml if the desired
>> encoding is XML. Set Content-Type header to application/json if the desired
>> encoding is JSON. Set Content-Type header to application/x-protobuf if the
>> desired encoding is protobufs. If not successful, returns appropriate HTTP
>> error status code. If successful, returns HTTP 200 status.
>
>
> The result of the two conflicting documentation pages is that PUT either
> updates or replaces and POST either updates or replaces the table schema.
> This can cause problems like setting the table max versions back to the
> default of 1.
>
> Does this make sense? Is it possible the documentation is incorrect here?
>
> The newest versions of HBase apidocs point to the HBase book. I have not
> checked if the behavior changed between HBase versions.
>
> 1. https://hbase.apache.org/book.html#_rest
> 2. https://hbase.apache.org/1.2/apidocs/org/apache/hadoop/
> hbase/rest/package-summary.html#operation_create_schema
>
> Kevin Risden
>

HBase - REST API - Table Schema PUT vs POST

[Kevin Risden]

We are using HDP 2.5 with HBase 1.2.x. We think we found that the PUT vs
POST documentation on the HBase book [1] website is incorrect.

POST - Create a new table, or replace an existing table’s schema
> PUT - Update an existing table with the provided schema fragment


This contradicts what is in the original HBase 1.2 API javadocs [2].

PUT //schema
> POST //schema
> Uploads table schema. PUT or POST creates table as necessary. PUT fully
> replaces schema. POST modifies schema (add or modify column family). Supply
> the full table schema for PUT or a well formed schema fragment for POST in
> the desired encoding. Set Content-Type header to text/xml if the desired
> encoding is XML. Set Content-Type header to application/json if the desired
> encoding is JSON. Set Content-Type header to application/x-protobuf if the
> desired encoding is protobufs. If not successful, returns appropriate HTTP
> error status code. If successful, returns HTTP 200 status.


The result of the two conflicting documentation pages is that PUT either
updates or replaces and POST either updates or replaces the table schema.
This can cause problems like setting the table max versions back to the
default of 1.

Does this make sense? Is it possible the documentation is incorrect here?

The newest versions of HBase apidocs point to the HBase book. I have not
checked if the behavior changed between HBase versions.

1. https://hbase.apache.org/book.html#_rest
2.
https://hbase.apache.org/1.2/apidocs/org/apache/hadoop/hbase/rest/package-summary.html#operation_create_schema

Kevin Risden

Re: HBase Thrift - HTTP - Kerberos & SPNEGO

[Kevin Risden]

"hopefully this week"... famous last words.

Finally got around to creating a JIRA: HBASE-19852 Close to having the
patch to submit done as well.

Kevin Risden

On Thu, Jan 11, 2018 at 10:02 AM, Kevin Risden <kris...@apache.org> wrote:

> "HBase Thrift2 "implementation" makes more sense to me"
>
> I agree with that statement since thrift2 follows the HBase API more
> closely. Thrift 1 follows the old HBase API. I don't think using Thrift2
> with Hue is an option right now. It still interacts with Thrift 1. (I'm not
> really looking to rewrite the Hue HBase Thrift module) There didn't look to
> be much code shared between Thrift 1 and Thrift 2 server implementations.
> Thrift 1 looks very much like HiveServer2 and the 401 bail out early might
> also apply there.
>
> I'll open a JIRA and throw up a patch hopefully this week.
>
> Kevin Risden
>
> On Thu, Jan 11, 2018 at 9:50 AM, Josh Elser <els...@apache.org> wrote:
>
>> Hey Kevin!
>>
>> Looks like you got some good changes in here.
>>
>> IMO, the HBase Thrift2 "implementation" makes more sense to me (I'm sure
>> there was a reason for having HTTP be involved at one point, but Thrift
>> today has the ability to do all of this RPC work for us). I'm not sure what
>> the HBase API implementations look like between the two versions.
>>
>> If you'd like to open up a JIRA and throw up a patch, you'd definitely
>> have my attention if no one else's :)
>>
>>
>> On 1/11/18 9:31 AM, Kevin Risden wrote:
>>
>>> I'm not 100% sure this should be posted to user list, but starting here
>>> before dev list/JIRA.
>>>
>>> I've been working on setting up the Hue HBase and it requires HBase
>>> Thrift
>>> v1 server. To support impersonation/proxyuser, the documentation states
>>> that this must be done with HTTP and not binary mode. The cluster has
>>> Kerberos and so the final setup ends up being HBase Thrift in HTTP mode
>>> with Kerberos.
>>>
>>> While setting up the HBase Thrift server with HTTP, there were a
>>> significant amount of 401 errors where the HBase Thrift wasn't able to
>>> handle the incoming Kerberos request. Documentation online is sparse when
>>> it comes to setting up the principal/keytab for HTTP Kerberos.
>>>
>>> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
>>> principal/keytab like other Thrift based servers (HiveServer2). It looks
>>> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
>>> were very close to the same at one point. I made the following changes to
>>> HBase Thrift v1 server implementation to make it work:
>>> * add SPNEGO principal/keytab if in HTTP mode
>>> * return 401 immediately if no authorization header instead of waiting
>>> for
>>> try/catch down in program flow
>>>
>>> The code changes are available here:
>>> https://github.com/risdenk/hortonworks-hbase-release/compare
>>> /HDP-2.5.3.126-base...fix_hbase_thrift_spnego
>>>
>>> Does this seem like the right approach?
>>>
>>> The same types of changes should apply to master as well. If this looks
>>> reasonable, I can create a JIRA and generate patch against Apache HBase
>>> master.
>>>
>>> Side note: I saw the notes about HBase Thrift v1 was meant to go away at
>>> some point but looks like it is still being depended on.
>>>
>>> Kevin Risden
>>>
>>>
>

Re: HBase Thrift - HTTP - Kerberos & SPNEGO

[Kevin Risden]

"HBase Thrift2 "implementation" makes more sense to me"

I agree with that statement since thrift2 follows the HBase API more
closely. Thrift 1 follows the old HBase API. I don't think using Thrift2
with Hue is an option right now. It still interacts with Thrift 1. (I'm not
really looking to rewrite the Hue HBase Thrift module) There didn't look to
be much code shared between Thrift 1 and Thrift 2 server implementations.
Thrift 1 looks very much like HiveServer2 and the 401 bail out early might
also apply there.

I'll open a JIRA and throw up a patch hopefully this week.

Kevin Risden

On Thu, Jan 11, 2018 at 9:50 AM, Josh Elser <els...@apache.org> wrote:

> Hey Kevin!
>
> Looks like you got some good changes in here.
>
> IMO, the HBase Thrift2 "implementation" makes more sense to me (I'm sure
> there was a reason for having HTTP be involved at one point, but Thrift
> today has the ability to do all of this RPC work for us). I'm not sure what
> the HBase API implementations look like between the two versions.
>
> If you'd like to open up a JIRA and throw up a patch, you'd definitely
> have my attention if no one else's :)
>
>
> On 1/11/18 9:31 AM, Kevin Risden wrote:
>
>> I'm not 100% sure this should be posted to user list, but starting here
>> before dev list/JIRA.
>>
>> I've been working on setting up the Hue HBase and it requires HBase Thrift
>> v1 server. To support impersonation/proxyuser, the documentation states
>> that this must be done with HTTP and not binary mode. The cluster has
>> Kerberos and so the final setup ends up being HBase Thrift in HTTP mode
>> with Kerberos.
>>
>> While setting up the HBase Thrift server with HTTP, there were a
>> significant amount of 401 errors where the HBase Thrift wasn't able to
>> handle the incoming Kerberos request. Documentation online is sparse when
>> it comes to setting up the principal/keytab for HTTP Kerberos.
>>
>> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
>> principal/keytab like other Thrift based servers (HiveServer2). It looks
>> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
>> were very close to the same at one point. I made the following changes to
>> HBase Thrift v1 server implementation to make it work:
>> * add SPNEGO principal/keytab if in HTTP mode
>> * return 401 immediately if no authorization header instead of waiting for
>> try/catch down in program flow
>>
>> The code changes are available here:
>> https://github.com/risdenk/hortonworks-hbase-release/compare
>> /HDP-2.5.3.126-base...fix_hbase_thrift_spnego
>>
>> Does this seem like the right approach?
>>
>> The same types of changes should apply to master as well. If this looks
>> reasonable, I can create a JIRA and generate patch against Apache HBase
>> master.
>>
>> Side note: I saw the notes about HBase Thrift v1 was meant to go away at
>> some point but looks like it is still being depended on.
>>
>> Kevin Risden
>>
>>

HBase Thrift - HTTP - Kerberos & SPNEGO

[Kevin Risden]

I'm not 100% sure this should be posted to user list, but starting here
before dev list/JIRA.

I've been working on setting up the Hue HBase and it requires HBase Thrift
v1 server. To support impersonation/proxyuser, the documentation states
that this must be done with HTTP and not binary mode. The cluster has
Kerberos and so the final setup ends up being HBase Thrift in HTTP mode
with Kerberos.

While setting up the HBase Thrift server with HTTP, there were a
significant amount of 401 errors where the HBase Thrift wasn't able to
handle the incoming Kerberos request. Documentation online is sparse when
it comes to setting up the principal/keytab for HTTP Kerberos.

I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
principal/keytab like other Thrift based servers (HiveServer2). It looks
like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
were very close to the same at one point. I made the following changes to
HBase Thrift v1 server implementation to make it work:
* add SPNEGO principal/keytab if in HTTP mode
* return 401 immediately if no authorization header instead of waiting for
try/catch down in program flow

The code changes are available here:
https://github.com/risdenk/hortonworks-hbase-release/compare/HDP-2.5.3.126-base...fix_hbase_thrift_spnego

Does this seem like the right approach?

The same types of changes should apply to master as well. If this looks
reasonable, I can create a JIRA and generate patch against Apache HBase
master.

Side note: I saw the notes about HBase Thrift v1 was meant to go away at
some point but looks like it is still being depended on.

Kevin Risden

Re: hbase datalocality while running mapreduce

[Kevin O'Dell]

Hi Rajeshkumar,

  When you execute a MR job over an HBase table, the reads should be
local(assuming you have good locality now). This is occurs because writes
are sent to the RegionServer that owns the particular Region. Once the
writes occur the block are written to the local DataNode first. Even if
locality gets messed up through RegionServer failures Major Compactions
should restore it.

On Tue, May 9, 2017 at 8:00 AM, Rajeshkumar J <rajeshkumarit8...@gmail.com>
wrote:

> Hi,
>
>
>I have a table with its region max file size is 10 gb. If i run any
> mapreduce over this table whether there will be any data locality. If not
> please explain
>
> Thanks
>



-- 
Kevin O'Dell
Field Engineer
850-496-1298 | ke...@rocana.com
@kevinrodell
<http://www.rocana.com>

Re: Compaction monitoring

[Kevin O'Dell]

Alexander,

  That is a great series of questions. What are you using for
instrumentation of your HBase cluster?  Cloudera Manager, Ambari, Ganglia,
Cacti, etc?  You are really asking a lot of performance based metric
questions. I don't think you will be able to answer your questions without
first being able to answer these questions:

Do you see the Major Compaction I/O/CPU/Memory spikes throughout the whole
"off-peak" window?

Do you have the host resources overhead to add additional compaction
threads to shorten it if so?

What do your responses times look like during your "off-peak hours" are you
still within your SLAs?

Answering these questions should quickly allow you to answer your first two
questions. Your last question is very interesting:

*how much degrades my performance if region size is becoming too large? <--
This is 100% depends, it depends on your environment, I/O usage, SLAs etc,
I am not sure if anyone has done documented compaction times based on
Region sizes.  You may have to do some trial and error here.

I hope this helps!



On Fri, May 5, 2017 at 8:47 AM, Alexander Ilyin <alexan...@weborama.com>
wrote:

> Hi,
>
> Tuning HBase performance I've found a lot of settings which affect
> compaction process (off-peak hours, time between compactions, compaction
> ratio, region sizes, etc.). They all seem to be useful and there are
> recommendations in the doc saying which values to set. But I found no way
> to assess how they actually affect my cluster performance, i.e. how much
> resources is taken by compaction and when. I would like to figure out which
> settings work best for my dataset and my specific workload but with only
> general recommendations in hand it seems difficult to do.
>
> For example, I have difficulties answering the following questions:
> * can I shorten my off-peak hours range?
> * can I afford to do compactions more often? or more aggressively?
> * how much degrades my performance if region size is becoming too large?
>
> HBase version I'm using is 1.1.2
>
>
> Alexander
>



-- 
Kevin O'Dell
Field Engineer
850-496-1298 | ke...@rocana.com
@kevinrodell
<http://www.rocana.com>

Re: Scan time increasing linearly

[Kevin O'Dell]

Hi Lydia,

  Welcome to the wonderful world of HBase! I don't think it is wrong that
you are seeing linear results from doing a scan.  When doing a scan HBase
will collect X amount of rows to return to the client. X being the value of
your scan cache. If each round trip grabs 100 rows and takes 1 second to do
it, then it is safe to assume time will grow in a linear nature.  The good
news is HBase is much faster than the example I gave.  I would recommend
looking at how much you are caching and raise that value, though I am not
surprised your scans are growing in a linear nature as the scan function is
rather linear itself. Does this make sense?

Also I may be completely wrong so I will defer to anyone else's expert
information.

On Wed, May 3, 2017 at 6:51 AM, Lydia <ickle...@googlemail.com> wrote:

> Hi,
>
> I would like to know if my query times seem appropriate since I do not
> have a lot experience with HBase.
>
> I have three tables - stored in HDFS, on one machine:
> table1: 5 million rows
> table2: 15 million rows
> table3: 90 million rows
>
> I do a scan using the Java API including a prefix-filter and some column
> filter.
> My rowkeys are encoded with geohashes.
>
> Execution Times:
> table1: ~   3.072 s
> table2: ~ 10.117 s
> table3: ~ 60.00 s
>
> It seems really odd to me that the execution time is increasing linear
> with the amount of rows!
> Am I doing something terribly wrong?
>
> Thanks in advance!
> Best regards,
> Lydia




-- 
Kevin O'Dell
Field Engineer
850-496-1298 | ke...@rocana.com
@kevinrodell
<http://www.rocana.com>

Re: how to optimize for heavy writes scenario

[Kevin O'Dell]

Hey Hef,

  What is the memstore size setting(how much heap is it allowed) that you
have on that cluster?  What is your region count per node?  Are you writing
evenly across all those regions or are only a few regions active per region
server at a time?  Can you paste your GC settings that you are currently
using?

On Fri, Mar 17, 2017 at 3:30 PM, Stack <st...@duboce.net> wrote:

> On Fri, Mar 17, 2017 at 9:31 AM, Hef <hef.onl...@gmail.com> wrote:
>
> > Hi group,
> > I'm using HBase to store large amount of time series data, the usage case
> > is heavy on writes then reads. My application stops at writing 600k
> > requests per second and I can't tune up for better tps.
> >
> > Hardware:
> > I have 6 Region Servers, each has 128G memory, 12 HDDs, 2cores with
> > 24threads,
> >
> > Schema:
> > The schema for these time series data is similar as OpenTSDB that the
> data
> > points of a same metric within an hour are store in one row, and there
> > could be maximum 3600 columns per row.
> > The cell is about 70bytes on its size, including the rowkey, column
> > qualifier, column family and value.
> >
> > HBase config:
> > CDH 5.6 HBase 1.0.0
> >
>
> Can you upgrade? There's a big diff between 1.2 and 1.0.
>
>
> > 100G memory for each RegionServer
> > hbase.hstore.compactionThreshold = 50
> > hbase.hstore.blockingStoreFiles = 100
> > hbase.hregion.majorcompaction disable
> > hbase.client.write.buffer = 20MB
> > hbase.regionserver.handler.count = 100
> >
>
> Could try halving the handler count.
>
>
> > hbase.hregion.memstore.flush.size = 128MB
> >
> >
> > Why are you flushing? If it is because you are hitting this flush limit,
> can you try upping it?
>
>
>
> > HBase Client:
> > write in BufferedMutator with 10/batch
> >
> > Inputs Volumes:
> > The input data throughput is more than 2millions/sec from Kafka
> >
> >
> How is the distribution? Evenly over the keyspace?
>
>
> > My writer applications are distributed, how ever I scaled them up, the
> > total write throughput won't get larger than 600K/sec.
> >
>
>
> Tell us more about this scaling up? How many writers?
>
>
>
> > The severs have 20% CPU usage and 5.6 wa,
> >
>
> 5.6 is high enough. Is the i/o spread over the disks?
>
>
>
> > GC  doesn't look good though, it shows a lot 10s+.
> >
> >
> What settings do you have?
>
>
>
> > In my opinion,  1M/s input data will result in only  70MByte/s write
> > throughput to the cluster, which is quite a small amount compare to the 6
> > region servers. The performance should not be bad like this.
> >
> > Is anybody has idea why the performance stops at 600K/s?
> > Is there anything I have to tune to increase the HBase write throughput?
> >
>
>
> If you double the clients writing do you see an up in the throughput?
>
> If you thread dump the servers, can you tell where they are held up? Or if
> they are doing any work at all relative?
>
> St.Ack
>



-- 
Kevin O'Dell
Field Engineer
850-496-1298 | ke...@rocana.com
@kevinrodell
<http://www.rocana.com>

Re: Optimizations for a Read-only database

[Kevin O'Dell]

Sorry for the Spam,

 As a follow up, if you are using large heaps with G1GC(~100GB) you can set
memstore to .1 and block cache to .8, though there might be a barrier for
how much you are allowed to set the systems to use. Leaving 10GB for
regular HBase operations is MORE than enough though.

On Fri, Mar 17, 2017 at 1:55 PM, Kevin O'Dell <ke...@rocana.com> wrote:

> Hi Jeff,
>
>   You can definitely lower the memstore, the last time I looked there it
> had to be set to .1 at lowest it could go. I would not recommend disabling
> compactions ever, bad things will occur and it can end up impacting your
> read performance greatly.  I would recommend looking at the Intel G1GC
> <https://software.intel.com/en-us/blogs/2014/06/18/part-1-tuning-java-garbage-collection-for-hbase>
> blog series to leverage really large chunks of block cache, and then using
> the remaining memory for off heap caching. You should make sure to turn on
> things like Snappy compression, FAST_DIFF for data block encoding, and with
> all the extra memory you will have available it might be worth using the
> ROW+COL bloom filters, though you should have very few underlying HFiles
> depending on how often you bulk load. I think short-circuit reads are on by
> default these days, but it will greatly speed up read performance if not
> already turned on. From an upfront design make sure you pre-split your
> tables so your first few bulk loads don't cause split and compaction
> pains.  Hope this helps!
>
> On Fri, Mar 17, 2017 at 1:32 PM, jeff saremi <jeffsar...@hotmail.com>
> wrote:
>
>> We're creating a readonly database and would like to know the recommended
>> optimizations we could do. We'd be loading data via direct write to HFiles.
>>
>> One thing i could immediately think of is to eliminate the memory for
>> Memstore. What is the minimum that we could get away with?
>>
>> How about disabling some regular operations to save CPU time. I think
>> Compaction is one of those we'd like to stop.
>>
>> thanks
>>
>> Jeff
>>
>
>
>
> --
> Kevin O'Dell
> Field Engineer
> 850-496-1298 <(850)%20496-1298> | ke...@rocana.com
> @kevinrodell
> <http://www.rocana.com>
>



-- 
Kevin O'Dell
Field Engineer
850-496-1298 | ke...@rocana.com
@kevinrodell
<http://www.rocana.com>

Re: Optimizations for a Read-only database

[Kevin O'Dell]

Hi Jeff,

  You can definitely lower the memstore, the last time I looked there it
had to be set to .1 at lowest it could go. I would not recommend disabling
compactions ever, bad things will occur and it can end up impacting your
read performance greatly.  I would recommend looking at the Intel G1GC
<https://software.intel.com/en-us/blogs/2014/06/18/part-1-tuning-java-garbage-collection-for-hbase>
blog series to leverage really large chunks of block cache, and then using
the remaining memory for off heap caching. You should make sure to turn on
things like Snappy compression, FAST_DIFF for data block encoding, and with
all the extra memory you will have available it might be worth using the
ROW+COL bloom filters, though you should have very few underlying HFiles
depending on how often you bulk load. I think short-circuit reads are on by
default these days, but it will greatly speed up read performance if not
already turned on. From an upfront design make sure you pre-split your
tables so your first few bulk loads don't cause split and compaction
pains.  Hope this helps!

On Fri, Mar 17, 2017 at 1:32 PM, jeff saremi <jeffsar...@hotmail.com> wrote:

> We're creating a readonly database and would like to know the recommended
> optimizations we could do. We'd be loading data via direct write to HFiles.
>
> One thing i could immediately think of is to eliminate the memory for
> Memstore. What is the minimum that we could get away with?
>
> How about disabling some regular operations to save CPU time. I think
> Compaction is one of those we'd like to stop.
>
> thanks
>
> Jeff
>



-- 
Kevin O'Dell
Field Engineer
850-496-1298 | ke...@rocana.com
@kevinrodell
<http://www.rocana.com>

Re: hbase java change baseZNode

[kevin]

val conf = HBaseConfiguration.create()
conf.set("hbase.zookeeper.property.clientPort", "2181")
conf.set("hbase.zookeeper.quorum", zkQuorum)
conf.set("zookeeper.znode.parent","/hbase-unsecure")

val connection = ConnectionFactory.createConnection(conf);

It's really worked.

2016-08-01 18:41 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:

> As mentioned in Kevin's first email, if /hbase-unsecure is the znode used
> by Ambari, setting zookeeper.znode.parent to hbase (or /hbase) wouldn't
> help.
>
> On Mon, Aug 1, 2016 at 3:39 AM, Adam Davidson <
> adam.david...@bigdatapartnership.com> wrote:
>
> > Hi Kevin,
> >
> > when creating the Configuration object for the HBase connection
> > (HBaseConfiguration.create()), you often need to set a number of
> properties
> > on the resulting object. The one you require in this case is
> > 'zookeeper.znode.parent'. Set that to 'hbase' and that should fix this
> > particular problem, though you may find other properties may need
> > attention. I believe it is all documented in the main Apache HBase docs.
> >
> > Regards,
> > Adam
> >
> > On Mon, 1 Aug 2016 at 11:36 kevin <kiss.kevin...@gmail.com> wrote:
> >
> > > hi,all:
> > > I install hbase by ambari ,I found it's zookeeper url is
> /hbase-unsecure
> > .
> > > when I use java api to connect to hbase ,program hung up .
> > > after kill it ,I found message :
> > > WARN ZKUtil: hconnection-0x4d1d2788-0x25617464bd80032,
> > > quorum=Centosle02:2181,Centosle03:2181,Centosle01:2181,
> baseZNode=/hbase
> > > Unable to get data of znode /hbase/meta-region-server
> > > java.lang.InterruptedException
> > >
> > > it read /hbase data not  /hbase-unsecure
> > >
> >
> > --
> >
> >
> > *We're hiring!*
> >  Please check out our current positions *here*
> > <https://www.bigdatapartnership.com/careers/>*.*
> > --
> >
> > *NOTICE AND DISCLAIMER*
> >
> > This email (including attachments) is confidential. If you are not the
> > intended recipient, notify the sender immediately, delete this email from
> > your system and do not disclose or use for any purpose.
> >
> > Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
> > Kingdom
> > Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE.
> United
> > Kingdom
> > Big Data Partnership Limited is a company registered in England & Wales
> > with Company No 7904824
> >
>

Re: hbase java change baseZNode

[kevin]

Thank you Adam Davidson.

2016-08-01 18:39 GMT+08:00 Adam Davidson <
adam.david...@bigdatapartnership.com>:

> Hi Kevin,
>
> when creating the Configuration object for the HBase connection
> (HBaseConfiguration.create()), you often need to set a number of properties
> on the resulting object. The one you require in this case is
> 'zookeeper.znode.parent'. Set that to 'hbase' and that should fix this
> particular problem, though you may find other properties may need
> attention. I believe it is all documented in the main Apache HBase docs.
>
> Regards,
> Adam
>
> On Mon, 1 Aug 2016 at 11:36 kevin <kiss.kevin...@gmail.com> wrote:
>
> > hi,all:
> > I install hbase by ambari ,I found it's zookeeper url is /hbase-unsecure
> .
> > when I use java api to connect to hbase ,program hung up .
> > after kill it ,I found message :
> > WARN ZKUtil: hconnection-0x4d1d2788-0x25617464bd80032,
> > quorum=Centosle02:2181,Centosle03:2181,Centosle01:2181, baseZNode=/hbase
> > Unable to get data of znode /hbase/meta-region-server
> > java.lang.InterruptedException
> >
> > it read /hbase data not  /hbase-unsecure
> >
>
> --
>
>
> *We're hiring!*
>  Please check out our current positions *here*
> <https://www.bigdatapartnership.com/careers/>*.*
> --
>
> *NOTICE AND DISCLAIMER*
>
> This email (including attachments) is confidential. If you are not the
> intended recipient, notify the sender immediately, delete this email from
> your system and do not disclose or use for any purpose.
>
> Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
> Kingdom
> Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United
> Kingdom
> Big Data Partnership Limited is a company registered in England & Wales
> with Company No 7904824
>

hbase java change baseZNode

[kevin]

hi,all:
I install hbase by ambari ,I found it's zookeeper url is /hbase-unsecure .
when I use java api to connect to hbase ,program hung up .
after kill it ,I found message :
WARN ZKUtil: hconnection-0x4d1d2788-0x25617464bd80032,
quorum=Centosle02:2181,Centosle03:2181,Centosle01:2181, baseZNode=/hbase
Unable to get data of znode /hbase/meta-region-server
java.lang.InterruptedException

it read /hbase data not  /hbase-unsecure

Re: May I run hbase on top of Alluxio/tacyon

[kevin]

0: jdbc:phoenix:master> select count(1) from STORE_SALES;
+--+
| COUNT(1) |
+--+
java.lang.RuntimeException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.hadoop.hbase.DoNotRetryIOException:
STORE_SALES,,1467706628930.ca35b82bd80c92d0d501c73956ef836f.: null
at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:84)
at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:52)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:205)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$52.call(RegionCoprocessorHost.java:1340)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1656)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1733)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1695)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(RegionCoprocessorHost.java:1335)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3250)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31068)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:105)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException
at
org.apache.hadoop.io.compress.snappy.SnappyDecompressor.setInput(SnappyDecompressor.java:111)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:104)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at
org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:426)
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:91)
at org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:508)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:398)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:540)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:588)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:287)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:201)
at
org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:316)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:260)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:740)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:715)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:540)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:142)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:4205)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4288)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4162)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:4149)
at
org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:284)
at
org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(BaseScannerRegionObserver.java:178)
... 12 more

at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
at sqlline.SqlLine.print(SqlLine.java:1653)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)

2016-06-21 9:15 GMT+08:00 kevin <kiss.kevin...@gmail.com>:

> I have worked out this questi

Re: May I run hbase on top of Alluxio/tacyon

[kevin]

I have worked out this question :
https://alluxio.atlassian.net/browse/ALLUXIO-2025

2016-06-20 21:02 GMT+08:00 Jean-Marc Spaggiari <jean-m...@spaggiari.org>:

> I think you might want to clean everything and retry. Clean the ZK /hbase
> content as well as your fs /hbase folder and restart...
>
> 2016-06-20 3:22 GMT-04:00 kevin <kiss.kevin...@gmail.com>:
>
> > *I got some error:*
> >
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.library.path=/home/dcos/hadoop-2.7.1/lib/native
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.io.tmpdir=/tmp
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.compiler=
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.name=Linux
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.arch=amd64
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.version=3.10.0-327.el7.x86_64
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.name=root
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.home=/root
> > 2016-06-20 14:50:45,453 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.dir=/home/dcos/hbase-0.98.16.1-hadoop2
> > 2016-06-20 14:50:45,454 INFO  [main] zookeeper.ZooKeeper: Initiating
> client
> > connection, connectString=slave1:2181,master:2181,slave2:2181
> > sessionTimeout=9
> > watcher=master:60x0, quorum=slave1:2181,master:2181,slave2:2181,
> > baseZNode=/hbase
> > 2016-06-20 14:50:45,490 INFO  [main-SendThread(slave2:2181)]
> > zookeeper.ClientCnxn: Opening socket connection to server slave2/
> > 10.1.3.177:2181. Will not attempt
> >  to authenticate using SASL (unknown error)
> > 2016-06-20 14:50:45,498 INFO  [main-SendThread(slave2:2181)]
> > zookeeper.ClientCnxn: Socket connection established to slave2/
> > 10.1.3.177:2181, initiating session
> > 2016-06-20 14:50:45,537 INFO  [main-SendThread(slave2:2181)]
> > zookeeper.ClientCnxn: Session establishment complete on server slave2/
> > 10.1.3.177:2181, sessionid =
> >  0x3556c8a93960004, negotiated timeout = 4
> > 2016-06-20 14:50:46,040 INFO  [RpcServer.responder] ipc.RpcServer:
> > RpcServer.responder: starting
> > 2016-06-20 14:50:46,043 INFO  [RpcServer.listener,port=6]
> > ipc.RpcServer: RpcServer.listener,port=6: starting
> > 2016-06-20 14:50:46,137 INFO  [master:master:6] mortbay.log: Logging
> to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2016-06-20 14:50:46,177 INFO  [master:master:6] http.HttpServer:
> Added
> > global filter 'safety'
> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> > 2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer:
> Added
> > filter static_user_filter
> > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
> > icUserFilter) to context master
> > 2016-06-20 14:50:46,180 INFO  [master:master:6] http.HttpServer:
> Added
> > filter static_user_filter
> > (class=org.apache.hadoop.http.lib.StaticUserWebFilter$Stat
> > icUserFilter) to context static
> > 2016-06-20 14:50:46,189 INFO  [master:master:6] http.HttpServer:
> Jetty
> > bound to port 60010
> > 2016-06-20 14:50:46,189 INFO  [master:master:6] mortbay.log:
> > jetty-6.1.26
> > 2016-06-20 14:50:46,652 INFO  [master:master:6] mortbay.log: Started
> > HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60010
> > 2016-06-20 14:50:47,122 INFO  [master:master:6]
> > master.ActiveMasterManager: Registered Active
> > Master=master,6,1466405444533
> > 2016-06-20 14:50:47,123 DEBUG [main-EventThread]
> > master.ActiveMasterManager: A master is now available
> > 2016-06-20 14:50:47,127 INFO  [master:master:6] logger.type:
> > getWorkingDirectory: /
> > 2016-06-20 14:50:47,128 INFO  [master:master:6]
> > Configuration.deprecation: fs.default.name is deprecated. Instead, use
> > fs.defaultFS
> > 2016-06-20 14:50:47,131 INFO  [master:master:6] logger.type:
> > getFileStatus(alluxio://master:19998/hbase)
> > 2016-06-20 14:50:47,153 INFO  [master:master:6] logger.type: Alluxio
> > client (version 1.1.1-SNAPSHOT) is trying to connect with
> > FileSystemMasterClient maste
> > r @ master/10.1.3.181:19998
> > 2016-06-20 14:50:47,159 INFO  [master:master:6] logger.type: Client
> &

Re: May I run hbase on top of Alluxio/tacyon

[kevin]

FO  [master:master:6] zookeeper.ZooKeeper:
Session: 0x3556c8a93960004 closed
2016-06-20 14:50:48,349 INFO  [master:master:6] master.HMaster: HMaster
main thread exiting
2016-06-20 14:50:48,349 ERROR [main] master.HMasterCommandLine: Master
exiting
java.lang.RuntimeException: HMaster Aborted
at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:201)
at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062)



can anybody tell me the root cause of this error? I think the info :
*"2016-06-20 14:50:48,180 INFO  [master:master:6] wal.FSHLog:
FileSystem's output stream doesn't support getNumCurrentReplicas;
--HDFS-826*
* not available; fsOut=alluxio.client.file.FileOutStream*
*2016-06-20 14:50:48,180 INFO  [master:master:6] wal.FSHLog:
FileSystem's output stream doesn't support getPipeline; not available;
fsOut*
*=alluxio.client.file.FileOutStream"*

*is important?*

2016-06-16 11:31 GMT+08:00 kevin <kiss.kevin...@gmail.com>:

> I want to test if run on alluxio could improve
> performance,because alluxio is a distribution filesystem top on memory and
>  under filesystem could be hdfs or s3 or something.
>
>
> 2016-06-16 10:32 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:
>
>> Since you already have hadoop 2.7.1, why is alluxio 1.1.0 needed ?
>>
>> Can you illustrate your use case ?
>>
>> Thanks
>>
>> On Wed, Jun 15, 2016 at 7:27 PM, kevin <kiss.kevin...@gmail.com> wrote:
>>
>> > hi,all:
>> >
>> > I wonder to know If run hbase on Alluxio/tacyon is possible and a good
>> > idea, and can anybody share the experience.,thanks.
>> > I will try hbase0.98.16 with hadoop2.7.1 on top of alluxio 1.1.0.
>> >
>>
>
>

Re: May I run hbase on top of Alluxio/tacyon

[kevin]

I want to test if run on alluxio could improve
performance,because alluxio is a distribution filesystem top on memory and
 under filesystem could be hdfs or s3 or something.


2016-06-16 10:32 GMT+08:00 Ted Yu <yuzhih...@gmail.com>:

> Since you already have hadoop 2.7.1, why is alluxio 1.1.0 needed ?
>
> Can you illustrate your use case ?
>
> Thanks
>
> On Wed, Jun 15, 2016 at 7:27 PM, kevin <kiss.kevin...@gmail.com> wrote:
>
> > hi,all:
> >
> > I wonder to know If run hbase on Alluxio/tacyon is possible and a good
> > idea, and can anybody share the experience.,thanks.
> > I will try hbase0.98.16 with hadoop2.7.1 on top of alluxio 1.1.0.
> >
>

May I run hbase on top of Alluxio/tacyon

[kevin]

hi,all:

I wonder to know If run hbase on Alluxio/tacyon is possible and a good
idea, and can anybody share the experience.,thanks.
I will try hbase0.98.16 with hadoop2.7.1 on top of alluxio 1.1.0.

Re: Slow sync cost

[Kevin Bowling]

Even G1GC will have a 100ms pause time which would trigger this warning.
Are there any real production clusters that don't constantly trigger this
warning?  What was the though process in 100ms?  When you go through
multiple JVMs that could be doing GCs over a network 100ms is not a long
time!  Spinning disks have tens of ms uncontested.  There's essentially
zero margin for normal operating latency.

On Wed, Apr 27, 2016 at 7:39 AM, Bryan Beaudreault <bbeaudrea...@hubspot.com
> wrote:

> We have 6 production clusters and all of them are tuned differently, so I'm
> not sure there is a setting I could easily give you. It really depends on
> the usage.  One of our devs wrote a blog post on G1GC fundamentals
> recently. It's rather long, but could be worth a read:
>
> http://product.hubspot.com/blog/g1gc-fundamentals-lessons-from-taming-garbage-collection
>
> We will also have a blog post coming out in the next week or so that talks
> specifically to tuning G1GC for HBase. I can update this thread when that's
> available.
>
> On Tue, Apr 26, 2016 at 8:08 PM Saad Mufti <saad.mu...@gmail.com> wrote:
>
> > That is interesting. Would it be possible for you to share what GC
> settings
> > you ended up on that gave you the most predictable performance?
> >
> > Thanks.
> >
> > 
> > Saad
> >
> >
> > On Tue, Apr 26, 2016 at 11:56 AM, Bryan Beaudreault <
> > bbeaudrea...@hubspot.com> wrote:
> >
> > > We were seeing this for a while with our CDH5 HBase clusters too. We
> > > eventually correlated it very closely to GC pauses. Through heavily
> > tuning
> > > our GC we were able to drastically reduce the logs, by keeping most
> GC's
> > > under 100ms.
> > >
> > > On Tue, Apr 26, 2016 at 6:25 AM Saad Mufti <saad.mu...@gmail.com>
> wrote:
> > >
> > > > From what I can see in the source code, the default is actually even
> > > lower
> > > > at 100 ms (can be overridden with
> hbase.regionserver.hlog.slowsync.ms
> > ).
> > > >
> > > > 
> > > > Saad
> > > >
> > > >
> > > > On Tue, Apr 26, 2016 at 3:13 AM, Kevin Bowling <
> > kevin.bowl...@kev009.com
> > > >
> > > > wrote:
> > > >
> > > > > I see similar log spam while system has reasonable performance.
> Was
> > > the
> > > > > 250ms default chosen with SSDs and 10ge in mind or something?  I
> > guess
> > > > I'm
> > > > > surprised a sync write several times through JVMs to 2 remote
> > datanodes
> > > > > would be expected to consistently happen that fast.
> > > > >
> > > > > Regards,
> > > > >
> > > > > On Mon, Apr 25, 2016 at 12:18 PM, Saad Mufti <saad.mu...@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In our large HBase cluster based on CDH 5.5 in AWS, we're
> > constantly
> > > > > seeing
> > > > > > the following messages in the region server logs:
> > > > > >
> > > > > > 2016-04-25 14:02:55,178 INFO
> > > > > > org.apache.hadoop.hbase.regionserver.wal.FSHLog: Slow sync cost:
> > 258
> > > > ms,
> > > > > > current pipeline:
> > > > > > [DatanodeInfoWithStorage[10.99.182.165:50010
> > > > > > ,DS-281d4c4f-23bd-4541-bedb-946e57a0f0fd,DISK],
> > > > > > DatanodeInfoWithStorage[10.99.182.236:50010
> > > > > > ,DS-f8e7e8c9-6fa0-446d-a6e5-122ab35b6f7c,DISK],
> > > > > > DatanodeInfoWithStorage[10.99.182.195:50010
> > > > > > ,DS-3beae344-5a4a-4759-ad79-a61beabcc09d,DISK]]
> > > > > >
> > > > > >
> > > > > > These happen regularly while HBase appear to be operating
> normally
> > > with
> > > > > > decent read and write performance. We do have occasional
> > performance
> > > > > > problems when regions are auto-splitting, and at first I thought
> > this
> > > > was
> > > > > > related but now I se it happens all the time.
> > > > > >
> > > > > >
> > > > > > Can someone explain what this means really and should we be
> > > concerned?
> > > > I
> > > > > > tracked down the source code that outputs it in
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
> > > > > >
> > > > > > but after going through the code I think I'd need to know much
> more
> > > > about
> > > > > > the code to glean anything from it or the associated JIRA ticket
> > > > > > https://issues.apache.org/jira/browse/HBASE-11240.
> > > > > >
> > > > > > Also, what is this "pipeline" the ticket and code talks about?
> > > > > >
> > > > > > Thanks in advance for any information and/or clarification anyone
> > can
> > > > > > provide.
> > > > > >
> > > > > > 
> > > > > >
> > > > > > Saad
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Slow sync cost

[Kevin Bowling]

I see similar log spam while system has reasonable performance.  Was the
250ms default chosen with SSDs and 10ge in mind or something?  I guess I'm
surprised a sync write several times through JVMs to 2 remote datanodes
would be expected to consistently happen that fast.

Regards,

On Mon, Apr 25, 2016 at 12:18 PM, Saad Mufti  wrote:

> Hi,
>
> In our large HBase cluster based on CDH 5.5 in AWS, we're constantly seeing
> the following messages in the region server logs:
>
> 2016-04-25 14:02:55,178 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Slow sync cost: 258 ms,
> current pipeline:
> [DatanodeInfoWithStorage[10.99.182.165:50010
> ,DS-281d4c4f-23bd-4541-bedb-946e57a0f0fd,DISK],
> DatanodeInfoWithStorage[10.99.182.236:50010
> ,DS-f8e7e8c9-6fa0-446d-a6e5-122ab35b6f7c,DISK],
> DatanodeInfoWithStorage[10.99.182.195:50010
> ,DS-3beae344-5a4a-4759-ad79-a61beabcc09d,DISK]]
>
>
> These happen regularly while HBase appear to be operating normally with
> decent read and write performance. We do have occasional performance
> problems when regions are auto-splitting, and at first I thought this was
> related but now I se it happens all the time.
>
>
> Can someone explain what this means really and should we be concerned? I
> tracked down the source code that outputs it in
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
>
> but after going through the code I think I'd need to know much more about
> the code to glean anything from it or the associated JIRA ticket
> https://issues.apache.org/jira/browse/HBASE-11240.
>
> Also, what is this "pipeline" the ticket and code talks about?
>
> Thanks in advance for any information and/or clarification anyone can
> provide.
>
> 
>
> Saad
>

Re: Balancing reads and writes

[Kevin Bowling]

Yeah here's the config I ended up with.  Out of the box it had really
severe blocking for write bursts, reads are much better with this and
handlers turned up a bit:

  
   hbase.ipc.server.callqueue.read.ratio
0.4 
  
   hbase.ipc.server.callqueue.scan.ratio
0.5 
  
   hbase.ipc.server.callqueue.handler.factor
0.5 

Regards,
Kevin

On Sat, Apr 16, 2016 at 9:27 PM, Vladimir Rodionov <vladrodio...@gmail.com>
wrote:

> There are separate RPC queues for read and writes in 1.0+ (not sure about
> 0.98). You need to set sizes of these queues accordingly.
>
> -Vlad
>
> On Sat, Apr 16, 2016 at 4:23 PM, Kevin Bowling <kevin.bowl...@kev009.com>
> wrote:
>
> > Hi,
> >
> > Using OpenTSDB 2.2 with its "appends" feature, I see significant impact
> on
> > read performance when writes are happening.  If a process injects a few
> > hundred thousand points in batch, the call queues on on the region
> servers
> > blow up and until they drain a new read request is basically blocked at
> the
> > end of the line.
> >
> > Any recommendations for keeping reads balanced vs writes?
> >
> > Regards,
> > Kevin
> >
>

Balancing reads and writes

[Kevin Bowling]

Hi,

Using OpenTSDB 2.2 with its "appends" feature, I see significant impact on
read performance when writes are happening.  If a process injects a few
hundred thousand points in batch, the call queues on on the region servers
blow up and until they drain a new read request is basically blocked at the
end of the line.

Any recommendations for keeping reads balanced vs writes?

Regards,
Kevin

Re: HBase 1.2 master CPU usage spin

[Kevin Bowling]

I confirm the fix, submitted a ports bump as
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=208739.

Regards,
Kevin

On Mon, Apr 11, 2016 at 7:09 AM, Matteo Bertozzi <theo.berto...@gmail.com>
wrote:

> that should be fixed in 1.2.1 with HBASE-15422
>
> Matteo
>
>
> On Mon, Apr 11, 2016 at 5:46 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Can you look at master log during this period to see what procedure was
> > retried ?
> >
> > Turning on DEBUG logging if necessary and pastebin relevant portion of
> > master log.
> >
> > Thanks
> >
> > > On Apr 11, 2016, at 1:11 AM, Kevin Bowling <kevin.bowl...@kev009.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > I'm running HBase 1.2.0 on FreeBSD via the ports system (
> > > http://www.freshports.org/databases/hbase/), and it is generally
> working
> > > well.  However, in an HA setup, the HBase master spins at 200% CPU
> usage
> > > when it is active and this follows the active master and disappears
> when
> > > standby.  Since this cluster is fairly idle, and an older production
> > Linux
> > > one uses much less master CPU, I am curious what is going on.
> > >
> > > Using visualvm, I can see that the ProcedureExecutor threads are quite
> > > busy.  Using dtrace, I can see that there is tons of native umtx
> > activity.
> > > I am guessing that some procedure is failing, and continuously retrying
> > as
> > > fast as the procedure dispatch allows.  Using dtrace, I can also see
> > that a
> > > lot of time seems to be spent in the JVM's 'libzip.so' native library.
> > I'm
> > > wondering if it's a classloader run amok or something.
> > >
> > > I need to rebuild the JVM with debugging to get more out of dtrace, but
> > the
> > > JVM doesn't implement a dtrace ustack helper for FreeBSD like it does
> on
> > > Solarish.  Hopefully then I can get some idea of what is going on.
> > >
> > > Does this speak to anyone for ideas to look into?  Other than noticing
> > the
> > > CPU usage in top, the master seems to function fine and is responsive.
> > >
> > > Here's a sample thread dump, this doesn't really jump out to me, but I
> > > don't know if any of my captures are at the right moment either:
> > >
> > > "ProcedureExecutor-7" - Thread t@116
> > >   java.lang.Thread.State: WAITING
> > >at sun.misc.Unsafe.park(Native Method)
> > >- parking to wait for <4653d04c> (a
> > > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > >at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > >at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> > >at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> > >at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> > >at
> > >
> >
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> > >at
> > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> > >at
> > >
> >
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:145)
> > >at
> > >
> >
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139)
> > >at
> > >
> >
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:803)
> > >at
> > >
> >
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
> > >at
> > >
> >
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494)
> > >
> > >   Locked ownable synchronizers:
> > >- None
> > >
> > > "ProcedureExecutor-5" - Thread t@114
> > >   java.lang.Thread.State: WAITING
> > >at sun.misc.Unsafe.park(Native Method)
> > >- parking to wait for <4653d04c> (a
> > > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> > >at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> > >at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheck

HBase 1.2 master CPU usage spin

[Kevin Bowling]

Hi,

I'm running HBase 1.2.0 on FreeBSD via the ports system (
http://www.freshports.org/databases/hbase/), and it is generally working
well.  However, in an HA setup, the HBase master spins at 200% CPU usage
when it is active and this follows the active master and disappears when
standby.  Since this cluster is fairly idle, and an older production Linux
one uses much less master CPU, I am curious what is going on.

Using visualvm, I can see that the ProcedureExecutor threads are quite
busy.  Using dtrace, I can see that there is tons of native umtx activity.
I am guessing that some procedure is failing, and continuously retrying as
fast as the procedure dispatch allows.  Using dtrace, I can also see that a
lot of time seems to be spent in the JVM's 'libzip.so' native library.  I'm
wondering if it's a classloader run amok or something.

I need to rebuild the JVM with debugging to get more out of dtrace, but the
JVM doesn't implement a dtrace ustack helper for FreeBSD like it does on
Solarish.  Hopefully then I can get some idea of what is going on.

Does this speak to anyone for ideas to look into?  Other than noticing the
CPU usage in top, the master seems to function fine and is responsive.

Here's a sample thread dump, this doesn't really jump out to me, but I
don't know if any of my captures are at the right moment either:

"ProcedureExecutor-7" - Thread t@116
   java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <4653d04c> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:145)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:803)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494)

   Locked ownable synchronizers:
- None

"ProcedureExecutor-5" - Thread t@114
   java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <4653d04c> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:145)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:803)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494)

   Locked ownable synchronizers:
- None

"ProcedureExecutor-4" - Thread t@113
   java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <4653d04c> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:145)
at

[ANNOUNCE] YCSB 0.7.0 Release

[Kevin Risden]

On behalf of the development community, I am pleased to announce the
release of YCSB 0.7.0.

Highlights:

* GemFire binding replaced with Apache Geode (incubating) binding
* Apache Solr binding was added
* OrientDB binding improvements
* HBase Kerberos support and use single connection
* Accumulo improvements
* JDBC improvements
* Couchbase scan implementation
* MongoDB improvements
* Elasticsearch version increase to 2.1.1

Full release notes, including links to source and convenience binaries:
https://github.com/brianfrankcooper/YCSB/releases/tag/0.7.0

This release covers changes from the last 1 month.

How to list the regions in an HBase table through the shell?

[Kevin Pauli]

I would like to get the same information about the regions of a table that
appear in the web UI (i.e. region name, region server, start/end key,
locality), but through the hbase shell.

(The UI is flaky/slow, and furthermore I want to process this information as
part of a script.)

After much googling, I can't find out how, and this surprises me immensely.
version is 1.0.0.-cdh5.4.0



--
View this message in context: 
http://apache-hbase.679495.n3.nabble.com/How-to-list-the-regions-in-an-HBase-table-through-the-shell-tp4076402.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: Rowkey design question

[Kevin O'dell]

 MB. A coprocessor
 do
   random
   access of 1-10 qualifiers per row.
  
   I would like to understand how HBase loads the data into
  memory.
   Will
   the
   entire row be loaded or only the qualifiers I ask for (like
   pointer
   access
   into a direct ByteBuffer) ?
  
   Cheers,
   -Kristoffer
  
   The opinions expressed here are mine, while they may reflect
 a
   cognitive
   thought, that is purely accidental.
   Use at your own risk.
   Michael Segel
   michael_segel (AT) hotmail.com
  
  
  
  
  
  
  
  
  
  
  
   The opinions expressed here are mine, while they may reflect a
   cognitive
   thought, that is purely accidental.
   Use at your own risk.
   Michael Segel
   michael_segel (AT) hotmail.com
  
  
  
  
  
  
  
   The opinions expressed here are mine, while they may reflect a
  cognitive
   thought, that is purely accidental.
   Use at your own risk.
   Michael Segel
   michael_segel (AT) hotmail.com
  
  
  
  
  
  
  
   The opinions expressed here are mine, while they may reflect a
 cognitive
   thought, that is purely accidental.
   Use at your own risk.
   Michael Segel
   michael_segel (AT) hotmail.com
  
  
  
  
  
  
 
  The opinions expressed here are mine, while they may reflect a cognitive
  thought, that is purely accidental.
  Use at your own risk.
  Michael Segel
  michael_segel (AT) hotmail.com
 
 
 
 
 
 


 --
 Best regards,

- Andy

 Problems worthy of attack prove their worth by hitting back. - Piet Hein
 (via Tom White)




-- 
Kevin O'Dell
Field Enablement, Cloudera

Re: introducing nodes w/ more storage

[Kevin O'dell]

Hi Mike,

  Sorry for the delay here.

How does the HDFS load balancer impact the load balancing of HBase? -- The
HDFS load balancer is not automatically run, it is a manual process that is
kicked off. It is not recommended to *ever run the HDFS balancer on a
cluster running HBase.  Similar to have HBase has no concept or care about
the underlying storage, HDFS has no concept or care of the region layout,
nor the locality we worked so hard to build through compactions.

Furthermore, once the HDFS balancer has saved us from running out of space
on the smaller nodes, we will run a major compaction, and re-write all of
the HBase data right back to where it was before.

one is the number of regions managed by a region server that’s HBase’s
load, right? And then there’s the data distribution of HBase files that is
really managed by HDFS load balancer, right? --- Right, until we run major
compaction and restore locality by moving the data back

Even still… eventually the data will be distributed equally across the
cluster. What’s happening with the HDFS balancer?  Is that heterogenous or
homogenous in terms of storage? -- Not quite, as I said before the HDFS
balancer is manual, so it is quite easy to build up a skew, especially if
you use a datanode as an edge node or thrift gateway etc.  Yes, the HDFS
balancer is heterogenous, but it doesn't play nice with HBase.

*The use of the word ever should not be construed as a true definitive.
Ever is being used to represent a best practice.  In many cases the HDFS
balancer needs to be run, especially in multi-tenant clusters
with archive data.  It is best to immediately run a major compaction to
restore HBase locality if the HDFS balancer is used.

On Mon, Mar 23, 2015 at 10:50 AM, Michael Segel michael_se...@hotmail.com
wrote:

 @lars,

 How does the HDFS load balancer impact the load balancing of HBase?

 Of course there are two loads… one is the number of regions managed by a
 region server that’s HBase’s load, right?
 And then there’s the data distribution of HBase files that is really
 managed by HDFS load balancer, right?

 OP’s question is having a heterogenous cluster where he would like to see
 a more even distribution of data/free space based on the capacity of the
 newer machines in the cluster.

 This is a storage question, not a memory/cpu core question.

 Or am I missing something?


 -Mike

  On Mar 22, 2015, at 10:56 PM, lars hofhansl la...@apache.org wrote:
 
  Seems that it should not be too hard to add that to the stochastic load
 balancer.
  We could add a spaceCost or something.
 
 
 
  - Original Message -
  From: Jean-Marc Spaggiari jean-m...@spaggiari.org
  To: user user@hbase.apache.org
  Cc: Development developm...@mentacapital.com
  Sent: Thursday, March 19, 2015 12:55 PM
  Subject: Re: introducing nodes w/ more storage
 
  You can extend the default balancer and assign the regions based on
  that.But at the end, the replicated blocks might still go all over the
  cluster and your small nodes are going to be full and will not be able
 to
  get anymore writes even for the regions they are supposed to get.
 
  I'm not sure there is a good solution for what you are looking for :(
 
  I build my own balancer but because of differences in the CPUs, not
 because
  of differences of the storage space...
 
 
  2015-03-19 15:50 GMT-04:00 Nick Dimiduk ndimi...@gmail.com:
 
  Seems more fantasy than fact, I'm afraid. The default load balancer [0]
  takes store file size into account, but has no concept of capacity. It
  doesn't know that nodes in a heterogenous environment have different
  capacity.
 
  This would be a good feature to add though.
 
  [0]:
 
 
 https://github.com/apache/hbase/blob/branch-1.0/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java
 
  On Tue, Mar 17, 2015 at 7:26 AM, Ted Tuttle t...@mentacapital.com
 wrote:
 
  Hello-
 
  Sometime back I asked a question about introducing new nodes w/ more
  storage that existing nodes.  I was told at the time that HBase will
 not
  be
  able to utilize the additional storage; I assumed at the time that
  regions
  are allocated to nodes in something like a round-robin fashion and the
  node
  with the least storage sets the limit for how much each node can
 utilize.
 
  My question this time around has to do with nodes w/ unequal numbers of
  volumes: Does HBase allocate regions based on nodes or volumes on the
  nodes?  I am hoping I can add a node with 8 volumes totaling 8X TB and
  all
  the volumes will be filled.  This even though legacy nodes have 5
 volumes
  and total storage of 5X TB.
 
  Fact or fantasy?
 
  Thanks,
  Ted
 
 
 
 

 The opinions expressed here are mine, while they may reflect a cognitive
 thought, that is purely accidental.
 Use at your own risk.
 Michael Segel
 michael_segel (AT) hotmail.com








-- 
Kevin O'Dell
Field Enablement, Cloudera

Re: introducing nodes w/ more storage

[Kevin O'dell]

Mike,

  I agree with all of the above, I am just saying from experience, even
clusters that do not run HBase at all rarely run the HDFS balancer except
when doing major overhauls such as adding nodes/racks.


And no, you do not use a data node as an edge node.
(Really saying that? C’mon, really? ) Never a good design. Ever. --
Sometimes you have to make do with what you got :)


On Thu, Apr 2, 2015 at 10:33 AM, Michael Segel michael_se...@hotmail.com
wrote:



 When you say … It is not recommended to *ever run the HDFS balancer on a
 cluster running HBase. “ … thats a very scary statement.

 Not really a good idea.  Unless you are building a cluster for a specific
 use case.


 When you look at the larger picture… in most use cases, the cluster will
 contain more data in flat files (HDFS) than they would inside HBase.
 (which you allude to in you last paragraph) so balancing is a good idea.
 (Even manual processes can be run in cron jobs ;-)

 And no, you do not use a data node as an edge node.
 (Really saying that? C’mon, really? ) Never a good design. Ever.


 I agree that you should run major compactions after running the load
 balancer. (HDFS)
 But the point I am trying to make is that with respect to HBase, you still
 need to think about the cluster as a whole.


  On Apr 2, 2015, at 7:41 AM, Kevin O'dell kevin.od...@cloudera.com
 wrote:
 
  Hi Mike,
 
   Sorry for the delay here.
 
  How does the HDFS load balancer impact the load balancing of HBase? --
 The
  HDFS load balancer is not automatically run, it is a manual process that
 is
  kicked off. It is not recommended to *ever run the HDFS balancer on a
  cluster running HBase.  Similar to have HBase has no concept or care
 about
  the underlying storage, HDFS has no concept or care of the region layout,
  nor the locality we worked so hard to build through compactions.
 
  Furthermore, once the HDFS balancer has saved us from running out of
 space
  on the smaller nodes, we will run a major compaction, and re-write all of
  the HBase data right back to where it was before.
 
  one is the number of regions managed by a region server that’s HBase’s
  load, right? And then there’s the data distribution of HBase files that
 is
  really managed by HDFS load balancer, right? --- Right, until we run
 major
  compaction and restore locality by moving the data back
 
  Even still… eventually the data will be distributed equally across the
  cluster. What’s happening with the HDFS balancer?  Is that heterogenous
 or
  homogenous in terms of storage? -- Not quite, as I said before the HDFS
  balancer is manual, so it is quite easy to build up a skew, especially if
  you use a datanode as an edge node or thrift gateway etc.  Yes, the HDFS
  balancer is heterogenous, but it doesn't play nice with HBase.
 
  *The use of the word ever should not be construed as a true definitive.
  Ever is being used to represent a best practice.  In many cases the HDFS
  balancer needs to be run, especially in multi-tenant clusters
  with archive data.  It is best to immediately run a major compaction to
  restore HBase locality if the HDFS balancer is used.
 
  On Mon, Mar 23, 2015 at 10:50 AM, Michael Segel 
 michael_se...@hotmail.com
  wrote:
 
  @lars,
 
  How does the HDFS load balancer impact the load balancing of HBase?
 
  Of course there are two loads… one is the number of regions managed by a
  region server that’s HBase’s load, right?
  And then there’s the data distribution of HBase files that is really
  managed by HDFS load balancer, right?
 
  OP’s question is having a heterogenous cluster where he would like to
 see
  a more even distribution of data/free space based on the capacity of the
  newer machines in the cluster.
 
  This is a storage question, not a memory/cpu core question.
 
  Or am I missing something?
 
 
  -Mike
 
  On Mar 22, 2015, at 10:56 PM, lars hofhansl la...@apache.org wrote:
 
  Seems that it should not be too hard to add that to the stochastic load
  balancer.
  We could add a spaceCost or something.
 
 
 
  - Original Message -
  From: Jean-Marc Spaggiari jean-m...@spaggiari.org
  To: user user@hbase.apache.org
  Cc: Development developm...@mentacapital.com
  Sent: Thursday, March 19, 2015 12:55 PM
  Subject: Re: introducing nodes w/ more storage
 
  You can extend the default balancer and assign the regions based on
  that.But at the end, the replicated blocks might still go all over the
  cluster and your small nodes are going to be full and will not be
 able
  to
  get anymore writes even for the regions they are supposed to get.
 
  I'm not sure there is a good solution for what you are looking for :(
 
  I build my own balancer but because of differences in the CPUs, not
  because
  of differences of the storage space...
 
 
  2015-03-19 15:50 GMT-04:00 Nick Dimiduk ndimi...@gmail.com:
 
  Seems more fantasy than fact, I'm afraid. The default load balancer
 [0]
  takes store file size into account, but has no concept

Re: how to explain read/write performance change after modifying the hfile.block.cache.size?

[Kevin O'dell]

I am also under the impression that HBase reads should basically not work
with block cache set to 0 since we store the indexes in block cache right?

On Thu, Nov 20, 2014 at 3:31 PM, lars hofhansl la...@apache.org wrote:

 That would explain it if memstores are flushed due to global memory
 pressure.

 But cache and memstore size are (unfortunately) configured independently.
 The memstore heap portion would be 40% (by default) in either case.So this
 is a bit curious still.
 Ming, can you tell us more details?- RAM on the boxes- heap setup for the
 region servers- any other relevant settings on hbase-site.xml- configs on
 the table/column family you're writing to (like bloom filters, etc).

 That would help us diagnose this.

 -- Lars

   From: Ted Yu yuzhih...@gmail.com
  To: user@hbase.apache.org user@hbase.apache.org
  Sent: Thursday, November 20, 2014 9:32 AM
  Subject: Re: how to explain read/write performance change after modifying
 the hfile.block.cache.size?

 When block cache size increases from 0 to 0.4, the amount of heap given to
 memstore decreases. This would slow down the writes.
 Please see:
 http://hbase.apache.org/book.html#store.memstore

 For your second question, see this thread:

 http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflushsubj=Re+Clarifications+on+HBase+Durability

 Cheers

 On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) ming.l...@hp.com
 wrote:

  Hello, all,
 
  I am playing with YCSB to test HBase performance. I am using HBase
 0.98.5.
  I tried to adjust the hfile.block.cache.size to see the difference, when
 I
  set hfile.block.cache.size to 0, read performance is very bad, but write
  performance is very very very good; when I set hfile.block.cache.size
  to 0.4, read is better, but write performance drop dramatically. I
 disable
  the client side writebuffer already.
  This is hard to understand for me:
  The HBase guide just said hfile.block.cache.size setting is about how
 much
  memory used as block cache used by StoreFile. I have no idea of how HBase
  works internally. Typically, it is easy to understand that increase the
  size of cache should help the read, but why it will harm the write
  operation? The write performance down from 30,000 to 4,000 for your
  reference, just by changing the hfile.block.cache.size from 0 to 0.4.
  Could anyone give me a brief explanation about this observation or give
 me
  some advices about what to study to understand what is block cache used
 for?
 
  Another question: HBase write will first write to WAL then to memstore.
  Will the write to WAL go to disk directly before hbase write memstore, a
  sync operation or it is possible that write to WAL is still buffered
  somewhere when hbase put the data into the memstore?
 
  Reading src code may cost me months, so a kindly reply will help me a
  lot... ...
  Thanks very much!
 
  Best Regards,
  Ming
 







-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: how to explain read/write performance change after modifying the hfile.block.cache.size?

[Kevin O'dell]

until you kill all the cache right?  Or was this an old JIRA I was thinking
of?

On Thu, Nov 20, 2014 at 3:37 PM, Ted Yu yuzhih...@gmail.com wrote:

 The indices are always cached.

 Cheers

 On Nov 20, 2014, at 12:33 PM, Kevin O'dell kevin.od...@cloudera.com
 wrote:

  I am also under the impression that HBase reads should basically not work
  with block cache set to 0 since we store the indexes in block cache
 right?
 
  On Thu, Nov 20, 2014 at 3:31 PM, lars hofhansl la...@apache.org wrote:
 
  That would explain it if memstores are flushed due to global memory
  pressure.
 
  But cache and memstore size are (unfortunately) configured
 independently.
  The memstore heap portion would be 40% (by default) in either case.So
 this
  is a bit curious still.
  Ming, can you tell us more details?- RAM on the boxes- heap setup for
 the
  region servers- any other relevant settings on hbase-site.xml- configs
 on
  the table/column family you're writing to (like bloom filters, etc).
 
  That would help us diagnose this.
 
  -- Lars
 
   From: Ted Yu yuzhih...@gmail.com
  To: user@hbase.apache.org user@hbase.apache.org
  Sent: Thursday, November 20, 2014 9:32 AM
  Subject: Re: how to explain read/write performance change after
 modifying
  the hfile.block.cache.size?
 
  When block cache size increases from 0 to 0.4, the amount of heap given
 to
  memstore decreases. This would slow down the writes.
  Please see:
  http://hbase.apache.org/book.html#store.memstore
 
  For your second question, see this thread:
 
 
 http://search-hadoop.com/m/DHED4TEvBy1/lars+hbase+hflushsubj=Re+Clarifications+on+HBase+Durability
 
  Cheers
 
  On Thu, Nov 20, 2014 at 8:05 AM, Liu, Ming (HPIT-GADSC) 
 ming.l...@hp.com
  wrote:
 
  Hello, all,
 
  I am playing with YCSB to test HBase performance. I am using HBase
  0.98.5.
  I tried to adjust the hfile.block.cache.size to see the difference,
 when
  I
  set hfile.block.cache.size to 0, read performance is very bad, but
 write
  performance is very very very good; when I set
 hfile.block.cache.size
  to 0.4, read is better, but write performance drop dramatically. I
  disable
  the client side writebuffer already.
  This is hard to understand for me:
  The HBase guide just said hfile.block.cache.size setting is about how
  much
  memory used as block cache used by StoreFile. I have no idea of how
 HBase
  works internally. Typically, it is easy to understand that increase the
  size of cache should help the read, but why it will harm the write
  operation? The write performance down from 30,000 to 4,000 for your
  reference, just by changing the hfile.block.cache.size from 0 to 0.4.
  Could anyone give me a brief explanation about this observation or give
  me
  some advices about what to study to understand what is block cache used
  for?
 
  Another question: HBase write will first write to WAL then to memstore.
  Will the write to WAL go to disk directly before hbase write memstore,
 a
  sync operation or it is possible that write to WAL is still buffered
  somewhere when hbase put the data into the memstore?
 
  Reading src code may cost me months, so a kindly reply will help me a
  lot... ...
  Thanks very much!
 
  Best Regards,
  Ming
 
 
 
  --
  Kevin O'Dell
  Systems Engineer, Cloudera




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Avoid GC Pauses on Scan MapReduces

[Kevin O'dell]

Hi Pere,

  I have a few thoughts here:

1.) I would make sure you have these settings - -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled
-XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled

2.) Does this happen with a lower scanner.caching?

3.) Did it go away when you turned the block cache off?

4.) Do you have MSLab turned on?

On Thu, Nov 13, 2014 at 10:12 PM, Dhaval Shah prince_mithi...@yahoo.co.in
wrote:

 You can do scan.setCacheBlocks (false) To disable scanner caching on map
 reduce scans. Also use parallel GC for new generation.  That will help
 reduce stop the world pauses with cms


 Sent from my T-Mobile 4G LTE Device


  Original message 
 From: Pere Kyle p...@whisper.sh
 Date:11/13/2014  8:54 PM  (GMT-05:00)
 To: user@hbase.apache.org
 Cc:
 Subject: Avoid GC Pauses on Scan MapReduces

 Hi there,

 Recently I have been experiencing instability when scanning our HBASE
 cluster. The table we are trying to scan is 1.5B records 1TB, we have 12GB
 heap and 17 servers. Our GC options are as so:
 -XX:OnOutOfMemoryError=kill -9 %p -Xmx12000m -XX:+UseConcMarkSweepGC
 -Xmx12g -Xmx12g

 The errors I am seeing in the logs:
 2014-11-14 00:00:32,656 WARN org.apache.hadoop.hbase.util.Sleeper
 (regionserver60020): We slept 17355ms instead of 3000ms, this is likely due
 to a long garbage collecting pause and it's usually bad, see
 http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
 2014-11-14 00:00:32,663 WARN org.apache.hadoop.hbase.util.Sleeper
 (regionserver60020.periodicFlusher): We slept 24734ms instead of 1ms,
 this is likely due to a long garbage collecting pause and it's usually bad,
 see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired

 We are using hbase.client.scanner.caching=1000. I suspect this may be a
 block cache issue. My question is if/how to disable the block cache for the
 scan queries? This is taking out writes and causing instability on the
 cluster.

 Thanks,
 Pere




-- 
Kevin O'Dell
Systems Engineer, Cloudera

HBase client machine with multiple interfaces

[Kevin]

Hi,

I have connected a client machine with two network interfaces to an
internal, isolated HBase cluster and an external network. The HBase cluster
is on its own private LAN, away from the external network. After installing
and updating the Hadoop and HBase configuration files on the client
machine, I can run the HBase shell and look at tables, but
TableInputFormatBase can't find any servers using its DNS checks. The
machine is on the DNS for the external network so, from my understanding,
the HBase client API will use this DNS to try and find the HBase cluster. I
know there are master and regionserver DNS interface configurations, but
are there anything for client libraries? My use case is running Pig scripts
from the client machine. Oddly enough, sometimes a Pig script works, other
times (most times) I get the NullPointerException from the
org.apache.hadoop.net.DNS.reverseDns call.

Any ideas?

Thanks,
Kevin

Re: custom filter on hbase 0.96

[Kevin]

Matt,

You should create your own proto file and compile that with the Google
Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
code:
https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327

You will need to override `public byte[] toByteArray()` and `public static
Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should be
the byte[] from serializing with the protocol buffer. This will also be the
input to parseFrom(byte[]), which is called using reflection on the
server-side to instantiate your custom filter and use it.

On Sun, Oct 19, 2014 at 11:31 AM, Matt K matvey1...@gmail.com wrote:

 Anyone?

 On Thursday, October 16, 2014, Matt K matvey1...@gmail.com wrote:

  Hi, can anyone help with above? Feels like I'm missing something obvious.
 
  On Wednesday, October 15, 2014, Nishanth S nishanth.2...@gmail.com
  javascript:_e(%7B%7D,'cvml','nishanth.2...@gmail.com'); wrote:
 
  Thanks Ted .I will take a look.
 
  -Nishanth
 
  On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu yuzhih...@gmail.com wrote:
 
   Nishanth:
   Good question.
  
   As a general coding guide, writing unit test is always a good start.
  Using
   Matt's case as an example, take a look at TestPrefixFilter.
  
   There're various unit tests for Filters in hbase code.
  
   Cheers
  
   On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S nishanth.2...@gmail.com
   wrote:
  
Hi Ted ,
 Since I am also working on similar thing is there a way we can
 first
   test
the filter on client side?.You know what I  mean without disrupting
   others
who are using the same cluster for other work?
   
Thanks,
Nishanth
   
On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu yuzhih...@gmail.com
 wrote:
   
 bq. Or create a new file, compile it into ...

 You should go with the above approach.

 On Wed, Oct 15, 2014 at 2:08 PM, Matt K matvey1...@gmail.com
  wrote:

  Hi all,
 
  I'm trying to get a custom filter to work on HBase 0.96. After
  some
  searching, I found that starting from 0.96, the implementer is
   required
 to
  implement toByteArray and parseFrom methods, using Protocol
Buffers.
  But I'm having trouble with the how.
 
  The proto file for the existing filters is located here:
 
 

   
  
 
 https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
 
  Am I supposed to modify that file? Or create a new file, compile
  it
into
  Java, and package it up with the filter?
 
  In the meantime, I've taken a shortcut that's not working.
 Here's
  my
 code:
  http://pastebin.com/iHFKu9Xz
 
  I'm using PrefixFilter, which comes with HBase, since I'm also
 filtering
  by prefix. However, that errors out with the following:
  http://pastebin.com/zBg47p6Z
 
  Thanks in advance for helping!
 
  -Matt
 

   
  
 
 
 
  --
  www.calcmachine.com - easy online calculator.
 


 --
 www.calcmachine.com - easy online calculator.

Re: HBase client machine with multiple interfaces

[Kevin]

All machines use ipv4

On Tue, Oct 21, 2014 at 1:36 PM, Ted Yu yuzhih...@gmail.com wrote:

 Do you use ipv6 ?

 If so, this is related:
 HBASE-12115

 Cheers

 On Tue, Oct 21, 2014 at 10:26 AM, Kevin kevin.macksa...@gmail.com wrote:

  Hi,
 
  I have connected a client machine with two network interfaces to an
  internal, isolated HBase cluster and an external network. The HBase
 cluster
  is on its own private LAN, away from the external network. After
 installing
  and updating the Hadoop and HBase configuration files on the client
  machine, I can run the HBase shell and look at tables, but
  TableInputFormatBase can't find any servers using its DNS checks. The
  machine is on the DNS for the external network so, from my understanding,
  the HBase client API will use this DNS to try and find the HBase
 cluster. I
  know there are master and regionserver DNS interface configurations, but
  are there anything for client libraries? My use case is running Pig
 scripts
  from the client machine. Oddly enough, sometimes a Pig script works,
 other
  times (most times) I get the NullPointerException from the
  org.apache.hadoop.net.DNS.reverseDns call.
 
  Any ideas?
 
  Thanks,
  Kevin
 

Re: custom filter on hbase 0.96

[Kevin]

BTW, the error looks like you didn't distribute your custom filter to your
region servers.

On Tue, Oct 21, 2014 at 1:34 PM, Kevin kevin.macksa...@gmail.com wrote:

 Matt,

 You should create your own proto file and compile that with the Google
 Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
 code:
 https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327

 You will need to override `public byte[] toByteArray()` and `public static
 Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should be
 the byte[] from serializing with the protocol buffer. This will also be the
 input to parseFrom(byte[]), which is called using reflection on the
 server-side to instantiate your custom filter and use it.

 On Sun, Oct 19, 2014 at 11:31 AM, Matt K matvey1...@gmail.com wrote:

 Anyone?

 On Thursday, October 16, 2014, Matt K matvey1...@gmail.com wrote:

  Hi, can anyone help with above? Feels like I'm missing something
 obvious.
 
  On Wednesday, October 15, 2014, Nishanth S nishanth.2...@gmail.com
  javascript:_e(%7B%7D,'cvml','nishanth.2...@gmail.com'); wrote:
 
  Thanks Ted .I will take a look.
 
  -Nishanth
 
  On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu yuzhih...@gmail.com wrote:
 
   Nishanth:
   Good question.
  
   As a general coding guide, writing unit test is always a good start.
  Using
   Matt's case as an example, take a look at TestPrefixFilter.
  
   There're various unit tests for Filters in hbase code.
  
   Cheers
  
   On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S nishanth.2...@gmail.com
 
   wrote:
  
Hi Ted ,
 Since I am also working on similar thing is there a way we can
 first
   test
the filter on client side?.You know what I  mean without disrupting
   others
who are using the same cluster for other work?
   
Thanks,
Nishanth
   
On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu yuzhih...@gmail.com
 wrote:
   
 bq. Or create a new file, compile it into ...

 You should go with the above approach.

 On Wed, Oct 15, 2014 at 2:08 PM, Matt K matvey1...@gmail.com
  wrote:

  Hi all,
 
  I'm trying to get a custom filter to work on HBase 0.96. After
  some
  searching, I found that starting from 0.96, the implementer is
   required
 to
  implement toByteArray and parseFrom methods, using Protocol
Buffers.
  But I'm having trouble with the how.
 
  The proto file for the existing filters is located here:
 
 

   
  
 
 https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
 
  Am I supposed to modify that file? Or create a new file,
 compile
  it
into
  Java, and package it up with the filter?
 
  In the meantime, I've taken a shortcut that's not working.
 Here's
  my
 code:
  http://pastebin.com/iHFKu9Xz
 
  I'm using PrefixFilter, which comes with HBase, since I'm
 also
 filtering
  by prefix. However, that errors out with the following:
  http://pastebin.com/zBg47p6Z
 
  Thanks in advance for helping!
 
  -Matt
 

   
  
 
 
 
  --
  www.calcmachine.com - easy online calculator.
 


 --
 www.calcmachine.com - easy online calculator.

Re: custom filter on hbase 0.96

[Kevin]

I haven't tried dynamic loading of filters on RS, but I know it does exist.
See https://issues.apache.org/jira/browse/HBASE-9301.

If you still can't get it to work, then I suggest distributing your filters
to the RS and restart them. Let us know how everything works out.

On Tue, Oct 21, 2014 at 9:02 PM, Matt K matvey1...@gmail.com wrote:

 Thanks Kevin!

 I was under impression, probably mistakingly, that as of 0.96 placing
 the filter on hdfs under hbase lib directory is sufficient and RS should
 load the filter dynamically from hdfs. Is that not the case?

 On Tuesday, October 21, 2014, Kevin kevin.macksa...@gmail.com wrote:

  BTW, the error looks like you didn't distribute your custom filter to
 your
  region servers.
 
  On Tue, Oct 21, 2014 at 1:34 PM, Kevin kevin.macksa...@gmail.com
  javascript:; wrote:
 
   Matt,
  
   You should create your own proto file and compile that with the Google
   Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
   code:
  
 
 https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
  
   You will need to override `public byte[] toByteArray()` and `public
  static
   Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should
 be
   the byte[] from serializing with the protocol buffer. This will also be
  the
   input to parseFrom(byte[]), which is called using reflection on the
   server-side to instantiate your custom filter and use it.
  
   On Sun, Oct 19, 2014 at 11:31 AM, Matt K matvey1...@gmail.com
  javascript:; wrote:
  
   Anyone?
  
   On Thursday, October 16, 2014, Matt K matvey1...@gmail.com
  javascript:; wrote:
  
Hi, can anyone help with above? Feels like I'm missing something
   obvious.
   
On Wednesday, October 15, 2014, Nishanth S nishanth.2...@gmail.com
  javascript:;
javascript:_e(%7B%7D,'cvml','nishanth.2...@gmail.com
 javascript:;');
  wrote:
   
Thanks Ted .I will take a look.
   
-Nishanth
   
On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu yuzhih...@gmail.com
  javascript:; wrote:
   
 Nishanth:
 Good question.

 As a general coding guide, writing unit test is always a good
  start.
Using
 Matt's case as an example, take a look at TestPrefixFilter.

 There're various unit tests for Filters in hbase code.

 Cheers

 On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S 
  nishanth.2...@gmail.com javascript:;
   
 wrote:

  Hi Ted ,
   Since I am also working on similar thing is there a way we can
   first
 test
  the filter on client side?.You know what I  mean without
  disrupting
 others
  who are using the same cluster for other work?
 
  Thanks,
  Nishanth
 
  On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu yuzhih...@gmail.com
  javascript:;
   wrote:
 
   bq. Or create a new file, compile it into ...
  
   You should go with the above approach.
  
   On Wed, Oct 15, 2014 at 2:08 PM, Matt K 
 matvey1...@gmail.com
  javascript:;
wrote:
  
Hi all,
   
I'm trying to get a custom filter to work on HBase 0.96.
  After
some
searching, I found that starting from 0.96, the implementer
  is
 required
   to
implement toByteArray and parseFrom methods, using
  Protocol
  Buffers.
But I'm having trouble with the how.
   
The proto file for the existing filters is located here:
   
   
  
 

   
  
 
 https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
   
Am I supposed to modify that file? Or create a new file,
   compile
it
  into
Java, and package it up with the filter?
   
In the meantime, I've taken a shortcut that's not working.
   Here's
my
   code:
http://pastebin.com/iHFKu9Xz
   
I'm using PrefixFilter, which comes with HBase, since I'm
   also
   filtering
by prefix. However, that errors out with the following:
http://pastebin.com/zBg47p6Z
   
Thanks in advance for helping!
   
-Matt
   
  
 

   
   
   
--
www.calcmachine.com - easy online calculator.
   
  
  
   --
   www.calcmachine.com - easy online calculator.
  
  
  
 


 --
 www.calcmachine.com - easy online calculator.

Re: custom filter on hbase 0.96

[Kevin]

Also, if you do end up using dynamic loading, you'll need a way to version
your filters because the RS will not reload a JAR if it changes.

On Tue, Oct 21, 2014 at 9:46 PM, Kevin kevin.macksa...@gmail.com wrote:

 I haven't tried dynamic loading of filters on RS, but I know it does
 exist. See https://issues.apache.org/jira/browse/HBASE-9301.

 If you still can't get it to work, then I suggest distributing your
 filters to the RS and restart them. Let us know how everything works out.

 On Tue, Oct 21, 2014 at 9:02 PM, Matt K matvey1...@gmail.com wrote:

 Thanks Kevin!

 I was under impression, probably mistakingly, that as of 0.96 placing
 the filter on hdfs under hbase lib directory is sufficient and RS should
 load the filter dynamically from hdfs. Is that not the case?

 On Tuesday, October 21, 2014, Kevin kevin.macksa...@gmail.com wrote:

  BTW, the error looks like you didn't distribute your custom filter to
 your
  region servers.
 
  On Tue, Oct 21, 2014 at 1:34 PM, Kevin kevin.macksa...@gmail.com
  javascript:; wrote:
 
   Matt,
  
   You should create your own proto file and compile that with the Google
   Protocol Buffer compiler. Take a look at the SingleColumnValueFilter's
   code:
  
 
 https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java#L327
  
   You will need to override `public byte[] toByteArray()` and `public
  static
   Filter parseFrom(byte[] pbBytes)`. The output of toByteArray() should
 be
   the byte[] from serializing with the protocol buffer. This will also
 be
  the
   input to parseFrom(byte[]), which is called using reflection on the
   server-side to instantiate your custom filter and use it.
  
   On Sun, Oct 19, 2014 at 11:31 AM, Matt K matvey1...@gmail.com
  javascript:; wrote:
  
   Anyone?
  
   On Thursday, October 16, 2014, Matt K matvey1...@gmail.com
  javascript:; wrote:
  
Hi, can anyone help with above? Feels like I'm missing something
   obvious.
   
On Wednesday, October 15, 2014, Nishanth S 
 nishanth.2...@gmail.com
  javascript:;
javascript:_e(%7B%7D,'cvml','nishanth.2...@gmail.com
 javascript:;');
  wrote:
   
Thanks Ted .I will take a look.
   
-Nishanth
   
On Wed, Oct 15, 2014 at 3:43 PM, Ted Yu yuzhih...@gmail.com
  javascript:; wrote:
   
 Nishanth:
 Good question.

 As a general coding guide, writing unit test is always a good
  start.
Using
 Matt's case as an example, take a look at TestPrefixFilter.

 There're various unit tests for Filters in hbase code.

 Cheers

 On Wed, Oct 15, 2014 at 2:30 PM, Nishanth S 
  nishanth.2...@gmail.com javascript:;
   
 wrote:

  Hi Ted ,
   Since I am also working on similar thing is there a way we
 can
   first
 test
  the filter on client side?.You know what I  mean without
  disrupting
 others
  who are using the same cluster for other work?
 
  Thanks,
  Nishanth
 
  On Wed, Oct 15, 2014 at 3:17 PM, Ted Yu yuzhih...@gmail.com
  javascript:;
   wrote:
 
   bq. Or create a new file, compile it into ...
  
   You should go with the above approach.
  
   On Wed, Oct 15, 2014 at 2:08 PM, Matt K 
 matvey1...@gmail.com
  javascript:;
wrote:
  
Hi all,
   
I'm trying to get a custom filter to work on HBase 0.96.
  After
some
searching, I found that starting from 0.96, the
 implementer
  is
 required
   to
implement toByteArray and parseFrom methods, using
  Protocol
  Buffers.
But I'm having trouble with the how.
   
The proto file for the existing filters is located here:
   
   
  
 

   
  
 
 https://github.com/apache/hbase/blob/master/hbase-protocol/src/main/protobuf/Filter.proto
   
Am I supposed to modify that file? Or create a new file,
   compile
it
  into
Java, and package it up with the filter?
   
In the meantime, I've taken a shortcut that's not working.
   Here's
my
   code:
http://pastebin.com/iHFKu9Xz
   
I'm using PrefixFilter, which comes with HBase, since
 I'm
   also
   filtering
by prefix. However, that errors out with the following:
http://pastebin.com/zBg47p6Z
   
Thanks in advance for helping!
   
-Matt
   
  
 

   
   
   
--
www.calcmachine.com - easy online calculator.
   
  
  
   --
   www.calcmachine.com - easy online calculator.
  
  
  
 


 --
 www.calcmachine.com - easy online calculator.

How to let hbase just return value or subset of the value

[Dai, Kevin]

Hi,

The value of my table is a Map.

I want to know how can I get only value(no any key sent from the region server) 
or get a subset of the value(Map) from hbase.

BR,
Kevin.

RE: How to let hbase just return value or subset of the value

[Dai, Kevin]

Hi, Ted

Thanks for your suggestion. But I want to know whether hbase can return the Map 
to me directly instead of cell.

BR,
Kevin.

-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com] 
Sent: 2014年9月19日 10:44
To: user@hbase.apache.org
Subject: Re: How to let hbase just return value or subset of the value

You can utilize Filter#transformCell(Cell)

Cheers

On Thu, Sep 18, 2014 at 7:33 PM, Dai, Kevin yun...@ebay.com wrote:

 Hi,

 The value of my table is a Map.

 I want to know how can I get only value(no any key sent from the region
 server) or get a subset of the value(Map) from hbase.

 BR,
 Kevin.

HBase custom filter protocol buffers

[Kevin]

Hi,

I'm making the switch from 0.92.1 to 0.98.1, and I'm in the process of
updating all my custom filters to conform to the new HBase Filter API. I
have quite a few custom filters, so my question is: Must I create a custom
protocol buffer for each of my filters or I can reuse the custom logic that
I had in writeFields() and readFields() in toByteArray() and
parseFrom(byte[]), respectively?

I did post this same question on Cloudera's CDH User Google group, but I
figured it was better suited to be asked on the official HBase mailing
list. (Sorry for posting in multiple locations.)

Thanks,
Kevin

ResultScanner performance

[Dai, Kevin]

Hi, everyone

My application will hold tens of thousands of ResultScanner to get Data. Will 
it hurt the performance and network resources?
If so, is there any way to solve it?
Thanks,
Kevin.

RE: ResultScanner performance

[Dai, Kevin]

Hi, Ted

We have a cluster of 48 machines and at least 100T data(which is still 
increasing).
The problem is that we have a lot of row keys (about tens of thousands ) to 
query in the meantime and we don't fetch all the data at once, instead we fetch 
them when needed,
so we may hold tens of thousands ResultScanner in the meantime.
I want to know whether it will hurt the performance and network resources and 
if so, is there any way to solve it?

Best regards,
Kevin.
-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com] 
Sent: 2014年8月26日 16:49
To: user@hbase.apache.org
Cc: user@hbase.apache.org; Huang, Jianshi
Subject: Re: ResultScanner performance

Can you give a bit more detail ?
What size is the cluster / dataset ?
What problem are you solving ?
Would using coprocessor help reduce the usage of ResultScanner ?

Cheers

On Aug 26, 2014, at 12:13 AM, Dai, Kevin yun...@ebay.com wrote:

 Hi, everyone
 
 My application will hold tens of thousands of ResultScanner to get Data. Will 
 it hurt the performance and network resources?
 If so, is there any way to solve it?
 Thanks,
 Kevin.

RE: ResultScanner performance

[Dai, Kevin]

Hi, Ted

I think you are right. But we must hold the ResultScanner for a while. So is 
there any way to reduce the performance loss? Or is there any way to share the 
connection?

Best regards,
Kevin.

-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com] 
Sent: 2014年8月27日 11:36
To: user@hbase.apache.org
Subject: Re: ResultScanner performance

Keeping many ResultScanners open at the same time is not good for performance.

Please see:
http://hbase.apache.org/book.html#perf.hbase.client.scannerclose

After fetching results from ResultScanner, you should close it ASAP.

Cheers


On Tue, Aug 26, 2014 at 8:18 PM, Dai, Kevin yun...@ebay.com wrote:

 Hi, Ted

 We have a cluster of 48 machines and at least 100T data(which is still 
 increasing).
 The problem is that we have a lot of row keys (about tens of thousands 
 ) to query in the meantime and we don't fetch all the data at once, 
 instead we fetch them when needed, so we may hold tens of thousands 
 ResultScanner in the meantime.
 I want to know whether it will hurt the performance and network 
 resources and if so, is there any way to solve it?

 Best regards,
 Kevin.
 -Original Message-
 From: Ted Yu [mailto:yuzhih...@gmail.com]
 Sent: 2014年8月26日 16:49
 To: user@hbase.apache.org
 Cc: user@hbase.apache.org; Huang, Jianshi
 Subject: Re: ResultScanner performance

 Can you give a bit more detail ?
 What size is the cluster / dataset ?
 What problem are you solving ?
 Would using coprocessor help reduce the usage of ResultScanner ?

 Cheers

 On Aug 26, 2014, at 12:13 AM, Dai, Kevin yun...@ebay.com wrote:

  Hi, everyone
 
  My application will hold tens of thousands of ResultScanner to get Data.
 Will it hurt the performance and network resources?
  If so, is there any way to solve it?
  Thanks,
  Kevin.

How to get table by HTablePool in HBaseTestingUtility

[Dai, Kevin]

Hi, all

I am doing unit testing with HBaseTestingUtility. I want to get the table by 
HTablePool. But it continually reports error as follows:

2014-08-07 15:14:07,231 INFO  [main-ScalaTest-running-HbaseQueryTest-SendTh
read(localhost:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect
(975)) - Opening socket connection to server localhost/127.0.0.1:2181. Will 
not attempt to authenticate using SASL (unknown error)
  2 2014-08-07 15:14:07,232 WARN  [main-ScalaTest-running-HbaseQueryTest-SendTh 
   read(localhost:2181)] zookeeper.ClientCnxn (ClientCnxn.java:run(1102)) - Se  
  ssion 0x0 for server null, unexpected error, closing socket connection and
 attempting reconnect
  3 java.net.ConnectException: 拒绝连接
  4 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
  5 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.jav 
   a:739)
  6 at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnS 
   ocketNIO.java:361)
  7 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1 
   081)

Can anyon help me to solve it.

Thanks,
Kevin.

Guava version incompatible

[Dai, Kevin]

Hi, all

I am now using spark to manipulate hbase. But I cant't use HBaseTestingUtility 
to do unit test. Because spark needs Guava 15.0 and above while Hbase needs 
Guava 14.0.1. These two versions are incompatible. Is there any way to solve 
this conflict with maven.

Thanks,
Kevin.

Re: hbase set memstore size to 0

[Kevin O'dell]

Hi Ozhang,

  If you are only bulk loading into HBase, then memstore flush size should
not matter. You most likely you looking to lower the upper/global memstore
limits.
On Aug 3, 2014 2:23 PM, ozhang ozhangu...@gmail.com wrote:

 Hello,
 In our hbase cluster memstore flush size is 128 mb. And to insert data to
 tables, we only use bulk load tool. Since bulk loading bypasses memstores,
 they are never used, so we want to minimize memstore flush size. But
 memstore flush size is used in many important calculations in hbase such
 that;

 region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
 “hbase.hregion.max.filesize”)

 So setting memstore value smaller or 0 for example,  results in some
 other
 problems.
 What do you suggest us in that case. Setting memstore size to 128 holds
 some
 memory for tens of regions in region server and we want to get rid of it.
 Thanks a lot.

 ozhan



 --
 View this message in context:
 http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
 Sent from the HBase User mailing list archive at Nabble.com.

Re: hbase set memstore size to 0

[Kevin O'dell]

Upon insert, lower the global setting not the flush size :)
On Aug 3, 2014 3:01 PM, ozhang ozhangu...@gmail.com wrote:

 Hi Kevin,
 We guess that, on region server start up, hbase gets some memory for each
 memstore. So we want to decrease this value. You are saying that memstore
 size doesnt matter. If memstore flush size 128 mb, does java take some
 memory for each memstore on region startup. Or it only takes memory while
 you are using it to insert data.
 Thanks a lot
 3 Ağu 2014 21:27 tarihinde Kevin O'dell [via Apache HBase] 
 ml-node+s679495n4062260...@n3.nabble.com yazdı:

  Hi Ozhang,
 
If you are only bulk loading into HBase, then memstore flush size
 should
  not matter. You most likely you looking to lower the upper/global
 memstore
  limits.
  On Aug 3, 2014 2:23 PM, ozhang [hidden email]
  http://user/SendEmail.jtp?type=nodenode=4062260i=0 wrote:
 
   Hello,
   In our hbase cluster memstore flush size is 128 mb. And to insert data
  to
   tables, we only use bulk load tool. Since bulk loading bypasses
  memstores,
   they are never used, so we want to minimize memstore flush size. But
   memstore flush size is used in many important calculations in hbase
 such
   that;
  
   region split size = Min (R^2 * “hbase.hregion.memstore.flush.size”,
   “hbase.hregion.max.filesize”)
  
   So setting memstore value smaller or 0 for example,  results in some
   other
   problems.
   What do you suggest us in that case. Setting memstore size to 128 holds
   some
   memory for tens of regions in region server and we want to get rid of
  it.
   Thanks a lot.
  
   ozhan
  
  
  
   --
   View this message in context:
  
 
 http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257.html
   Sent from the HBase User mailing list archive at Nabble.com.
  
 
 
  --
   If you reply to this email, your message will be added to the discussion
  below:
 
 
 http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062260.html
   To unsubscribe from hbase set memstore size to 0, click here
  
 http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=4062257code=b3poYW5ndWxlbkBnbWFpbC5jb218NDA2MjI1N3wxMDY0OTA3MTE4
 
  .
  NAML
  
 http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
 
 




 --
 View this message in context:
 http://apache-hbase.679495.n3.nabble.com/hbase-set-memstore-size-to-0-tp4062257p4062261.html
 Sent from the HBase User mailing list archive at Nabble.com.

Bulk load to multiple tables

[Kevin]

I am reading data off of HDFS that don't all get loaded into a single
table. With the current way of bulk loading I can load to the table that
most of the data will end up in, and I can use the client API (i.e., Put)
to load the other data from the file into the other tables.

The current bulk loading process involves creating the same number of
reducers as there are regions for the specified table. I think I understand
that once the appropriate region servers adopt the HFiles minor compactions
will merge them into the regions' storefiles.

It seems like you could set the number of reducers to the total number of
regions for all the tables considered. Then you write the partitions file
as key-values where the key is the destination table and the value is a
region start key (instead of the key being the start key and the value
being NullWritable). Mappers could then prefix rows with their destination
table before doing a context.write(). The TotalOrderPartitioner needs to be
modified to account for all these changes. I have a feeling this is an
overly complicated approach or if it would even work.

Maybe you could do it without all those changes and just use
MultipleOutputs?

Has anyone else thought about or done bulk loading with multiple tables?

Re: Does compression ever improve performance?

[Kevin O'dell]

Hi Jeremy,

  I always recommend turning on snappy compression,  I have ~20%
performance increases.
On Jun 14, 2014 10:25 AM, Ted Yu yuzhih...@gmail.com wrote:

 You may have read Doug Meil's writeup where he tried out different
 ColumnFamily
 compressions :

 https://blogs.apache.org/hbase/

 Cheers


 On Fri, Jun 13, 2014 at 11:33 AM, jeremy p athomewithagroove...@gmail.com
 
 wrote:

  Thank you -- I'll go ahead and try compression.
 
  --Jeremy
 
 
  On Fri, Jun 13, 2014 at 10:59 AM, Dima Spivak dspi...@cloudera.com
  wrote:
 
   I'd highly recommend it. In general, compressing your column families
  will
   improve performance by reducing the resources required to get data from
   disk (even when taking into account the CPU overhead of compressing and
   decompressing).
  
   -Dima
  
  
   On Fri, Jun 13, 2014 at 10:35 AM, jeremy p 
  athomewithagroove...@gmail.com
   
   wrote:
  
Hey all,
   
Right now, I'm not using compression on any of my tables, because our
   data
doesn't take up a huge amount of space.  However, I would turn on
compression if there was a chance it would improve HBase's
 performance.
By
performance, I'm talking about the speed with which HBase responds to
requests and retrieves data.
   
Should I turn compression on?
   
--Jeremy
   
  
 

Re: Effect of region size on compaction performance

[Kevin O'dell]

Hey David,

 What is your write pattern?  If you are bulkloading the data into HBase
this gives you the ability to add more regions and control your
compactions.  If not, a high number of regions as Vlad indicated can cause
some weird issues.  How many region servers do you have?  What is the
current region count?  How many MB/s are you ingesting into your cluster?
 Do you write equally to all regions during ingest?


On Sun, Mar 23, 2014 at 3:51 PM, Vladimir Rodionov
vrodio...@carrieriq.comwrote:

 How small is small and how large is large?
 Recommended region size is usually between 5-10GB. Too small regions
 results in more frequent flushes/compactions
 and have additional overhead in RS RAM.

 I am thinking about extending TableInputFormat to override the
 1-map-per-region default policy as an alternative.

 This looks better approach.

 Best regards,
 Vladimir Rodionov
 Principal Platform Engineer
 Carrier IQ, www.carrieriq.com
 e-mail: vrodio...@carrieriq.com

 
 From: David Koch [ogd...@googlemail.com]
 Sent: Saturday, March 22, 2014 6:58 PM
 To: user@hbase.apache.org
 Subject: Effect of region size on compaction performance

 Hello,

 We run M/Rs over several HBase tables at the same time and chose to reduce
 region sizes in order to make map tasks faster and improve map-slot
 turnaround between the concurrent jobs. However, I am worried many regions
 will cause longer overall compactions of the HBase data. Is this the case?

 I am thinking about extending TableInputFormat to override the
 1-map-per-region default policy as an alternative.

 Regards,

 /David

 Confidentiality Notice:  The information contained in this message,
 including any attachments hereto, may be confidential and is intended to be
 read only by the individual or entity to whom this message is addressed. If
 the reader of this message is not the intended recipient or an agent or
 designee of the intended recipient, please note that any review, use,
 disclosure or distribution of this message or its attachments, in any form,
 is strictly prohibited.  If you have received this message in error, please
 immediately notify the sender and/or notificati...@carrieriq.com and
 delete or destroy any copy of this message and its attachments.




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: 答复: what is the default size of each Column family memstore....

[Kevin O'dell]

Upendra,

  Todd L wrote a great set of blogs that explains the problem and some
solutions:
http://blog.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/


On Mon, Mar 3, 2014 at 12:00 PM, Ted Yu yuzhih...@gmail.com wrote:

 See http://hbase.apache.org/book.html#gcpause


 On Mon, Mar 3, 2014 at 8:06 AM, Upendra Yadav upendra1...@gmail.com
 wrote:

  On this link(https://hbase.apache.org/book/regions.arch.html) i got :
 
  MSLAB requires 2mb per memstore (that's 2mb per family per region).
 
  what is MSLAB. Is this related to each Data block size inside HFile?
 
 
  On Mon, Mar 3, 2014 at 7:30 PM, Upendra Yadav upendra1...@gmail.com
  wrote:
 
   Thanks...
  
   So in HFile, each data Block size will be 128 MB(After Major
   compaction if we have more than one block...)?
  
  
   On Mon, Mar 3, 2014 at 5:21 PM, 冯宏华 fenghong...@xiaomi.com wrote:
  
   128M
   
   发件人: Upendra Yadav [upendra1...@gmail.com]
   发送时间: 2014年3月3日 19:20
   收件人: user@hbase.apache.org
   主题: what is the default size of each Column family memstore
  
   Hi
  
   what is the default size of each Column family memstore
  
  
  
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: 0.92 = 0.96 in one go?

[Kevin O'dell]

Just your first born


On Fri, Jan 31, 2014 at 2:35 PM, Otis Gospodnetic 
otis.gospodne...@gmail.com wrote:

 You've got that much time!?  I mean, sure, sure, but how much is that going
 to cost me?  A pinky?

 Otis
 --
 Performance Monitoring * Log Analytics * Search Analytics
 Solr  Elasticsearch Support * http://sematext.com/


 On Fri, Jan 31, 2014 at 2:32 PM, Stack st...@duboce.net wrote:

  Want me to try it for you Otis w/ artificial data? (Usual disclaimers...)
  St.Ack
 
 
  On Fri, Jan 31, 2014 at 6:35 AM, Otis Gospodnetic 
  otis.gospodne...@gmail.com wrote:
 
   Hi,
  
   It looks like going from 0.96 to 0.98 should be relatively easy, which
 is
   great.
  
   We are on 0.92 and I'm wondering if it's possible do the 0.92 == 0.96
  jump
   without making two jumps: 0.92 == 0.94 and then 0.94. == 0.96 ?
  
   Thanks,
   Otis
   --
   Performance Monitoring * Log Analytics * Search Analytics
   Solr  Elasticsearch Support * http://sematext.com/
  
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Hbase tuning for heavy write cluster

[Kevin O'dell]

Rohit,

  64GB heap is not ideal, you will run into some weird issues. How many
regions are you running per server, how many drives in each node, any other
settings you changed from default?
On Jan 24, 2014 6:22 PM, Rohit Dev rohitdeve...@gmail.com wrote:

 Hi,

 We are running Opentsdb on CDH 4.3 hbase cluster, with most of the
 default settings. The cluster is heavy on write and I'm trying to see
 what parameters I can tune to optimize the write performance.


 # I get messages related to Memstore[1] and Slow Response[2] very
 often, is this an indication of any issue ?

 I tried increasing some parameters on one node:
  - hbase.hstore.blockingStoreFiles - from default 7 to 15
  - hbase.hregion.memstore.block.multiplier - from default 2 to 8
  - and heap size from 16GB to 64GB

  * 'Compaction queue' went up to ~200 within 60 mins after restarting
 region server with new parameters and the log started to get even more
 noisy.

 Can anyone please suggest if I'm going to right direction with these
 new settings ? or if there are other thing that I could monitor or
 change to make it better.

 Thank you!


 [1]
 INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates
 for 'IPC Server handler 19 on 60020' on region

 tsdb,\x008XR\xE0i\x90\x00\x00\x02Q\x7F\x1D\x00\x00(\x00\x0B]\x00\x008M(r\x00\x00Bl\xA7\x8C,1390556781703.0771bf90cab25c503d3400206417f6bf.:
 memstore size 256.3 M is = than blocking 256 M size

 [2]
  WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow):

 {processingtimems:17887,call:multi(org.apache.hadoop.hbase.client.MultiAction@586940ea
 ),
 rpc version=1, client version=29,
 methodsFingerPrint=0,client:192.168.10.10:54132
 ,starttimems:1390587959182,queuetimems:1498,class:HRegionServer,responsesize:0,method:multi}

Re: Hbase Performance Issue

[Kevin O'dell]

Have you tried writing out an hfile and then bulk loading the data?
On Jan 4, 2014 4:01 PM, Ted Yu yuzhih...@gmail.com wrote:

 bq. Output is written to either Hbase

 Looks like Akhtar wants to boost write performance to HBase.
 MapReduce over snapshot files targets higher read throughput.

 Cheers


 On Sat, Jan 4, 2014 at 12:55 PM, Vladimir Rodionov
 vrodio...@carrieriq.comwrote:

  You cay try MapReduce over snapshot files
  https://issues.apache.org/jira/browse/HBASE-8369
 
  but you will need to patch 0.94.
 
  Best regards,
  Vladimir Rodionov
  Principal Platform Engineer
  Carrier IQ, www.carrieriq.com
  e-mail: vrodio...@carrieriq.com
 
  
  From: Akhtar Muhammad Din [akhtar.m...@gmail.com]
  Sent: Saturday, January 04, 2014 12:44 PM
  To: user@hbase.apache.org
  Subject: Re: Hbase Performance Issue
 
  im  using CDH 4.5:
  Hadoop:  2.0.0-cdh4.5.0
  HBase:   0.94.6-cdh4.5.0
 
  Regards
 
 
  On Sun, Jan 5, 2014 at 1:24 AM, Ted Yu yuzhih...@gmail.com wrote:
 
   What version of HBase / hdfs are you running with ?
  
   Cheers
  
  
  
   On Sat, Jan 4, 2014 at 12:17 PM, Akhtar Muhammad Din
   akhtar.m...@gmail.comwrote:
  
Hi,
I have been running a map reduce job that joins 2 datasets of 1.3
 and 4
   GB
in size. Joining is done at reduce side. Output is written to either
   Hbase
or HDFS depending upon configuration. The problem I am having is that
   Hbase
takes about 60-80 minutes to write the processed data, on the other
  hand
HDFS takes only 3-5 mins to write the same data. I really want to
  improve
the Hbase speed and bring it down to 1-2 min.
   
I am using amazon EC2 instances, launched a cluster of size 3 and
 later
   10,
have tried both c3.4xlarge and c3.8xlarge instances.
   
I can see significant increase in performance while writing to HDFS
 as
  i
use cluster with more nodes, having high specifications, but in the
  case
   of
Hbase there was no significant change in performance.
   
I have been going through different posts, articles and have read
 Hbase
book to solve the Hbase performance issue but have not been able to
   succeed
so far.
Here are the few things i have tried out so far:
   
*Client Side*
- Turned off writing to WAL
- Experimented with write buffer size
- Turned off auto flush on table
- Used cache, experimented with different sizes
   
   
*Hbase Server Side*
- Increased region servers heap size to 8 GB
- Experimented with handlers count
- Increased Memstore flush size to 512 MB
- Experimented with hbase.hregion.max.filesize, tried different sizes
   
There are many other parameters i have tried out following the
   suggestions
from  different sources, but nothing worked so far.
   
Your help will be really appreciated.
   
--
Regards
Akhtar Muhammad Din
   
  
 
 
 
  --
  Regards
  Akhtar Muhammad Din
 
  Confidentiality Notice:  The information contained in this message,
  including any attachments hereto, may be confidential and is intended to
 be
  read only by the individual or entity to whom this message is addressed.
 If
  the reader of this message is not the intended recipient or an agent or
  designee of the intended recipient, please note that any review, use,
  disclosure or distribution of this message or its attachments, in any
 form,
  is strictly prohibited.  If you have received this message in error,
 please
  immediately notify the sender and/or notificati...@carrieriq.com and
  delete or destroy any copy of this message and its attachments.
 

Re: Hbase Performance Issue

[Kevin O'dell]

Could you give us a region server log to look at during a job?
On Jan 4, 2014 4:35 PM, Akhtar Muhammad Din akhtar.m...@gmail.com wrote:

 Thanks guys for your precious time.
 Vladimir, as Ted rightly said i want to improve write performance currently
 (of course i want to read data as fast as possible later on)
 Kevin, my current understanding of bulk load is that you generate
 StoreFiles and later load through a command line program. I dont want to do
 any manual step. Our system is getting data after every 15 minutes, so
 requirement is to automate it through client API completely.



 On Sun, Jan 5, 2014 at 2:19 AM, Kevin O'dell kevin.od...@cloudera.com
 wrote:

  Have you tried writing out an hfile and then bulk loading the data?
  On Jan 4, 2014 4:01 PM, Ted Yu yuzhih...@gmail.com wrote:
 
   bq. Output is written to either Hbase
  
   Looks like Akhtar wants to boost write performance to HBase.
   MapReduce over snapshot files targets higher read throughput.
  
   Cheers
  
  
   On Sat, Jan 4, 2014 at 12:55 PM, Vladimir Rodionov
   vrodio...@carrieriq.comwrote:
  
You cay try MapReduce over snapshot files
https://issues.apache.org/jira/browse/HBASE-8369
   
but you will need to patch 0.94.
   
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com
   

From: Akhtar Muhammad Din [akhtar.m...@gmail.com]
Sent: Saturday, January 04, 2014 12:44 PM
To: user@hbase.apache.org
Subject: Re: Hbase Performance Issue
   
im  using CDH 4.5:
Hadoop:  2.0.0-cdh4.5.0
HBase:   0.94.6-cdh4.5.0
   
Regards
   
   
On Sun, Jan 5, 2014 at 1:24 AM, Ted Yu yuzhih...@gmail.com wrote:
   
 What version of HBase / hdfs are you running with ?

 Cheers



 On Sat, Jan 4, 2014 at 12:17 PM, Akhtar Muhammad Din
 akhtar.m...@gmail.comwrote:

  Hi,
  I have been running a map reduce job that joins 2 datasets of 1.3
   and 4
 GB
  in size. Joining is done at reduce side. Output is written to
  either
 Hbase
  or HDFS depending upon configuration. The problem I am having is
  that
 Hbase
  takes about 60-80 minutes to write the processed data, on the
 other
hand
  HDFS takes only 3-5 mins to write the same data. I really want to
improve
  the Hbase speed and bring it down to 1-2 min.
 
  I am using amazon EC2 instances, launched a cluster of size 3 and
   later
 10,
  have tried both c3.4xlarge and c3.8xlarge instances.
 
  I can see significant increase in performance while writing to
 HDFS
   as
i
  use cluster with more nodes, having high specifications, but in
 the
case
 of
  Hbase there was no significant change in performance.
 
  I have been going through different posts, articles and have read
   Hbase
  book to solve the Hbase performance issue but have not been able
 to
 succeed
  so far.
  Here are the few things i have tried out so far:
 
  *Client Side*
  - Turned off writing to WAL
  - Experimented with write buffer size
  - Turned off auto flush on table
  - Used cache, experimented with different sizes
 
 
  *Hbase Server Side*
  - Increased region servers heap size to 8 GB
  - Experimented with handlers count
  - Increased Memstore flush size to 512 MB
  - Experimented with hbase.hregion.max.filesize, tried different
  sizes
 
  There are many other parameters i have tried out following the
 suggestions
  from  different sources, but nothing worked so far.
 
  Your help will be really appreciated.
 
  --
  Regards
  Akhtar Muhammad Din
 

   
   
   
--
Regards
Akhtar Muhammad Din
   
Confidentiality Notice:  The information contained in this message,
including any attachments hereto, may be confidential and is intended
  to
   be
read only by the individual or entity to whom this message is
  addressed.
   If
the reader of this message is not the intended recipient or an agent
 or
designee of the intended recipient, please note that any review, use,
disclosure or distribution of this message or its attachments, in any
   form,
is strictly prohibited.  If you have received this message in error,
   please
immediately notify the sender and/or notificati...@carrieriq.com and
delete or destroy any copy of this message and its attachments.
   
  
 



 --
 Regards
 Akhtar Muhammad Din

Re: HBase Client

[Kevin O'dell]

It depends, are you looking for a GUI interface to HBase? In the HBase
ecosystem we have tons of great tools.

HappyBase
Java API
Phoenix
etc

HRider is a good tool as well.  Try them all and pick whichever one makes
you happy.  I have NO CLUE if any of those will work with M7 though...again
I would point you back to proprietary keepers of the code:

[image: Inline image 1]


On Fri, Dec 13, 2013 at 5:37 PM, ados1...@gmail.com ados1...@gmail.comwrote:

 Any one having experience with HRider tool? Is it good or do we have better
 solutions than that in HBase eco system?


 On Fri, Dec 13, 2013 at 5:33 PM, ados1...@gmail.com ados1...@gmail.com
 wrote:

  Thank you all. I will try to reach them and ask about it.
 
 
  On Fri, Dec 13, 2013 at 5:30 PM, Jean-Marc Spaggiari 
  jean-m...@spaggiari.org wrote:
 
  Hi Andy,
 
  Sorry, I don't want to say anything wrong, but I think this is not
 really
  HBase. It's something working like HBase, doing almost the same things,
  but
  not 100% the same. So I'm not sure I can recommend any tool working with
  it. You might want to ask to a M7 mailing list if they have anything
  specific working with that.
 
  Good luck.
 
  JM
 
 
  2013/12/13 ados1...@gmail.com ados1...@gmail.com
 
   Am using MapR M7 HBase distribution (
   http://www.mapr.com/products/mapr-editions/m7-edition)
  
  
   On Fri, Dec 13, 2013 at 5:24 PM, Jean-Marc Spaggiari 
   jean-m...@spaggiari.org wrote:
  
Hi Andy,
   
Then I'm not really sure what to recommend you.
   
What version of HBase are you using with your mapR distribution?
  0.94.x?
0.92.x?
   
JM
   
   
2013/12/13 ados1...@gmail.com ados1...@gmail.com
   
 Thank you Ted and Jean,

 I should have been more clear stating what is the best hbase
  client to
use
 with *mapR*? Hue is not officially supported with mapR.


 On Fri, Dec 13, 2013 at 5:16 PM, Ted Yu yuzhih...@gmail.com
  wrote:

  Search for 'hbase sql client' gives this as top hit:
  https://github.com/forcedotcom/phoenix
 
 
  On Fri, Dec 13, 2013 at 2:14 PM, ados1...@gmail.com 
ados1...@gmail.com
  wrote:
 
   Thanks Ted but am looking for something like toad/sql
 developer
  for
   querying/viewing data in hbase.
  
  
   On Fri, Dec 13, 2013 at 5:09 PM, Ted Yu yuzhih...@gmail.com
   wrote:
  
Hi,
See http://hbase.apache.org/book.html#client
and http://hbase.apache.org/book.html#rest
   
Cheers
   
   
On Fri, Dec 13, 2013 at 2:06 PM, ados1...@gmail.com 
  ados1...@gmail.com
wrote:
   
 Hello All,

 I am newbie in hbase and wanted to see if there are any
 good
hbase
   client
 that i can use to query underlying hbase datastore or what
  is
   the
  best
tool
 to use?

 I am using command line but looking for any other best
alternative.

 Regards,
 Andy.

   
  
 

   
  
 
 
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Get all columns in a column family

[Kevin O'dell]

Hey JC,

  Is this what you are looking for
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Get.html#addFamily(byte[])


On Thu, Dec 12, 2013 at 3:02 AM, JC jcash...@marketo.com wrote:

 I have use case where if one column of a column family changes, I would
 like
 to bring back all the columns in that column family. I can use the
 timestamp
 to identify the column that changes but it only returns the one column. Is
 there a way I can get all the columns of the column family back with one
 pass of the data? What are some other options here?

 Thanks in advance





 --
 View this message in context:
 http://apache-hbase.679495.n3.nabble.com/Get-all-columns-in-a-column-family-tp4053696.html
 Sent from the HBase User mailing list archive at Nabble.com.




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Problem When Prefetch META table

[Kevin O'dell]

Hey Andrea,

  That is very weird, I hit this similar extremely strange one off issue
that I could never reproduce.  After dropping a table doing X amount of
inserts, I would receive the same error and my META table would be
corrupted.  What I did to correct this issue was to sideline the META table
and then use offlineMetaRepair to rebuild it, and then I never ran into it
again.  It was quite strange and I am sorry I don't have more to offer.


On Tue, Dec 10, 2013 at 8:34 AM, Jean-Marc Spaggiari 
jean-m...@spaggiari.org wrote:

 Hi Andrea,

 Which version of HBase are you testing with?

 JM


 2013/12/10 Andrea andrea.gandini...@gmail.com

  Hi, I encounter this problem:
 
  13/12/10 13:25:15 WARN
 client.HConnectionManager$HConnectionImplementation:
  Encountered problems when prefetch META table:
  java.io.IOException: HRegionInfo was null or empty in Meta for usertable,
  row=usertable,user4300292263938371081,99
  at
  org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:170)
  at
 
 org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
  at
 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
  at
 
 org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
  at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager
  .java:360)
  at
  org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
  at
  org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
  at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
  prefetchRegionCache(HConnectionManager.java:918)
  at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
  locateRegionInMeta(HConnectionManager.java:973)
  at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
  locateRegion(HConnectionManager.java:860)
  at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
  locateRegion(HConnectionManager.java:817)
  at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
  processBatchCallback(HConnectionManager.java:1507)
  at
 
 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
  processBatch(HConnectionManager.java:1392)
  at
  org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:918)
  at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:774)
  at org.apache.hadoop.hbase.client.HTable.put(HTable.java:749)
  at com.yahoo.ycsb.db.HBaseClient.update(HBaseClient.java:328)
  at com.yahoo.ycsb.db.HBaseClient.insert(HBaseClient.java:357)
  at com.yahoo.ycsb.DBWrapper.insert(DBWrapper.java:148)
  at
  com.yahoo.ycsb.workloads.CoreWorkload.doInsert(CoreWorkload.java:461)
  at com.yahoo.ycsb.ClientThread.run(Client.java:269)
 
  It will happen when, on a netowrk with one zookeeper, one master and one
  regionserver I try to load a lot of data. This is the number of
 operations
  before the error:
 
  200 sec: 888708 operations; 7767.62 current ops/sec; [UPDATE
  AverageLatency(us)=27089963] [INSERT AverageLatency(us)=9738.99] [CLEANUP
  AverageLatency(us)=27089976]
   210 sec: 893742 operations; 502.7 current ops/sec; [UPDATE
  AverageLatency(us)=14887298.5] [INSERT AverageLatency(us)=6937.27]
 [CLEANUP
  AverageLatency(us)=14887312.5]
   221 sec: 928277 operations; 3226.97 current ops/sec; [UPDATE
  AverageLatency(us)=27006074] [INSERT AverageLatency(us)=8529.21] [CLEANUP
  AverageLatency(us)=27006089]
   231 sec: 930079 operations; 180.18 current ops/sec;  [INSERT
  AverageLatency(us)=457.08]
  13/12/10 13:25:13 WARN
 client.HConnectionManager$HConnectionImplementation:
  Encountered problems when prefetch META table: ...
 
  Does this depend on the number of thread in whereby I insert the data?
 
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Table state

[Kevin O'dell]

Hey Raheem,

  You can sideline the table into tmp(mv /hbase/table /tmp/table, then
bring HBase back online.  Once HBase is back you can use HBCK to repair
your META -fixMeta -fixAssignments.  Once HBase is consistent again, you
can move the table back out of tmp and use HBCK to reupdate META.  If the
issue reoccurs let us know.


On Tue, Dec 10, 2013 at 11:50 AM, Daya, Raheem
raheem.d...@relayhealth.comwrote:

 I have a distributed Hbase cluster that will not start.  It looks like
 there is a table that is an inconsistent state:
 2013-12-10 07:40:50,447 FATAL org.apache.hadoop.hbase.master.HMaster:
 Unexpected state :
 ct_claims,204845|81V6SO4EF56DD1TKOIU7AS4L5D,1386050670937.6d138b97cde8bc3e49ff34639913109c.
 state=PENDING_OPEN, ts=1386690050445, server=rhf-045,60020,1386689069486 ..
 Cannot transit it to OFFLINE.

 Is there a way to manually set the table to OFFLINE?  I have tried
 deleting the /hbase node in zookeeper.  I tried bringing up the master and
 then a region server and vice versa.  In the case of bringing the master up
 first, the master starts.  As soon as I bring up a region server the master
 goes down.  My thought is to move the tables to OFFLINE, (assuming it is
 possible), and try bringing up the cluster again.  hbck will not work as
 none of the region servers are up.  Any one have any other ideas?
 Thanks,
 Raheem







-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Table state

[Kevin O'dell]

Just to close the loop, the previous recommended steps help to get us back
up, but one of the HMasters is not happy now.  I will update with a final
analysis shortly.


On Tue, Dec 10, 2013 at 1:10 PM, Jean-Marc Spaggiari 
jean-m...@spaggiari.org wrote:

 Also, might be interesting to look in the RS logs to see what this region
 can not come back online...

 JM


 2013/12/10 Kevin O'dell kevin.od...@cloudera.com

  Hey Raheem,
 
You can sideline the table into tmp(mv /hbase/table /tmp/table, then
  bring HBase back online.  Once HBase is back you can use HBCK to repair
  your META -fixMeta -fixAssignments.  Once HBase is consistent again, you
  can move the table back out of tmp and use HBCK to reupdate META.  If the
  issue reoccurs let us know.
 
 
  On Tue, Dec 10, 2013 at 11:50 AM, Daya, Raheem
  raheem.d...@relayhealth.comwrote:
 
   I have a distributed Hbase cluster that will not start.  It looks like
   there is a table that is an inconsistent state:
   2013-12-10 07:40:50,447 FATAL org.apache.hadoop.hbase.master.HMaster:
   Unexpected state :
  
 
 ct_claims,204845|81V6SO4EF56DD1TKOIU7AS4L5D,1386050670937.6d138b97cde8bc3e49ff34639913109c.
   state=PENDING_OPEN, ts=1386690050445,
 server=rhf-045,60020,1386689069486
  ..
   Cannot transit it to OFFLINE.
  
   Is there a way to manually set the table to OFFLINE?  I have tried
   deleting the /hbase node in zookeeper.  I tried bringing up the master
  and
   then a region server and vice versa.  In the case of bringing the
 master
  up
   first, the master starts.  As soon as I bring up a region server the
  master
   goes down.  My thought is to move the tables to OFFLINE, (assuming it
 is
   possible), and try bringing up the cluster again.  hbck will not work
 as
   none of the region servers are up.  Any one have any other ideas?
   Thanks,
   Raheem
  
  
  
  
  
 
 
  --
  Kevin O'Dell
  Systems Engineer, Cloudera
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Maximum number of regions for a table?

[Kevin O'dell]

Hi Kim,

  I would take a look at:

http://hadoop-hbase.blogspot.com/2013/01/hbase-region-server-memory-sizing.html


On Tue, Dec 3, 2013 at 2:42 PM, Kim Chew kchew...@gmail.com wrote:

 I am wondering if there is a restrain on the number of regions that a table
 could have. For example, if I have a table that grows very fast so the
 region keeps splitting, is it possible that the table could have as many
 regions as it could until all the resource run out?

 Thanks.

 Kim




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Region server block cache and memstore size

[Kevin O'dell]

I agree with what Anoop said here, just because they are scans, it doesn't
make a lot of sense to turn off your block cache.  Are you trying to save
memory?  As for the memstore global limits, you will want to set those to
something like

upper .11
lower .10

  You have to leave at the minimum .10, as a safety value of .09 has been
hardcoded.  On a related topic, does anyone know why we have that safety
value?  I would recommend bumping your block cache to .65, on a 16GB heap
that now leaves you with 10.4GB of block cache per node.  This may help
some of your scans speed up.


On Thu, Nov 28, 2013 at 4:50 AM, Anoop John anoop.hb...@gmail.com wrote:

 So you use Bulk load with HFileOpFormat for writing data?  Then you can
 reduce the hbase.regionserver.global.memstore.upperLimit and
 hbase.regionserver.global.memstore.lowerLimit  and give more heap % for the
 block cache.  Not getting why u try to reduce that also.

 -Anoop-


 On Thu, Nov 28, 2013 at 3:16 PM, Ivan Tretyakov 
 itretya...@griddynamics.com
  wrote:

  Hi!
 
  We are using HBase 0.92.1-cdh4.1.1. To import data the only way we use is
  bulk load. And our common access pattern is sequential scans of different
  parts of the tables.
 
  Since that we are considering to disable block cache by setting
  hbase.block.cache.size to zero.
  But We've found following in HBase book (
  http://hbase.apache.org/book/important_configurations.html):
 
  Do not turn off block cache (You'd do it by setting
 hbase.block.cache.size
  to zero). Currently we do not do well if you do this because the
  regionserver will spend all its time loading hfile indices over and over
  again. If your working set it such that block cache does you no good, at
  least size the block cache such that hfile indices will stay up in the
  cache (you can get a rough idea on the size you need by surveying
  regionserver UIs; you'll see index block size accounted near the top of
 the
  webpage).
 
  Another thing we consider to reduce is memstore size by tuning following
  options hbase.regionserver.global.memstore.upperLimit and
  hbase.regionserver.global.memstore.lowerLimit.
 
  So, my questions are:
 
  Does it make sense to touch these options in our case?
  Is this memory reserved or other processes inside regionserver can use
 it?
 
  Thanks in advance!
 
  --
  Best Regards
  Ivan Tretyakov
 
  Deployment Engineer
  Grid Dynamics
  +7 812 640 38 76
 https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#
 
  Skype: ivan.v.tretyakov
  www.griddynamics.com
  itretya...@griddynamics.com
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

HBase Replication Progress

[Kevin Su]

Hi,

I am having trouble searching for answers regarding HBase replication, so I
thought I would email the mailing list.

Does HBase provide an API/way to see what has/hasn't been replicated yet?

My use case is the following:

I run a map reduce job in Cluster A and stick the output in HBase. I would
like to transport this output to Cluster B as (part of) the input to
another map reduce job. I hope to achieve this transport via HBase
replication. The second map reduce job cannot start until all the data from
Cluster A has been replicated to Cluster B. So what is the best way to
check if everything has been replicated? Do I query Zookeeper and check if
the RS queues are empty? Or is HBase replication not the right fit for my
use case?

I am using HBase 0.94.2.

Thanks in advance for any advice!

--
Kevin

Re: HBase Region Server crash if column size become to big

[Kevin O'dell]

You can also check the messages file in /var/log.  The OOME may also be
there as well.  I would be willing to bet this is a batching issue.


On Wed, Sep 11, 2013 at 11:15 AM, Dhaval Shah
prince_mithi...@yahoo.co.inwrote:

 John can you check the .out file as well. We used to have a similar issue
 and turned out that query for such a large row ran the region server out of
 memory causing the crash and oome does not show up in the .log files but
 rather in the .out files.

 In such a situation setBatch for scans or column pagination filter for
 gets can help your case

 Sent from Yahoo! Mail on Android




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: HBase Region Server crash if column size become to big

[Kevin O'dell]

John,

  Out of Memory Error.  You can add this to your code(assuming it is in
your release) scan.setBatch(batch);


On Wed, Sep 11, 2013 at 11:26 AM, John johnnyenglish...@gmail.com wrote:

 @Kevin: I changed the  hbase.client.keyvalue.maxsize from 10MB to 500MB,
 but the regionserver still crashs. How can i change the batch size in the
 hbase shell? Whats OOME?

 @Dhaval: there is only the *.out file in /var/log/hbase. Is the .log file
 located in another directory?


 2013/9/11 Kevin O'dell kevin.od...@cloudera.com

  You can also check the messages file in /var/log.  The OOME may also be
  there as well.  I would be willing to bet this is a batching issue.
 
 
  On Wed, Sep 11, 2013 at 11:15 AM, Dhaval Shah
  prince_mithi...@yahoo.co.inwrote:
 
   John can you check the .out file as well. We used to have a similar
 issue
   and turned out that query for such a large row ran the region server
 out
  of
   memory causing the crash and oome does not show up in the .log files
 but
   rather in the .out files.
  
   In such a situation setBatch for scans or column pagination filter for
   gets can help your case
  
   Sent from Yahoo! Mail on Android
  
  
 
 
  --
  Kevin O'Dell
  Systems Engineer, Cloudera
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: HBase Java client - unknown host: localhost.localdomain

[Kevin O'dell]

Can you attach a screen shot of the HMaster UI? It appears ZK is connecting
fine, but can't find .META.
On Aug 25, 2013 8:57 AM, Shengjie Min shengjie@gmail.com wrote:

 Hi Jean-Marc,

 You meant my cloudera vm or my client? Here is my /etc/hosts

 cloudera vm:

 127.0.0.1   localhost.localdomain   localhost

 client:
 127.0.0.1   localhost
 255.255.255.255 broadcasthost
 ::1 localhost
 fe80::1%lo0 localhost

 172.16.144.150  cloudera


 Shengjie


 On 25 August 2013 20:28, Jean-Marc Spaggiari jean-m...@spaggiari.org
 wrote:

  Hi,
 
  Do you have your host+ip in your host file?
 
  JM
 
  2013/8/25 Shengjie Min shengjie@gmail.com
 
   Hi guys,
  
   I am running cloudera quick start vm:
  
   Versions:
  
   Hadoop: 2.0.0-cdh4.3.1
  
   HBase: 0.94.6-cdh4.3.1
  
   Here is my little remote HBase Java client(HbaseClient.java), all it
 does
   is:
  
   public static void main(String[] args) throws IOException {
   Configuration config = HBaseConfiguration.create();
   HTable table = new HTable(config, s1);
   System.out.println(table.getTableName());
   }
  
   hbase-site.xml:
  
   property
   namehbase.rootdir/name
   valuehdfs://172.16.144.150:8020/hbase/value
 /property
 property
   namezookeeper.znode.parent/name
   value/hbase/value
 /property
 ...
 property
   namezookeeper.znode.rootserver/name
   valueroot-region-server/value
 /property
 property
   namehbase.zookeeper.quorum/name
   value172.16.144.150/value
 /property
  
   When I run my java client, I get the error unknown host:
   localhost.localdomain:
  
   Initiating client connection, connectString=172.16.144.150:2181
   sessionTimeout=6 watcher=hconnection
   zookeeper.disableAutoWatchReset is false
   The identifier of this process is 41939@smin-MacBook-Pro.local
   Opening socket connection to server cloudera/172.16.144.150:2181. Will
   not attempt to authenticate using SASL (unknown error)
   Socket connection established to cloudera/172.16.144.150:2181,
   initiating session
   Session establishment request sent on cloudera/172.16.144.150:2181
   Session establishment complete on server cloudera/172.16.144.150:2181,
   sessionid = 0x14076b058850045, negotiated timeout = 6
   hconnection Received ZooKeeper Event, type=None, state=SyncConnected,
   path=null
   hconnection-0x14076b058850045 connected
  
   locateRegionInMeta parentTable=-ROOT-,
   metaLocation={region=-ROOT-,,0.70236052,
   hostname=localhost.localdomain, port=60020},
   attempt=0 of 10 failed; retrying after sleep of 1000 because:
   unknown host: localhost.localdomain
  
   Reading reply sessionid:0x14076b058850045, packet:: clientPath:null
   serverPath:null finished:false header:: 17,3  replyHeader:: 17,460,0
   request:: '/hbase,F  response::
   s{20,20,1376375496826,1376375496826,0,32,0,0,0,12,430}
   Reading reply sessionid:0x14076b058850045, packet:: clientPath:null
   serverPath:null finished:false header:: 18,4  replyHeader:: 18,460,0
   request:: '/hbase/root-region-server,T  response::
  
  
 
 #0001b3635323630406c6f63616c686f73742e6c6f63616c646f6d61696e6c6f63616c686f73742e6c6f63616c646f6d61696e2c36303032302c31333737333538373930303037,s{430,430,1377358805621,1377358805621,0,0,0,0,73,0,430}
   hconnection-0x14076b058850045 Retrieved 41 byte(s) of data from znode
   /hbase/root-region-server and set watcher;
   localhost.localdomain,60020,1...
   Looked up root region location,
  
  
 
 connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@68a7a3a7
   ;
   serverName=localhost.localdomain,60020,1377358790007
  
   locateRegionInMeta parentTable=.META., metaLocation=null, attempt=2 of
   10 failed;
   retrying after sle
  
   Any ideas?
  
   Thanks,
  
   SHengjie
  
 

Re: HBase Java client - unknown host: localhost.localdomain

[Kevin O'dell]

Shengjie,

  Good news on being able to connect to the ui from the client. Can you
post a screen shot on imgur?
On Aug 25, 2013 10:12 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:

 Can you share your client code? Or at least the section where you do the
 connection?

 Also, have you tried to run the same code from inside the VM?

 JM

 2013/8/25 Shengjie Min shengjie@gmail.com

  Hi Jean-Marc,
 
  Sorry about the screenshot, yes, my Hbase client is outside of my VM.
 And I
  can access WebUI from outside: http://cloudera:60010
 
  Shengjie
 
 
  On 25 August 2013 21:43, Jean-Marc Spaggiari jean-m...@spaggiari.org
  wrote:
 
   I think you need to post the screenshot on an external website and send
  the
   link here.
  
   So, you are trying to access your VM from outside of your VM? Or your
   client application is into the VM too?
  
   If you are outside of your VM, are you able to access the VM from
  outside?
  
   Like, are you able to access the WebUI from outside of the VM with
   something like http://cloudera:60010;?
  
   JM
  
   2013/8/25 Shengjie Min shengjie@gmail.com
  
   
   
   
On 25 August 2013 21:08, Kevin O'dell kevin.od...@cloudera.com
  wrote:
   
Can you attach a screen shot of the HMaster UI? It appears ZK is
connecting
fine, but can't find .META.
On Aug 25, 2013 8:57 AM, Shengjie Min shengjie@gmail.com
  wrote:
   
 Hi Jean-Marc,

 You meant my cloudera vm or my client? Here is my /etc/hosts

 cloudera vm:

 127.0.0.1   localhost.localdomain   localhost

 client:
 127.0.0.1   localhost
 255.255.255.255 broadcasthost
 ::1 localhost
 fe80::1%lo0 localhost

 172.16.144.150  cloudera


 Shengjie


 On 25 August 2013 20:28, Jean-Marc Spaggiari 
  jean-m...@spaggiari.org
 wrote:

  Hi,
 
  Do you have your host+ip in your host file?
 
  JM
 
  2013/8/25 Shengjie Min shengjie@gmail.com
 
   Hi guys,
  
   I am running cloudera quick start vm:
  
   Versions:
  
   Hadoop: 2.0.0-cdh4.3.1
  
   HBase: 0.94.6-cdh4.3.1
  
   Here is my little remote HBase Java client(HbaseClient.java),
  all
   it
 does
   is:
  
   public static void main(String[] args) throws IOException {
   Configuration config = HBaseConfiguration.create();
   HTable table = new HTable(config, s1);
   System.out.println(table.getTableName());
   }
  
   hbase-site.xml:
  
   property
   namehbase.rootdir/name
   valuehdfs://172.16.144.150:8020/hbase/value
 /property
 property
   namezookeeper.znode.parent/name
   value/hbase/value
 /property
 ...
 property
   namezookeeper.znode.rootserver/name
   valueroot-region-server/value
 /property
 property
   namehbase.zookeeper.quorum/name
   value172.16.144.150/value
 /property
  
   When I run my java client, I get the error unknown host:
   localhost.localdomain:
  
   Initiating client connection, connectString=
 172.16.144.150:2181
   sessionTimeout=6 watcher=hconnection
   zookeeper.disableAutoWatchReset is false
   The identifier of this process is 41939@smin-MacBook-Pro.local
   Opening socket connection to server cloudera/
  172.16.144.150:2181.
Will
   not attempt to authenticate using SASL (unknown error)
   Socket connection established to cloudera/172.16.144.150:2181
 ,
   initiating session
   Session establishment request sent on cloudera/
   172.16.144.150:2181
   Session establishment complete on server cloudera/
172.16.144.150:2181,
   sessionid = 0x14076b058850045, negotiated timeout = 6
   hconnection Received ZooKeeper Event, type=None,
state=SyncConnected,
   path=null
   hconnection-0x14076b058850045 connected
  
   locateRegionInMeta parentTable=-ROOT-,
   metaLocation={region=-ROOT-,,0.70236052,
   hostname=localhost.localdomain, port=60020},
   attempt=0 of 10 failed; retrying after sleep of 1000 because:
   unknown host: localhost.localdomain
  
   Reading reply sessionid:0x14076b058850045, packet::
   clientPath:null
   serverPath:null finished:false header:: 17,3  replyHeader::
   17,460,0
   request:: '/hbase,F  response::
   s{20,20,1376375496826,1376375496826,0,32,0,0,0,12,430}
   Reading reply sessionid:0x14076b058850045, packet::
   clientPath:null
   serverPath:null finished:false header:: 18,4  replyHeader::
   18,460,0
   request:: '/hbase/root-region-server,T  response::
  
  
 

   
  
 
 #0001b3635323630406c6f63616c686f73742e6c6f63616c646f6d61696e6c6f63616c686f73742e6c6f63616c646f6d61696e2c36303032302c31333737333538373930303037,s

Re: HBase Java client - unknown host: localhost.localdomain

[Kevin O'dell]

Shengjie,

  Looks like you are binding to localhost on your services. Please make
sure you correct it so you bind on the interface for zk.
On Aug 25, 2013 10:32 AM, Shengjie Min shengjie@gmail.com wrote:

 Sure, Kevin,

 http://imgur.com/SQ3Zao9

 Shengjie


 On 25 August 2013 22:22, Kevin O'dell kevin.od...@cloudera.com wrote:

  Shengjie,
 
Good news on being able to connect to the ui from the client. Can you
  post a screen shot on imgur?
  On Aug 25, 2013 10:12 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org
 
  wrote:
 
   Can you share your client code? Or at least the section where you do
 the
   connection?
  
   Also, have you tried to run the same code from inside the VM?
  
   JM
  
   2013/8/25 Shengjie Min shengjie@gmail.com
  
Hi Jean-Marc,
   
Sorry about the screenshot, yes, my Hbase client is outside of my VM.
   And I
can access WebUI from outside: http://cloudera:60010
   
Shengjie
   
   
On 25 August 2013 21:43, Jean-Marc Spaggiari 
 jean-m...@spaggiari.org
wrote:
   
 I think you need to post the screenshot on an external website and
  send
the
 link here.

 So, you are trying to access your VM from outside of your VM? Or
 your
 client application is into the VM too?

 If you are outside of your VM, are you able to access the VM from
outside?

 Like, are you able to access the WebUI from outside of the VM with
 something like http://cloudera:60010;?

 JM

 2013/8/25 Shengjie Min shengjie@gmail.com

 
 
 
  On 25 August 2013 21:08, Kevin O'dell kevin.od...@cloudera.com
wrote:
 
  Can you attach a screen shot of the HMaster UI? It appears ZK is
  connecting
  fine, but can't find .META.
  On Aug 25, 2013 8:57 AM, Shengjie Min shengjie@gmail.com
 
wrote:
 
   Hi Jean-Marc,
  
   You meant my cloudera vm or my client? Here is my /etc/hosts
  
   cloudera vm:
  
   127.0.0.1   localhost.localdomain   localhost
  
   client:
   127.0.0.1   localhost
   255.255.255.255 broadcasthost
   ::1 localhost
   fe80::1%lo0 localhost
  
   172.16.144.150  cloudera
  
  
   Shengjie
  
  
   On 25 August 2013 20:28, Jean-Marc Spaggiari 
jean-m...@spaggiari.org
   wrote:
  
Hi,
   
Do you have your host+ip in your host file?
   
JM
   
2013/8/25 Shengjie Min shengjie@gmail.com
   
 Hi guys,

 I am running cloudera quick start vm:

 Versions:

 Hadoop: 2.0.0-cdh4.3.1

 HBase: 0.94.6-cdh4.3.1

 Here is my little remote HBase Java
  client(HbaseClient.java),
all
 it
   does
 is:

 public static void main(String[] args) throws IOException
 {
 Configuration config = HBaseConfiguration.create();
 HTable table = new HTable(config, s1);
 System.out.println(table.getTableName());
 }

 hbase-site.xml:

 property
 namehbase.rootdir/name
 valuehdfs://172.16.144.150:8020/hbase/value
   /property
   property
 namezookeeper.znode.parent/name
 value/hbase/value
   /property
   ...
   property
 namezookeeper.znode.rootserver/name
 valueroot-region-server/value
   /property
   property
 namehbase.zookeeper.quorum/name
 value172.16.144.150/value
   /property

 When I run my java client, I get the error unknown host:
 localhost.localdomain:

 Initiating client connection, connectString=
   172.16.144.150:2181
 sessionTimeout=6 watcher=hconnection
 zookeeper.disableAutoWatchReset is false
 The identifier of this process is
  41939@smin-MacBook-Pro.local
 Opening socket connection to server cloudera/
172.16.144.150:2181.
  Will
 not attempt to authenticate using SASL (unknown error)
 Socket connection established to cloudera/
  172.16.144.150:2181
   ,
 initiating session
 Session establishment request sent on cloudera/
 172.16.144.150:2181
 Session establishment complete on server cloudera/
  172.16.144.150:2181,
 sessionid = 0x14076b058850045, negotiated timeout = 6
 hconnection Received ZooKeeper Event, type=None,
  state=SyncConnected,
 path=null
 hconnection-0x14076b058850045 connected

 locateRegionInMeta parentTable=-ROOT-,
 metaLocation={region=-ROOT-,,0.70236052,
 hostname=localhost.localdomain, port=60020},
 attempt=0 of 10 failed; retrying after sleep of 1000
  because:
 unknown host: localhost.localdomain

Re: Question about the time to execute joins in HBase!

[Kevin O'dell]

QQ what is your caching set to?
On Aug 22, 2013 11:25 AM, Pavan Sudheendra pavan0...@gmail.com wrote:

 Hi all,

 A serious question.. I know this isn't one of the best hbase practices but
 I really want to know..

 I am doing a join across 3 table in hbase.. One table contain 19m records,
 one contains 2m and another contains 1m records.

 I'm doing this inside the mapper function.. I know this can be done with
 pig and hive etc. Leaving the specifics out, how long would experts think
 it would take for the mapper to finish aggregating them across a 6 node
 cluster.. One is the job tracker and 5 are task trackers.. By the time I
 see the map reduce job status for input records reach 600,000 it's taking
 an hour.. It can't be right..

 Any tips? Please help.

 Thanks.

 --
 Regards-
 Pavan

Re: How many column families in one table ?

[Kevin O'dell]

Pablo,

  That is correct.


On Mon, Aug 5, 2013 at 10:00 AM, Pablo Medina pablomedin...@gmail.comwrote:

 Lars,

 when you say 'when one memstore needs to be flushed all other column
 families are flushed', are you referring to other column families of the
 same table, right?




 2013/8/4 Rohit Kelkar rohitkel...@gmail.com

  Regarding slow scan- only fetch the columns /qualifiers that you need. It
  may be that you are fetching a whole lot of data that you don't need. Try
  scan.addColumn() and let us know.
 
  - R
 
  On Sunday, August 4, 2013, lars hofhansl wrote:
 
   BigTable has one more level of abstraction: Locality Groups
   A Column Family in HBase is both a Column Faimily and a Locality Group:
  It
   is a group of columns *and* it defines storage parameters (compression,
   versions, TTL, etc).
  
   As to how many make sense. It depends.
   If you can group your columns such that a scan is often limited to a
   single Column Family, you'll get huge benefit by using more Column
  Families.
   The main consideration for many Column Families and that each has its
 own
   store files, and hence scanning involves more seeking for each Column
   Families included in a scan.
  
   They are also flushed together; when one memstore (which is per Column
   Family) needs to be flushed all other Column Families are also flushed
   leading to many small files until they are compacted. If all your
 Column
   Faimilies are roughly the same size this is less of a problem. It's
 also
   possible to mitigate this by tweaking the compaction policies.
  
  
   -- Lars
  
  
  
   
From: Vimal Jain vkj...@gmail.com javascript:;
   To: user@hbase.apache.org javascript:;
   Sent: Saturday, August 3, 2013 11:28 PM
   Subject: Re: How many column families in one table ?
  
  
   Hi,
   I have tested read performance after reducing number of column families
   from 14 to 3 and yes there is improvement.
   Meanwhile i was going through the paper published by google on
 BigTable.
   It says
  
   It is our intent that the number of distinct column
   families in a table be small (in the hundreds at most), and
   that families rarely change during operation.
  
   So Is that theoretical value ( 100 CFs )  or its possible but not with
  the
   current version of Hbase ?
  
  
   On Tue, Jul 2, 2013 at 12:48 AM, Viral Bajaria 
 viral.baja...@gmail.com
  javascript:;
   wrote:
  
On Mon, Jul 1, 2013 at 10:06 AM, Vimal Jain vkj...@gmail.com
  javascript:;
   wrote:
   
 Sorry for the typo .. please ignore previous mail.. Here is the
   corrected
 one..
 1)I have around 140 columns for each row , out of 140 , around 100
columns
 hold java primitive data type , remaining 40 columns  contain
   serialized
 java object as byte array(Inside each object is an ArrayList). Yes
 ,
  I
   do
 delete data but the frequency is very less ( 1 out of 5K operations
  ).
   I
 dont run any compaction.

   
This answers the type of data in each cell not the size of data. Can
  you
figure out the average size of data that you insert in that size. For
example what is the length of the byte array ? Also for java
 primitive,
   is
it 8-byte long ? 4-byte int ?
In addition to that, what is in the row key ? How long is that in
  bytes ?
Same for column family, can you share the names of the column family
 ?
   How
about qualifiers ?
   
If you have disabled major compactions, you should run it once a few
  days
(if not once a day) to consolidate the # of files that each scan will
   have
to open.
   
2) I had ran scan keeping in mind the CPU,IO and other system related
 parameters.I found them to be normal with system load being
 0.1-0.3.

   
How many disks do you have in your box ? Have you ever benchmarked
 the
hardware ?
   
Thanks,
Viral
   
  
  
  
   --
   Thanks and Regards,
   Vimal Jain
 




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Memory distribution for Hadoop/Hbase processes

[Kevin O'dell]

My questions are :
1) How this thing is working ? It is working because java can over allocate
memory. You will know you are using too much memory when the kernel starts
killing processes.
2) I just have one table whose size at present is about 10-15 GB , so what
should be ideal memory distribution ? Really you should get a box with more
memory. You can currently only hold about ~400 MB in memory.
On Aug 4, 2013 9:58 AM, Ted Yu yuzhih...@gmail.com wrote:

 What OS are you using ?

 What is the output from the following command ?
  ps aux | grep pid
 where pid is the process Id for Namenode, Datanode, etc.

 Cheers

 On Sun, Aug 4, 2013 at 6:33 AM, Vimal Jain vkj...@gmail.com wrote:

  Hi,
  I have configured Hbase in pseudo distributed mode with HDFS as
 underlying
  storage.I am not using map reduce framework as of now
  I have 4GB RAM.
  Currently i have following distribution of memory
 
  Data Node,Name Node,Secondary Name Node each :1000MB(default
  HADOOP_HEAPSIZE
  property)
 
  Hmaster - 512 MB
  HRegion - 1536 MB
  Zookeeper - 512 MB
 
  So total heap allocation becomes - 5.5 GB which is absurd as my total RAM
  is only 4 GB , but still the setup is working fine on production. :-0
 
  My questions are :
  1) How this thing is working ?
  2) I just have one table whose size at present is about 10-15 GB , so
 what
  should be ideal memory distribution ?
  --
  Thanks and Regards,
  Vimal Jain
 

Re: How many column families in one table ?

[Kevin O'dell]

Hi Inder,

  Here is an excellent blog post which is a little dated:

http://www.larsgeorge.com/2009/11/hbase-vs-bigtable-comparison.html?m=1
On Aug 4, 2013 10:55 AM, Inder Pall inder.p...@gmail.com wrote:

 Kevin

 Would love to hear your thoughts around hbase  not big table.

 Thanks

 inder
 you are the average of 5 people you spend the most time with
 On Aug 4, 2013 8:15 PM, Kevin O'dell kevin.od...@cloudera.com wrote:

  Hi Vimal,
 
It really depends on your usage pattern but HBase != Bigtable.
  On Aug 4, 2013 2:29 AM, Vimal Jain vkj...@gmail.com wrote:
 
   Hi,
   I have tested read performance after reducing number of column families
   from 14 to 3 and yes there is improvement.
   Meanwhile i was going through the paper published by google on
 BigTable.
   It says
  
   It is our intent that the number of distinct column
   families in a table be small (in the hundreds at most), and
   that families rarely change during operation.
  
   So Is that theoretical value ( 100 CFs )  or its possible but not with
  the
   current version of Hbase ?
  
  
   On Tue, Jul 2, 2013 at 12:48 AM, Viral Bajaria 
 viral.baja...@gmail.com
   wrote:
  
On Mon, Jul 1, 2013 at 10:06 AM, Vimal Jain vkj...@gmail.com
 wrote:
   
 Sorry for the typo .. please ignore previous mail.. Here is the
   corrected
 one..
 1)I have around 140 columns for each row , out of 140 , around 100
columns
 hold java primitive data type , remaining 40 columns  contain
   serialized
 java object as byte array(Inside each object is an ArrayList). Yes
 ,
  I
   do
 delete data but the frequency is very less ( 1 out of 5K operations
  ).
   I
 dont run any compaction.

   
This answers the type of data in each cell not the size of data. Can
  you
figure out the average size of data that you insert in that size. For
example what is the length of the byte array ? Also for java
 primitive,
   is
it 8-byte long ? 4-byte int ?
In addition to that, what is in the row key ? How long is that in
  bytes ?
Same for column family, can you share the names of the column family
 ?
   How
about qualifiers ?
   
If you have disabled major compactions, you should run it once a few
  days
(if not once a day) to consolidate the # of files that each scan will
   have
to open.
   
2) I had ran scan keeping in mind the CPU,IO and other system related
 parameters.I found them to be normal with system load being
 0.1-0.3.

   
How many disks do you have in your box ? Have you ever benchmarked
 the
hardware ?
   
Thanks,
Viral
   
  
  
  
   --
   Thanks and Regards,
   Vimal Jain
  
 

Re: AssignmentManager looping?

[Kevin O'dell]

Does it exist in meta or hdfs?
On Aug 1, 2013 8:24 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:

 My master keep logging that:

 2013-07-31 21:52:59,201 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:52:59,201 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
 it doesn't exist anymore, probably already processed its split
 2013-07-31 21:52:59,339 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:52:59,339 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
 it doesn't exist anymore, probably already processed its split
 2013-07-31 21:52:59,461 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:52:59,461 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
 it doesn't exist anymore, probably already processed its split
 2013-07-31 21:52:59,636 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:52:59,636 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
 it doesn't exist anymore, probably already processed its split
 2013-07-31 21:53:00,074 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:53:00,074 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
 it doesn't exist anymore, probably already processed its split
 2013-07-31 21:53:00,261 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:53:00,261 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
 it doesn't exist anymore, probably already processed its split
 2013-07-31 21:53:00,417 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:53:00,417 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
 it doesn't exist anymore, probably already processed its split

 hbase@node3:~/hbase-0.94.3$ cat logs/hbase-hbase-master-node3.log* | grep
 Region 270a9c371fcbe9cd9a04986e0b77d16b not found  | wc
5042   65546  927728


 Then crashed.
 2013-07-31 22:22:46,072 FATAL org.apache.hadoop.hbase.master.HMaster:
 Master server abort: loaded coprocessors are: []
 2013-07-31 22:22:46,073 FATAL org.apache.hadoop.hbase.master.HMaster:
 Unexpected state : work_proposed,\x02\xE8\x92'\x00\x00\x00\x00

 http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
 state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 ..
 Cannot
 transit it to OFFLINE.
 java.lang.IllegalStateException: Unexpected state :
 work_proposed,\x02\xE8\x92'\x00\x00\x00\x00

 http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
 state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 ..
 Cannot
 transit it to OFFLINE.
 at

 org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
 at

 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
 at

 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
 at

 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
 at

 org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
 at

 org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
 at
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
 at

 

Re: AssignmentManager looping?

[Kevin O'dell]

Yes you can if HBase is down, first I would copy .META out of HDFS local
and then you can search it for split issues. Deleting those znodes should
clear this up though.
On Aug 1, 2013 8:52 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:

 I can't check the meta since HBase is down.

 Regarding HDFS, I took few random lines like:
 2013-08-01 08:45:57,260 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 28328fdb7181cbd9cc4d6814775e8895 not found on server
 node4,60020,1375319042033; failed processing
 2013-08-01 08:45:57,260 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
 28328fdb7181cbd9cc4d6814775e8895 from server node4,60020,1375319042033 but
 it doesn't exist anymore, probably already processed its split

 And each time, there is nothing like that.
 hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -lsr / | grep
 28328fdb7181cbd9cc4d6814775e8895

 On ZK side:
 [zk: localhost:2181(CONNECTED) 3] ls /hbase/splitlog

 [zk: localhost:2181(CONNECTED) 10] ls /hbase/unassigned
 [28328fdb7181cbd9cc4d6814775e8895, a8781a598c46f19723a2405345b58470,
 b7ebfeb63b10997736fd12920fde2bb8, d95bb27cc026511c2a8c8ad155e79bf6,
 270a9c371fcbe9cd9a04986e0b77d16b, aff4d1d8bf470458bb19525e8aef0759]

 Can I just delete those zknodes? Worst case hbck will find them back from
 HDFS if required?

 JM

 2013/8/1 Kevin O'dell kevin.od...@cloudera.com

  Does it exist in meta or hdfs?
  On Aug 1, 2013 8:24 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org
  wrote:
 
   My master keep logging that:
  
   2013-07-31 21:52:59,201 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,201 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,339 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,339 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,461 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,461 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,636 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,636 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:53:00,074 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:53:00,074 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:53:00,261 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:53:00,261 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:53:00,417 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:53:00,417 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
  
   hbase@node3:~/hbase-0.94.3$ cat logs/hbase-hbase-master-node3.log* |
  grep
   Region 270a9c371fcbe9cd9a04986e0b77d16b not found  | wc
  5042   65546  927728
  
  
   Then crashed.
   2013-07-31 22:22:46,072 FATAL org.apache.hadoop.hbase.master.HMaster

Re: AssignmentManager looping?

[Kevin O'dell]

JM,

Stop HBase
rmr /hbase from zkcli
Sideline META
Run offline meta repair
Start HBase
On Aug 1, 2013 1:01 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:

 Hi Jimmy,

 I should still have all the logs.

 What I did is pretty simple.

 I tried to turn the cluster off while a single regioned 250GB table was
 under major_compaction to get splitted.

 I will targz all the logs for the few last days and make that available.

 On the other side, I'm still not able to bring it back up...

 JM

 2013/8/1 Jimmy Xiang jxi...@cloudera.com

  Something went wrong with split.  It should be easy to fix your cluster.
  However, it will be more interesting to find out how it happened. Do you
  remember what has happened since it was good previously? Do you have all
  the logs?
 
 
  On Thu, Aug 1, 2013 at 7:08 AM, Jean-Marc Spaggiari 
  jean-m...@spaggiari.org
   wrote:
 
   I tried to remove the znodes but got the same result. So I shutted down
  all
   the RS and restarted HBase, and now I have 0 regions for this table.
   Running HBCK. Seems that it has a lot to do...
  
   2013/8/1 Kevin O'dell kevin.od...@cloudera.com
  
Yes you can if HBase is down, first I would copy .META out of HDFS
  local
and then you can search it for split issues. Deleting those znodes
  should
clear this up though.
On Aug 1, 2013 8:52 AM, Jean-Marc Spaggiari 
 jean-m...@spaggiari.org
  
wrote:
   
 I can't check the meta since HBase is down.

 Regarding HDFS, I took few random lines like:
 2013-08-01 08:45:57,260 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 28328fdb7181cbd9cc4d6814775e8895 not found on server
 node4,60020,1375319042033; failed processing
 2013-08-01 08:45:57,260 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT
 for
region
 28328fdb7181cbd9cc4d6814775e8895 from server
  node4,60020,1375319042033
but
 it doesn't exist anymore, probably already processed its split

 And each time, there is nothing like that.
 hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -lsr / | grep
 28328fdb7181cbd9cc4d6814775e8895

 On ZK side:
 [zk: localhost:2181(CONNECTED) 3] ls /hbase/splitlog

 [zk: localhost:2181(CONNECTED) 10] ls /hbase/unassigned
 [28328fdb7181cbd9cc4d6814775e8895,
 a8781a598c46f19723a2405345b58470,
 b7ebfeb63b10997736fd12920fde2bb8, d95bb27cc026511c2a8c8ad155e79bf6,
 270a9c371fcbe9cd9a04986e0b77d16b, aff4d1d8bf470458bb19525e8aef0759]

 Can I just delete those zknodes? Worst case hbck will find them
 back
   from
 HDFS if required?

 JM

 2013/8/1 Kevin O'dell kevin.od...@cloudera.com

  Does it exist in meta or hdfs?
  On Aug 1, 2013 8:24 AM, Jean-Marc Spaggiari 
   jean-m...@spaggiari.org

  wrote:
 
   My master keep logging that:
  
   2013-07-31 21:52:59,201 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,201 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server
node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,339 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,339 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server
node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,461 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,461 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server
node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,636 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,636 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server
node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:53:00,074 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region

Re: AssignmentManager looping?

[Kevin O'dell]

If that doesn't work you probably have an invalid reference file and you
will find that in RS logs for the HLog split that is never finishing.
On Aug 1, 2013 1:38 PM, Kevin O'dell kevin.od...@cloudera.com wrote:

 JM,

 Stop HBase
 rmr /hbase from zkcli
 Sideline META
 Run offline meta repair
 Start HBase
 On Aug 1, 2013 1:01 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org
 wrote:

 Hi Jimmy,

 I should still have all the logs.

 What I did is pretty simple.

 I tried to turn the cluster off while a single regioned 250GB table was
 under major_compaction to get splitted.

 I will targz all the logs for the few last days and make that available.

 On the other side, I'm still not able to bring it back up...

 JM

 2013/8/1 Jimmy Xiang jxi...@cloudera.com

  Something went wrong with split.  It should be easy to fix your cluster.
  However, it will be more interesting to find out how it happened. Do you
  remember what has happened since it was good previously? Do you have all
  the logs?
 
 
  On Thu, Aug 1, 2013 at 7:08 AM, Jean-Marc Spaggiari 
  jean-m...@spaggiari.org
   wrote:
 
   I tried to remove the znodes but got the same result. So I shutted
 down
  all
   the RS and restarted HBase, and now I have 0 regions for this table.
   Running HBCK. Seems that it has a lot to do...
  
   2013/8/1 Kevin O'dell kevin.od...@cloudera.com
  
Yes you can if HBase is down, first I would copy .META out of HDFS
  local
and then you can search it for split issues. Deleting those znodes
  should
clear this up though.
On Aug 1, 2013 8:52 AM, Jean-Marc Spaggiari 
 jean-m...@spaggiari.org
  
wrote:
   
 I can't check the meta since HBase is down.

 Regarding HDFS, I took few random lines like:
 2013-08-01 08:45:57,260 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 28328fdb7181cbd9cc4d6814775e8895 not found on server
 node4,60020,1375319042033; failed processing
 2013-08-01 08:45:57,260 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT
 for
region
 28328fdb7181cbd9cc4d6814775e8895 from server
  node4,60020,1375319042033
but
 it doesn't exist anymore, probably already processed its split

 And each time, there is nothing like that.
 hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -lsr / | grep
 28328fdb7181cbd9cc4d6814775e8895

 On ZK side:
 [zk: localhost:2181(CONNECTED) 3] ls /hbase/splitlog

 [zk: localhost:2181(CONNECTED) 10] ls /hbase/unassigned
 [28328fdb7181cbd9cc4d6814775e8895,
 a8781a598c46f19723a2405345b58470,
 b7ebfeb63b10997736fd12920fde2bb8,
 d95bb27cc026511c2a8c8ad155e79bf6,
 270a9c371fcbe9cd9a04986e0b77d16b,
 aff4d1d8bf470458bb19525e8aef0759]

 Can I just delete those zknodes? Worst case hbck will find them
 back
   from
 HDFS if required?

 JM

 2013/8/1 Kevin O'dell kevin.od...@cloudera.com

  Does it exist in meta or hdfs?
  On Aug 1, 2013 8:24 AM, Jean-Marc Spaggiari 
   jean-m...@spaggiari.org

  wrote:
 
   My master keep logging that:
  
   2013-07-31 21:52:59,201 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,201 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server
node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,339 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,339 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server
node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,461 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,461 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b from server
node7,60020,1375319044055
  but
   it doesn't exist anymore, probably already processed its split
   2013-07-31 21:52:59,636 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   270a9c371fcbe9cd9a04986e0b77d16b not found on server
   node7,60020,1375319044055; failed processing
   2013-07-31 21:52:59,636 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   270a9c371fcbe9cd9a04986e0b77d16b

Re: AssignmentManager looping?

[Kevin O'dell]

Jimmy,

  Sounds like our dreaded reference file issue again. I spoke with JM and
he is going to try to reproduce this  My gut tells me our point of no
return may be in the wrong place due to some code change along the way, but
hbck could also just be doing something wonky.

JM,

  This cluster is not CM managed correct?
On Aug 1, 2013 1:49 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:

 So I had to remove few reference files and run few hbck to get everything
 back online.

 Summary: don't stop your cluster while it's major compacting huge tables ;)

 Thanks all!

 JM

 2013/8/1 Kevin O'dell kevin.od...@cloudera.com

  If that doesn't work you probably have an invalid reference file and you
  will find that in RS logs for the HLog split that is never finishing.
  On Aug 1, 2013 1:38 PM, Kevin O'dell kevin.od...@cloudera.com wrote:
 
   JM,
  
   Stop HBase
   rmr /hbase from zkcli
   Sideline META
   Run offline meta repair
   Start HBase
   On Aug 1, 2013 1:01 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org
 
   wrote:
  
   Hi Jimmy,
  
   I should still have all the logs.
  
   What I did is pretty simple.
  
   I tried to turn the cluster off while a single regioned 250GB table
 was
   under major_compaction to get splitted.
  
   I will targz all the logs for the few last days and make that
 available.
  
   On the other side, I'm still not able to bring it back up...
  
   JM
  
   2013/8/1 Jimmy Xiang jxi...@cloudera.com
  
Something went wrong with split.  It should be easy to fix your
  cluster.
However, it will be more interesting to find out how it happened. Do
  you
remember what has happened since it was good previously? Do you have
  all
the logs?
   
   
On Thu, Aug 1, 2013 at 7:08 AM, Jean-Marc Spaggiari 
jean-m...@spaggiari.org
 wrote:
   
 I tried to remove the znodes but got the same result. So I shutted
   down
all
 the RS and restarted HBase, and now I have 0 regions for this
 table.
 Running HBCK. Seems that it has a lot to do...

 2013/8/1 Kevin O'dell kevin.od...@cloudera.com

  Yes you can if HBase is down, first I would copy .META out of
 HDFS
local
  and then you can search it for split issues. Deleting those
 znodes
should
  clear this up though.
  On Aug 1, 2013 8:52 AM, Jean-Marc Spaggiari 
   jean-m...@spaggiari.org

  wrote:
 
   I can't check the meta since HBase is down.
  
   Regarding HDFS, I took few random lines like:
   2013-08-01 08:45:57,260 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Region
   28328fdb7181cbd9cc4d6814775e8895 not found on server
   node4,60020,1375319042033; failed processing
   2013-08-01 08:45:57,260 WARN
   org.apache.hadoop.hbase.master.AssignmentManager: Received
 SPLIT
   for
  region
   28328fdb7181cbd9cc4d6814775e8895 from server
node4,60020,1375319042033
  but
   it doesn't exist anymore, probably already processed its split
  
   And each time, there is nothing like that.
   hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -lsr / | grep
   28328fdb7181cbd9cc4d6814775e8895
  
   On ZK side:
   [zk: localhost:2181(CONNECTED) 3] ls /hbase/splitlog
  
   [zk: localhost:2181(CONNECTED) 10] ls /hbase/unassigned
   [28328fdb7181cbd9cc4d6814775e8895,
   a8781a598c46f19723a2405345b58470,
   b7ebfeb63b10997736fd12920fde2bb8,
   d95bb27cc026511c2a8c8ad155e79bf6,
   270a9c371fcbe9cd9a04986e0b77d16b,
   aff4d1d8bf470458bb19525e8aef0759]
  
   Can I just delete those zknodes? Worst case hbck will find
 them
   back
 from
   HDFS if required?
  
   JM
  
   2013/8/1 Kevin O'dell kevin.od...@cloudera.com
  
Does it exist in meta or hdfs?
On Aug 1, 2013 8:24 AM, Jean-Marc Spaggiari 
 jean-m...@spaggiari.org
  
wrote:
   
 My master keep logging that:

 2013-07-31 21:52:59,201 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:52:59,201 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received
   SPLIT
 for
region
 270a9c371fcbe9cd9a04986e0b77d16b from server
  node7,60020,1375319044055
but
 it doesn't exist anymore, probably already processed its
  split
 2013-07-31 21:52:59,339 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Region
 270a9c371fcbe9cd9a04986e0b77d16b not found on server
 node7,60020,1375319044055; failed processing
 2013-07-31 21:52:59,339 WARN
 org.apache.hadoop.hbase.master.AssignmentManager: Received
   SPLIT
 for
region
 270a9c371fcbe9cd9a04986e0b77d16b from server
  node7,60020,1375319044055
but
 it doesn't exist anymore

Re: Multiple region servers per physical node

[Kevin O'dell]

Elliot,

  Can you elaborate on the blocking being a major factor?  Are you
referring to the default value of 7 slowing down writes?  I don't think
removing that feature is a great idea.  Here are a couple things that it is
helpful for:

1.) Slows down the write path so we are less likely to end up with 1000
storefiles per region server
2.) Slows down the write path enough for the end user to realize it is
slow, and thus start troubleshooting/optimizing the write path.

I think it might be fair to bump to 15 - 20 by default, but this is the
LAST option I touch when troubleshooting the write path.  Usually when we
start to have blocking issues it is caused by a few other issues:

1.) Too many regions for the allowed memstore upper/lower limit and we are
flushing too small
2.) Too small of Hlogs/number of HLogs and we are prematurely rolling
3.) Ingest rate is too fast for the memstore size and needs to be raised.
4.) Slow/not enough drives to keep up with the compaction churn and needs
to be tuned

The defaults for HBase out of the box here are decent for testing, but not
optimized for a production workload.  If this is not what you were talking
about sorry for the long rant :)


On Tue, Jul 30, 2013 at 2:34 AM, Elliott Clark ecl...@apache.org wrote:

 G1 doesn't really make our write path much better if you have uneven
 region writes (zipfian distribution or the like).
 Lately I've been seeing the memstore blocking size per region being a
 major factor.  In fact I'm thinking of opening a jira to remove it by
 default.

 On Mon, Jul 29, 2013 at 4:12 PM, Andrew Purtell apurt...@apache.org
 wrote:
  Having a difficult time avoiding evacuation failures under concurrent
 read
  and write stress. Tuning helps but raises contention for CPU. This is
 with
  interesting heap sizes - 32-128GB.
 
 
  On Mon, Jul 29, 2013 at 3:44 PM, Bryan Beaudreault 
 bbeaudrea...@hubspot.com
  wrote:
 
  Due to the contention for CPU resources?  Or why?
 
 
  On Mon, Jul 29, 2013 at 6:23 PM, Andrew Purtell apurt...@apache.org
  wrote:
 
   On Mon, Jul 29, 2013 at 3:14 PM, Elliott Clark ecl...@apache.org
  wrote:
  
Yes the G1 looks promising if you have a very read heavy workload
(We've been running it for integration tests for about a month or
 so).
   
  
   That's my experience too. Unfortunately it's less promising once the
   workload is mixed.
  
   --
   Best regards,
  
  - Andy
  
   Problems worthy of attack prove their worth by hitting back. - Piet
 Hein
   (via Tom White)
  
 
 
 
 
  --
  Best regards,
 
 - Andy
 
  Problems worthy of attack prove their worth by hitting back. - Piet Hein
  (via Tom White)




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: Hbase: Is possible to filter by integer value if the value is saved as string?

[Kevin]

The only way I was able to do negative numbers was to create a new
WritableComparable that took signed-ness into account, but that involved
deploying the class to the cluster.


On Fri, Jul 19, 2013 at 10:44 AM, Frank Luo j...@merkleinc.com wrote:

 OK, let's say it is stored as an integer, how to compare then?

 As Jean-Marc pointed out, negative numbers is greater than positive ones
 using ByteArrayComparable.

 -Original Message-
 From: Jean-Marc Spaggiari [mailto:jean-m...@spaggiari.org]
 Sent: Friday, July 19, 2013 5:08 AM
 To: user@hbase.apache.org
 Subject: Re: Hbase: Is possible to filter by integer value if the value is
 saved as string?

 I agree with Anil... Store it as an integer, not as as String. That will
 help for the comparison (be carreful with negative values comparisons.).

 Worst case, run a MR job to convert them.

 2013/7/19 anil gupta anilgupt...@gmail.com

  Only way to achieve this is to write your own ByteArrayComparable.
  BinaryComparator wont work for your case.
  But, I am wondering why you would store an Integer as String when you
  want to do numerical comparison?
 
 
  On Thu, Jul 18, 2013 at 6:03 PM, Ted Yu yuzhih...@gmail.com wrote:
 
   What would happen to this ?
  
   System.out.println(c.compareTo(Bytes.toBytes(30)));
  
   On Thu, Jul 18, 2013 at 5:55 PM, Kevin kevin.macksa...@gmail.com
  wrote:
  
Sure, try using the BinaryComparator. For example,
   
BinaryComparator c = new
  BinaryComparator(Bytes.toBytes(200));
System.out.println(c.compareTo(Bytes.toBytes(201))); //
  returns
-1
   
   
On Thu, Jul 18, 2013 at 4:28 PM, Frank Luo j...@merkleinc.com
 wrote:
   
 That requires creating my own ByteArrayComparable class and
 deploy to
   all
 servers, right?

 My company doesn't want to customize hbase, hence is not an
 option
  to
me.

 -Original Message-
 From: Ted Yu [mailto:yuzhih...@gmail.com]
 Sent: Thursday, July 18, 2013 3:25 PM
 To: user@hbase.apache.org
 Subject: Re: Hbase: Is possible to filter by integer value if
 the
  value
is
 saved as string?

 Looks like you should be able to do so by passing your own
 comparator
   to:

   public SingleColumnValueFilter(final byte [] family, final
 byte [] qualifier,

   final CompareOp compareOp, final ByteArrayComparable
  comparator)
   {
 Cheers

 On Thu, Jul 18, 2013 at 1:20 PM, Frank Luo j...@merkleinc.com
  wrote:

  I don't think it is possible, but would like to confirm with
  smart folks out there.
 
  Supposing I have a cell storing an integer but in string
   presentation.
  For example, here is how I put a value of 200:
 
  put.add(family, qualifier, Bytes.toBytes(200));
 
  Now, I want to scan with a filter that only return if the
  value is larger than 250. Is that possible?
 
  Thanks in advance
 
  Frank
 


   
  
 
 
 
  --
  Thanks  Regards,
  Anil Gupta
 

Re: Hbase: Is possible to filter by integer value if the value is saved as string?

[Kevin]

Sure, try using the BinaryComparator. For example,

BinaryComparator c = new BinaryComparator(Bytes.toBytes(200));
System.out.println(c.compareTo(Bytes.toBytes(201))); // returns -1


On Thu, Jul 18, 2013 at 4:28 PM, Frank Luo j...@merkleinc.com wrote:

 That requires creating my own ByteArrayComparable class and deploy to all
 servers, right?

 My company doesn't want to customize hbase, hence is not an option to me.

 -Original Message-
 From: Ted Yu [mailto:yuzhih...@gmail.com]
 Sent: Thursday, July 18, 2013 3:25 PM
 To: user@hbase.apache.org
 Subject: Re: Hbase: Is possible to filter by integer value if the value is
 saved as string?

 Looks like you should be able to do so by passing your own comparator to:

   public SingleColumnValueFilter(final byte [] family, final byte []
 qualifier,

   final CompareOp compareOp, final ByteArrayComparable comparator) {
 Cheers

 On Thu, Jul 18, 2013 at 1:20 PM, Frank Luo j...@merkleinc.com wrote:

  I don't think it is possible, but would like to confirm with smart
  folks out there.
 
  Supposing I have a cell storing an integer but in string presentation.
  For example, here is how I put a value of 200:
 
  put.add(family, qualifier, Bytes.toBytes(200));
 
  Now, I want to scan with a filter that only return if the value is
  larger than 250. Is that possible?
 
  Thanks in advance
 
  Frank
 

Re: G1 before/after GC time graph

[Kevin O'dell]

Anyway you can test this with a 32GB heap? :)


On Wed, Jul 10, 2013 at 5:23 PM, Otis Gospodnetic 
otis.gospodne...@gmail.com wrote:

 But, very importantly:

 CPU after changes: https://apps.sematext.com/spm/s/CpZ9aPhL9L
 Load after changes: https://apps.sematext.com/spm/s/i7f3x6MXdZ

 So in practical terms, use something like -XX:MaxGCPauseMillis=200
 -XX:NewRatio=4 to force short GC pauses, but count on having more of
 them and CPU sweating more.  This gives you smaller pauses, no big STW
 pauses, but be prepared to pay in CPU cycles.

 HTH

 Otis
 --
 HBase Performance Monitoring -- http://sematext.com/spm



 On Wed, Jul 10, 2013 at 4:20 PM, Otis Gospodnetic
 otis.gospodne...@gmail.com wrote:
  Not big in this case - 1800m
 
  Otis
  --
  HBase Performance Monitoring -- http://sematext.com/spm
 
 
 
  On Wed, Jul 10, 2013 at 3:57 PM, Vladimir Rodionov
  vrodio...@carrieriq.com wrote:
 
  What is  -Xmx heap size?
 
  Best regards,
  Vladimir Rodionov
  Principal Platform Engineer
  Carrier IQ, www.carrieriq.com
  e-mail: vrodio...@carrieriq.com
 
  
  From: Otis Gospodnetic [otis.gospodne...@gmail.com]
  Sent: Wednesday, July 10, 2013 12:00 PM
  To: user@hbase.apache.org
  Subject: G1 before/after GC time graph
 
  Hi,
 
  There was talk of G1 and HBase here over the past few days.  I wanted
  to share a fresh and telling SPM graph showing G1 + some other params
  in action:
 
  Result first: https://apps.sematext.com/spm/s/wN0hKVixxX
 
  This is what we had before:
-XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC
  -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=25
 
  And then we added:
-XX:MaxGCPauseMillis=200 -XX:NewRatio=4
 
  I hope this helps.
 
  Otis
  --
  HBase Performance Monitoring -- http://sematext.com/spm
 
  Confidentiality Notice:  The information contained in this message,
 including any attachments hereto, may be confidential and is intended to be
 read only by the individual or entity to whom this message is addressed. If
 the reader of this message is not the intended recipient or an agent or
 designee of the intended recipient, please note that any review, use,
 disclosure or distribution of this message or its attachments, in any form,
 is strictly prohibited.  If you have received this message in error, please
 immediately notify the sender and/or notificati...@carrieriq.com and
 delete or destroy any copy of this message and its attachments.




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: GC recommendations for large Region Server heaps

[Kevin O'dell]

Hey Suraj,

  I would recommend turning on MSlab and using the following settings:

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled



On Mon, Jul 8, 2013 at 2:09 PM, Suraj Varma svarma...@gmail.com wrote:

 Hello:
 We have an HBase cluster with region servers running on 8GB heap size with
 a 0.6 block cache (it is a read heavy cluster, with bursty write traffic
 via MR jobs). (version: hbase-0.94.6.1)

 During HBaseCon, while speaking to a few attendees, I heard some folks were
 running region servers as high as 24GB and some others in the 16GB range.

 So - question: Are there any special GC recommendations (tuning parameters,
 flags, etc) that folks who run at these large heaps can recommend while
 moving up from an 8GB heap? i.e. for 16GB and for 24GB RS heaps ... ?

 I'm especially concerned about long pauses causing zk session timeouts and
 consequent RS shutdowns. Our boxes do have a lot of RAM and we are
 exploring how we can use more of it for the cluster while maintaining
 overall stability.

 Also - if there are clusters running multiple region servers per host, I'd
 be very interested to know what RS heap sizes those are being run at ...
 and whether this was chosen as an alternative to running a single RS with
 large heap.

 (I know I'll have to test the GC stuff out on my cluster and for my
 workloads anyway ... but just trying to get a feel of what sort of tuning
 options had to be used to have a stable HBase cluster with 16 or 24GB RS
 heaps).

 Thanks in advance,
 --Suraj




-- 
Kevin O'Dell
Systems Engineer, Cloudera

Re: how many severs in a hbase cluster

[Kevin O'dell]

Mohammad,

  The NN is low write, and has pretty static memory usage.  You will see
the NN memory usage go up as you add blocks/files.  Since, HBase has memory
limitations(GC's Fault), and should have ~1 file per store you will not
have a lot of memory pressure on the NN.  The JT is the same way, it scales
up usage based on number of MR jobs.  In a sane HBase environment you are
not going to be running 1000s of MR jobs against HBase.  ZK also has pretty
minimum requirements - 1GB of memory, dedicated CPU core, and place to
write to with low I/O wait.  I have always found the NN, SNN, and JT to be
the next best place to put the ZK if dedicated HW is not available.  I have
seen some strange behavior with ZK runs on DN/TT/RS nodes.  From
unexplained timeouts to corrupt znodes causing failures(This one was real
nasty).


On Sat, Jun 22, 2013 at 7:21 PM, Mohammad Tariq donta...@gmail.com wrote:

 Hello Iain,

  You would put a lot of pressure on the RAM if you do that. NN
 already has high memory requirement and then having JT+ZK on the same
 machine would be too heavy, IMHO.

 Warm Regards,
 Tariq
 cloudfront.blogspot.com


 On Sun, Jun 23, 2013 at 4:07 AM, iain wright iainw...@gmail.com wrote:

  Hi Mohammad,
 
  I am curious why you chose not to put the third ZK on the NN+JT? I was
  planning on doing that on a new cluster and want to confirm it would be
  okay.
 
 
  --
  Iain Wright
  Cell: (562) 852-5916
 
  http://www.labctsi.org/
  This email message is confidential, intended only for the recipient(s)
  named above and may contain information that is privileged, exempt from
  disclosure under applicable law. If you are not the intended recipient,
 do
  not disclose or disseminate the message to anyone except the intended
  recipient. If you have received this message in error, or are not the
 named
  recipient(s), please immediately notify the sender by return email, and
  delete all copies of this message.
 
 
  On Sat, Jun 22, 2013 at 10:05 AM, Mohammad Tariq donta...@gmail.com
  wrote:
 
   Yeah, I forgot to mention that no. of ZKs should be odd. Perhaps those
   parentheses made that statement look like an optional statement. Just
 to
   clarify it was mandatory.
  
   Warm Regards,
   Tariq
   cloudfront.blogspot.com
  
  
   On Sat, Jun 22, 2013 at 9:45 PM, Kevin O'dell 
 kevin.od...@cloudera.com
   wrote:
  
If you run ZK with a DN/TT/RS please make sure to dedicate a hard
 drive
   and
a core to the ZK process. I have seen many strange occurrences.
On Jun 22, 2013 12:10 PM, Jean-Marc Spaggiari 
  jean-m...@spaggiari.org
   
wrote:
   
 You HAVE TO run a ZK3, or else you don't need to have ZK2 and any
 ZK
 failure will be an issue. You need to have an odd number of ZK
 servers...

 Also, if you don't run MR jobs, you don't need the TT and JT...
 Else,
 everything below is correct. But there is many other options, all
 depend on your needs and the hardware you have ;)

 JM

 2013/6/22 Mohammad Tariq donta...@gmail.com:
  With 8 machines you can do something like this :
 
  Machine 1 - NN+JT
  Machine 2 - SNN+ZK1
  Machine 3 - HM+ZK2
  Machine 4-8 - DN+TT+RS
  (You can run ZK3 on a slave node with some additional memory).
 
  DN and RS run on the same machine. Although RSs are said to hold
  the
 data,
  the data is actually stored in DNs. Replication is managed at
 HDFS
level.
  You don't have to worry about that.
 
  You can visit this link 
http://hbase.apache.org/book/perf.writing.html
 to
  see how to write efficiently into HBase. With a small field there
should
  not be any problem except storage and increased metadata, as
 you'll
have
  many small cells. If possible club several small fields into one
  and
put
  them together in one cell.
 
  HTH
 
  Warm Regards,
  Tariq
  cloudfront.blogspot.com
 
 
  On Sat, Jun 22, 2013 at 8:31 PM, myhbase myhb...@126.com
 wrote:
 
  Thanks for your response.
 
  Now if 5 servers are enough, how can I install  and configure my
nodes?
 If
  I need 3 replicas in case data loss, I should at least have 3
 datanodes, we
  still have namenode, regionserver and HMaster nodes, zookeeper
   nodes,
 some
  of them must be installed in the same machine. The datanode
 seems
   the
 disk
  IO sensitive node while region server is the mem sensitive, can
 I
 install
  them in the same machine? Any suggestion on the deployment plan?
 
  My business requirement is that the write is much more than
   read(7:3),
 and
  I have another concern that I have a field which will have the
   8~15KB
in
   data size, I am not sure, there will be any problem in hbase
 when
   it
 runs
  compaction and split in regions.
 
   Oh, you already have heavyweight's input :).
 
  Thanks

Re: how many severs in a hbase cluster

[Kevin O'dell]

If you run ZK with a DN/TT/RS please make sure to dedicate a hard drive and
a core to the ZK process. I have seen many strange occurrences.
On Jun 22, 2013 12:10 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:

 You HAVE TO run a ZK3, or else you don't need to have ZK2 and any ZK
 failure will be an issue. You need to have an odd number of ZK
 servers...

 Also, if you don't run MR jobs, you don't need the TT and JT... Else,
 everything below is correct. But there is many other options, all
 depend on your needs and the hardware you have ;)

 JM

 2013/6/22 Mohammad Tariq donta...@gmail.com:
  With 8 machines you can do something like this :
 
  Machine 1 - NN+JT
  Machine 2 - SNN+ZK1
  Machine 3 - HM+ZK2
  Machine 4-8 - DN+TT+RS
  (You can run ZK3 on a slave node with some additional memory).
 
  DN and RS run on the same machine. Although RSs are said to hold the
 data,
  the data is actually stored in DNs. Replication is managed at HDFS level.
  You don't have to worry about that.
 
  You can visit this link http://hbase.apache.org/book/perf.writing.html
 to
  see how to write efficiently into HBase. With a small field there should
  not be any problem except storage and increased metadata, as you'll have
  many small cells. If possible club several small fields into one and put
  them together in one cell.
 
  HTH
 
  Warm Regards,
  Tariq
  cloudfront.blogspot.com
 
 
  On Sat, Jun 22, 2013 at 8:31 PM, myhbase myhb...@126.com wrote:
 
  Thanks for your response.
 
  Now if 5 servers are enough, how can I install  and configure my nodes?
 If
  I need 3 replicas in case data loss, I should at least have 3
 datanodes, we
  still have namenode, regionserver and HMaster nodes, zookeeper nodes,
 some
  of them must be installed in the same machine. The datanode seems the
 disk
  IO sensitive node while region server is the mem sensitive, can I
 install
  them in the same machine? Any suggestion on the deployment plan?
 
  My business requirement is that the write is much more than read(7:3),
 and
  I have another concern that I have a field which will have the 8~15KB in
   data size, I am not sure, there will be any problem in hbase when it
 runs
  compaction and split in regions.
 
   Oh, you already have heavyweight's input :).
 
  Thanks JM.
 
  Warm Regards,
  Tariq
  cloudfront.blogspot.com
 
 
  On Sat, Jun 22, 2013 at 8:05 PM, Mohammad Tariq donta...@gmail.com
  wrote:
 
   Hello there,
 
   IMHO, 5-8 servers are sufficient enough to start with. But
 it's
  all relative to the data you have and the intensity of your
 reads/writes.
  You should have different strategies though, based on whether it's
 'read'
  or 'write'. You actually can't define 'big' in absolute terms. My
 cluster
  might be big for me, but for someone else it might still be not big
  enough
  or for someone it might be very big. Long story short it depends on
 your
  needs. If you are able to achieve your goal with 5-8 RSs, then having
  more
  machines will be a wastage, I think.
 
  But you should always keep in mind that HBase is kinda greedy when it
  comes to memory. For a decent load 4G is sufficient, IMHO. But it
 again
  depends on operations you are gonna perform. If you have large
 clusters
  where you are planning to run MR jobs frequently you are better off
 with
  additional 2G.
 
 
  Warm Regards,
  Tariq
  cloudfront.blogspot.com
 
 
  On Sat, Jun 22, 2013 at 7:51 PM, myhbase myhb...@126.com wrote:
 
   Hello All,
 
  I learn hbase almost from papers and books, according to my
  understanding, HBase is the kind of architecture which is more
 appliable
  to a big cluster. We should have many HDFS nodes, and many
 HBase(region
  server) nodes. If we only have several severs(5-8), it seems hbase is
  not a good choice, please correct me if I am wrong. In addition, how
  many nodes usually we can start to consider the hbase solution and
 how
  about the physic mem size and other hardware resource in each node,
 any
  reference document or cases? Thanks.
 
  --Ning