Re: Out Of Memory on region servers upon bulk import

2010-08-27 Thread Martin Arnandze
Thanks Todd, and Slack for such fast responses,

Its very good to know the expected memory consumption per handler.

Below is my conf. By the way I'm using version 0.20.5. When I introduced the 
memstore limits things OOME disappeared, but not sure for how long. I also set 
the setCacheBlocks(false). 


hbase.zookeeper.quorum
zk1,zk2,zk3



zookeeper.session.timeout
18



hbase.rootdir
hdfs://hbaseserver:50001/hbase



hbase.cluster.distributed
true



   hbase.regionserver.handler.count
   25
 


  hbase.zookeeper.property.maxClientCnxns
  30



hbase.regionserver.global.memstore.upperLimit
0.3



hbase.regionserver.global.memstore.lowerLimit
0.25


 

On Aug 27, 2010, at 12:16 AM, Todd Lipcon wrote:

> Hi Martin,
> 
> Can you paste your conf?
> 
> Have you by any chance upped your handler count a lot? Each handler takes up
> an amount of RAM equal to the largest Puts you do. With normal write buffer
> sizes, you're looking at around 2MB per handler, so while it sounds nice to
> bump the handler count up to a really high number, you can get OOMEs like
> you're seeing.
> 
> Thanks
> -Todd
> 
> On Thu, Aug 26, 2010 at 9:31 AM, Martin Arnandze wrote:
> 
>> I provide the answers below.
>> Thanks!
>> Martin
>> 
>> On Aug 26, 2010, at 11:45 AM, Stack wrote:
>> 
>>> On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze 
>> wrote:
 Hi,
 I'm doing an experiment on an 8 node cluster, each of which has 6GB of
>> RAM allocated to hbase region server. Basically, doing a bulk import
>> processing large files,
>>> 
>>> 
>>> How large?
>> 
>> about 10 million records each a few Kb.
>> 
>>> 
>>> Unless very large, it should not be OOMEing.
>>> 
>>> but some imports require to do gets and scans as well. In the master
>>> UI I see that the heap used gets very close to the 6GB limit, but I
>>> know hbase is eager for memory and will use the heap as much as
>>> possible.I use block caching. Looking at similar posts I see that
>>> modifying the handler count and memory store upper/ower limits may be
>>> key to solving this issue. Nevertheless I wanted to ask if there is a
>>> way to estimate the extra memory used by hbase that makes it crash and
>>> if there are other configuration settings I should be looking into to
>>> prevent OOME. The job runs correctly for some time but region servers
>>> eventually crash.
 
 More information about the cluster:
 
 - All nodes have 16GM total memory.
 - 7 nodes running region server (6GB) +  datanodes (1GB) + task trackers
>> (1GB Heap).  Map reduce jobs running w/ 756MB tops each.
>>> 
>>> Good.  How many MR child tasks can run on each node concurrently?
>> 
>> three mappers and two reducers
>> 
>>> 
 - 1 node running hbase master (2GB Heap allocated), namenode (4GB),
>> Secondary Namenode (4GB), JobTracker (4GB) and Master (2GB).
 - 3 of the nodes have zookeeper running with 512MB Heap
 
 Many thanks,
  Martin
 
>>> 
>>> 
>>> Can we see the lines before the below is thrown?   Also, do a listing
>>> (ls -r) on this region in hdfs and lets see if anything pops out about
>>> files sizes, etc.  You'll need to manually map the below region name
>>> to its encoded name to figure the region but the encoded name should
>>> be earlier in the log.  You'll do something like:
>>> 
>>> bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME
>> 
>> /usr/lib/hadoop-0.20/bin/hadoop fs -lsr /hbase/table_import/1698505444
>> -rw-r--r--   3 hadoop supergroup   1450 2010-08-25 23:17
>> /hbase/table_import/1698505444/.regioninfo
>> drwxr-xr-x   - hadoop supergroup  0 2010-08-26 11:07
>> /hbase/table_importl/1698505444/fam
>> -rw-r--r--   3 hadoop supergroup4244491 2010-08-26 11:07
>> /hbase/table_import/1698505444/fam/5785049964186428982
>> -rw-r--r--   3 hadoop supergroup  180147216 2010-08-26 06:09
>> /hbase/table_import/1698505444/fam/705757673046090229
>> 
>> 
>> Previous log:
>> 
>> 010-08-26 07:18:14,691 INFO org.apache.hadoop.hbase.regionserver.HRegion:
>> region
>> table_import,f1cbb42c-b6ae-404d-800c-043da5409441-9223370754623831807WkmpwnRDmveKYzWEfw/tb4GpP9yHDl+/G7OCaZWEgrmGcW+XEF131YDTQwDqZsO93tDicdPcOdRq\x0AU7zDBqoxpA==,1282790086498/1451783432
>> available; sequence id is 1518500874
>> 2010-08-26 07:18:14,691 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
>> table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399
>> 2010-08-26 07:18:14,691 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
>> Creating region
>> table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399,
>> encoded=1510556231
>> 2010-08-26 07:18:21,085 DEBUG
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
>> Total=958.38617MB (1004940736), Free=238.2MB (249864000), Max=11

Re: Out Of Memory on region servers upon bulk import

2010-08-26 Thread Stack
On Thu, Aug 26, 2010 at 9:31 AM, Martin Arnandze  wrote:
> I provide the answers below.
> Thanks!
>  Martin
>
> On Aug 26, 2010, at 11:45 AM, Stack wrote:
>
>> On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze  wrote:
>>> Hi,
>>>  I'm doing an experiment on an 8 node cluster, each of which has 6GB of RAM 
>>> allocated to hbase region server. Basically, doing a bulk import processing 
>>> large files,
>>
>>
>> How large?
>
> about 10 million records each a few Kb.
>

So the files are not large?  They are but a few kb each?

St.Ack


Re: Out Of Memory on region servers upon bulk import

2010-08-26 Thread Todd Lipcon
Hi Martin,

Can you paste your conf?

Have you by any chance upped your handler count a lot? Each handler takes up
an amount of RAM equal to the largest Puts you do. With normal write buffer
sizes, you're looking at around 2MB per handler, so while it sounds nice to
bump the handler count up to a really high number, you can get OOMEs like
you're seeing.

Thanks
-Todd

On Thu, Aug 26, 2010 at 9:31 AM, Martin Arnandze wrote:

> I provide the answers below.
> Thanks!
>  Martin
>
> On Aug 26, 2010, at 11:45 AM, Stack wrote:
>
> > On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze 
> wrote:
> >> Hi,
> >>  I'm doing an experiment on an 8 node cluster, each of which has 6GB of
> RAM allocated to hbase region server. Basically, doing a bulk import
> processing large files,
> >
> >
> > How large?
>
> about 10 million records each a few Kb.
>
> >
> > Unless very large, it should not be OOMEing.
> >
> > but some imports require to do gets and scans as well. In the master
> > UI I see that the heap used gets very close to the 6GB limit, but I
> > know hbase is eager for memory and will use the heap as much as
> > possible.I use block caching. Looking at similar posts I see that
> > modifying the handler count and memory store upper/ower limits may be
> > key to solving this issue. Nevertheless I wanted to ask if there is a
> > way to estimate the extra memory used by hbase that makes it crash and
> > if there are other configuration settings I should be looking into to
> > prevent OOME. The job runs correctly for some time but region servers
> > eventually crash.
> >>
> >> More information about the cluster:
> >>
> >> - All nodes have 16GM total memory.
> >> - 7 nodes running region server (6GB) +  datanodes (1GB) + task trackers
> (1GB Heap).  Map reduce jobs running w/ 756MB tops each.
> >
> > Good.  How many MR child tasks can run on each node concurrently?
>
> three mappers and two reducers
>
> >
> >> - 1 node running hbase master (2GB Heap allocated), namenode (4GB),
> Secondary Namenode (4GB), JobTracker (4GB) and Master (2GB).
> >> - 3 of the nodes have zookeeper running with 512MB Heap
> >>
> >> Many thanks,
> >>   Martin
> >>
> >
> >
> > Can we see the lines before the below is thrown?   Also, do a listing
> > (ls -r) on this region in hdfs and lets see if anything pops out about
> > files sizes, etc.  You'll need to manually map the below region name
> > to its encoded name to figure the region but the encoded name should
> > be earlier in the log.  You'll do something like:
> >
> > bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME
>
> /usr/lib/hadoop-0.20/bin/hadoop fs -lsr /hbase/table_import/1698505444
> -rw-r--r--   3 hadoop supergroup   1450 2010-08-25 23:17
> /hbase/table_import/1698505444/.regioninfo
> drwxr-xr-x   - hadoop supergroup  0 2010-08-26 11:07
> /hbase/table_importl/1698505444/fam
> -rw-r--r--   3 hadoop supergroup4244491 2010-08-26 11:07
> /hbase/table_import/1698505444/fam/5785049964186428982
> -rw-r--r--   3 hadoop supergroup  180147216 2010-08-26 06:09
> /hbase/table_import/1698505444/fam/705757673046090229
>
>
> Previous log:
>
> 010-08-26 07:18:14,691 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region
> table_import,f1cbb42c-b6ae-404d-800c-043da5409441-9223370754623831807WkmpwnRDmveKYzWEfw/tb4GpP9yHDl+/G7OCaZWEgrmGcW+XEF131YDTQwDqZsO93tDicdPcOdRq\x0AU7zDBqoxpA==,1282790086498/1451783432
> available; sequence id is 1518500874
> 2010-08-26 07:18:14,691 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN:
> table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399
> 2010-08-26 07:18:14,691 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Creating region
> table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399,
> encoded=1510556231
> 2010-08-26 07:18:21,085 DEBUG
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes:
> Total=958.38617MB (1004940736), Free=238.2MB (249864000), Max=1196.675MB
> (1254804736), Counts: Blocks=115717, Access=51364517, Hit=231796,
> Miss=51132721, Evictions=15, Evicted=218920, Ratios: Hit
> Ratio=0.45127649791538715%, Miss Ratio=99.54872131347656%,
> Evicted/Run=14594.6669921875
> 2010-08-26 07:18:27,659 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> loaded /hbase/table_import/1510556231/fam/2639910770219077750,
> isReference=false, sequence id=1518500860, length=200014693,
> majorCompaction=false
> 2010-08-26 07:18:35,188 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region
> table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399/1510556231
> available; sequence id is 1518500861
> 2010-08-26 07:18:35,188 INFO
> org.apache.h

Re: Out Of Memory on region servers upon bulk import

2010-08-26 Thread Martin Arnandze
I provide the answers below.
Thanks!
  Martin

On Aug 26, 2010, at 11:45 AM, Stack wrote:

> On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze  wrote:
>> Hi,
>>  I'm doing an experiment on an 8 node cluster, each of which has 6GB of RAM 
>> allocated to hbase region server. Basically, doing a bulk import processing 
>> large files,
> 
> 
> How large?

about 10 million records each a few Kb.

> 
> Unless very large, it should not be OOMEing.
> 
> but some imports require to do gets and scans as well. In the master
> UI I see that the heap used gets very close to the 6GB limit, but I
> know hbase is eager for memory and will use the heap as much as
> possible.I use block caching. Looking at similar posts I see that
> modifying the handler count and memory store upper/ower limits may be
> key to solving this issue. Nevertheless I wanted to ask if there is a
> way to estimate the extra memory used by hbase that makes it crash and
> if there are other configuration settings I should be looking into to
> prevent OOME. The job runs correctly for some time but region servers
> eventually crash.
>> 
>> More information about the cluster:
>> 
>> - All nodes have 16GM total memory.
>> - 7 nodes running region server (6GB) +  datanodes (1GB) + task trackers 
>> (1GB Heap).  Map reduce jobs running w/ 756MB tops each.
> 
> Good.  How many MR child tasks can run on each node concurrently? 

three mappers and two reducers

> 
>> - 1 node running hbase master (2GB Heap allocated), namenode (4GB), 
>> Secondary Namenode (4GB), JobTracker (4GB) and Master (2GB).
>> - 3 of the nodes have zookeeper running with 512MB Heap
>> 
>> Many thanks,
>>   Martin
>> 
> 
> 
> Can we see the lines before the below is thrown?   Also, do a listing
> (ls -r) on this region in hdfs and lets see if anything pops out about
> files sizes, etc.  You'll need to manually map the below region name
> to its encoded name to figure the region but the encoded name should
> be earlier in the log.  You'll do something like:
> 
> bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME

/usr/lib/hadoop-0.20/bin/hadoop fs -lsr /hbase/table_import/1698505444
-rw-r--r--   3 hadoop supergroup   1450 2010-08-25 23:17 
/hbase/table_import/1698505444/.regioninfo
drwxr-xr-x   - hadoop supergroup  0 2010-08-26 11:07 
/hbase/table_importl/1698505444/fam
-rw-r--r--   3 hadoop supergroup4244491 2010-08-26 11:07 
/hbase/table_import/1698505444/fam/5785049964186428982
-rw-r--r--   3 hadoop supergroup  180147216 2010-08-26 06:09 
/hbase/table_import/1698505444/fam/705757673046090229


Previous log:

010-08-26 07:18:14,691 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
region 
table_import,f1cbb42c-b6ae-404d-800c-043da5409441-9223370754623831807WkmpwnRDmveKYzWEfw/tb4GpP9yHDl+/G7OCaZWEgrmGcW+XEF131YDTQwDqZsO93tDicdPcOdRq\x0AU7zDBqoxpA==,1282790086498/1451783432
 available; sequence id is 1518500874
2010-08-26 07:18:14,691 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: 
table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399
2010-08-26 07:18:14,691 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Creating region 
table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399,
 encoded=1510556231
2010-08-26 07:18:21,085 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: 
Cache Stats: Sizes: Total=958.38617MB (1004940736), Free=238.2MB 
(249864000), Max=1196.675MB (1254804736), Counts: Blocks=115717, 
Access=51364517, Hit=231796, Miss=51132721, Evictions=15, Evicted=218920, 
Ratios: Hit Ratio=0.45127649791538715%, Miss Ratio=99.54872131347656%, 
Evicted/Run=14594.6669921875
2010-08-26 07:18:27,659 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
loaded /hbase/table_import/1510556231/fam/2639910770219077750, 
isReference=false, sequence id=1518500860, length=200014693, 
majorCompaction=false
2010-08-26 07:18:35,188 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
region 
table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399/1510556231
 available; sequence id is 1518500861
2010-08-26 07:18:35,188 INFO 
org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: 
table_import,8ded1642-1c52-444a-bfdc-43521b220714-9223370754627999807UbbWwFDcGatAe8OniLMUXoaVeEdOvSkqiwXfJgUxNlt0aosKXsWevrlra8QDbEvTZelj/jLyux8y\x0AcCBiLeHbqg==,1282792675254
2010-08-26 07:18:35,189 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Creating region 
table_import,8ded1642-1c52-444a-bfdc-43521b220714-9223370754627999807UbbWwFDcGatAe8OniLMUXoaVeEdOvSkqiwXfJgUxNlt0aosKXsWevrlra8QDbEvTZelj/jLyux8y\x0AcCBiLeHbqg==,1282792675254,
 encoded=1698505444
2010-08-26 07:19:14,859 ERROR 
o

Re: Out Of Memory on region servers upon bulk import

2010-08-26 Thread Stack
On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze  wrote:
> Hi,
>  I'm doing an experiment on an 8 node cluster, each of which has 6GB of RAM 
> allocated to hbase region server. Basically, doing a bulk import processing 
> large files,


How large?

Unless very large, it should not be OOMEing.

but some imports require to do gets and scans as well. In the master
UI I see that the heap used gets very close to the 6GB limit, but I
know hbase is eager for memory and will use the heap as much as
possible.I use block caching. Looking at similar posts I see that
modifying the handler count and memory store upper/ower limits may be
key to solving this issue. Nevertheless I wanted to ask if there is a
way to estimate the extra memory used by hbase that makes it crash and
if there are other configuration settings I should be looking into to
prevent OOME. The job runs correctly for some time but region servers
eventually crash.
>
> More information about the cluster:
>
> - All nodes have 16GM total memory.
> - 7 nodes running region server (6GB) +  datanodes (1GB) + task trackers (1GB 
> Heap).  Map reduce jobs running w/ 756MB tops each.

Good.  How many MR child tasks can run on each node concurrently?

> - 1 node running hbase master (2GB Heap allocated), namenode (4GB), Secondary 
> Namenode (4GB), JobTracker (4GB) and Master (2GB).
> - 3 of the nodes have zookeeper running with 512MB Heap
>
> Many thanks,
>   Martin
>


Can we see the lines before the below is thrown?   Also, do a listing
(ls -r) on this region in hdfs and lets see if anything pops out about
files sizes, etc.  You'll need to manually map the below region name
to its encoded name to figure the region but the encoded name should
be earlier in the log.  You'll do something like:

bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME

Thanks,
St.Ack


>
>
> 2010-08-26 07:19:14,859 ERROR 
> org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening 
> table_import,8ded1642-1c52-444a-bfdc-43521b220714-9223370754627999807UbbWwFDcGatAe8OniLMUXoaVeEdOvSkqiwXfJgUxNlt0aosKXsWevrlra8QDbEvTZelj/jLyux8y\x0AcCBiLeHbqg==,1282792675254
> java.lang.OutOfMemoryError: Java heap space
>       at 
> org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.readIndex(HFile.java:1538)
>       at 
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:806)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273)
>       at 
> org.apache.hadoop.hbase.regionserver.StoreFile.(StoreFile.java:129)
>       at 
> org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410)
>       at org.apache.hadoop.hbase.regionserver.Store.(Store.java:221)
>       at 
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636)
>       at 
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321)
>       at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1571)
>       at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1538)
>       at 
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1458)
>       at java.lang.Thread.run(Thread.java:619)
>