Re: Out Of Memory on region servers upon bulk import
Thanks Todd, and Slack for such fast responses, Its very good to know the expected memory consumption per handler. Below is my conf. By the way I'm using version 0.20.5. When I introduced the memstore limits things OOME disappeared, but not sure for how long. I also set the setCacheBlocks(false). hbase.zookeeper.quorum zk1,zk2,zk3 zookeeper.session.timeout 18 hbase.rootdir hdfs://hbaseserver:50001/hbase hbase.cluster.distributed true hbase.regionserver.handler.count 25 hbase.zookeeper.property.maxClientCnxns 30 hbase.regionserver.global.memstore.upperLimit 0.3 hbase.regionserver.global.memstore.lowerLimit 0.25 On Aug 27, 2010, at 12:16 AM, Todd Lipcon wrote: > Hi Martin, > > Can you paste your conf? > > Have you by any chance upped your handler count a lot? Each handler takes up > an amount of RAM equal to the largest Puts you do. With normal write buffer > sizes, you're looking at around 2MB per handler, so while it sounds nice to > bump the handler count up to a really high number, you can get OOMEs like > you're seeing. > > Thanks > -Todd > > On Thu, Aug 26, 2010 at 9:31 AM, Martin Arnandze wrote: > >> I provide the answers below. >> Thanks! >> Martin >> >> On Aug 26, 2010, at 11:45 AM, Stack wrote: >> >>> On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze >> wrote: Hi, I'm doing an experiment on an 8 node cluster, each of which has 6GB of >> RAM allocated to hbase region server. Basically, doing a bulk import >> processing large files, >>> >>> >>> How large? >> >> about 10 million records each a few Kb. >> >>> >>> Unless very large, it should not be OOMEing. >>> >>> but some imports require to do gets and scans as well. In the master >>> UI I see that the heap used gets very close to the 6GB limit, but I >>> know hbase is eager for memory and will use the heap as much as >>> possible.I use block caching. Looking at similar posts I see that >>> modifying the handler count and memory store upper/ower limits may be >>> key to solving this issue. Nevertheless I wanted to ask if there is a >>> way to estimate the extra memory used by hbase that makes it crash and >>> if there are other configuration settings I should be looking into to >>> prevent OOME. The job runs correctly for some time but region servers >>> eventually crash. More information about the cluster: - All nodes have 16GM total memory. - 7 nodes running region server (6GB) + datanodes (1GB) + task trackers >> (1GB Heap). Map reduce jobs running w/ 756MB tops each. >>> >>> Good. How many MR child tasks can run on each node concurrently? >> >> three mappers and two reducers >> >>> - 1 node running hbase master (2GB Heap allocated), namenode (4GB), >> Secondary Namenode (4GB), JobTracker (4GB) and Master (2GB). - 3 of the nodes have zookeeper running with 512MB Heap Many thanks, Martin >>> >>> >>> Can we see the lines before the below is thrown? Also, do a listing >>> (ls -r) on this region in hdfs and lets see if anything pops out about >>> files sizes, etc. You'll need to manually map the below region name >>> to its encoded name to figure the region but the encoded name should >>> be earlier in the log. You'll do something like: >>> >>> bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME >> >> /usr/lib/hadoop-0.20/bin/hadoop fs -lsr /hbase/table_import/1698505444 >> -rw-r--r-- 3 hadoop supergroup 1450 2010-08-25 23:17 >> /hbase/table_import/1698505444/.regioninfo >> drwxr-xr-x - hadoop supergroup 0 2010-08-26 11:07 >> /hbase/table_importl/1698505444/fam >> -rw-r--r-- 3 hadoop supergroup4244491 2010-08-26 11:07 >> /hbase/table_import/1698505444/fam/5785049964186428982 >> -rw-r--r-- 3 hadoop supergroup 180147216 2010-08-26 06:09 >> /hbase/table_import/1698505444/fam/705757673046090229 >> >> >> Previous log: >> >> 010-08-26 07:18:14,691 INFO org.apache.hadoop.hbase.regionserver.HRegion: >> region >> table_import,f1cbb42c-b6ae-404d-800c-043da5409441-9223370754623831807WkmpwnRDmveKYzWEfw/tb4GpP9yHDl+/G7OCaZWEgrmGcW+XEF131YDTQwDqZsO93tDicdPcOdRq\x0AU7zDBqoxpA==,1282790086498/1451783432 >> available; sequence id is 1518500874 >> 2010-08-26 07:18:14,691 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: >> table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399 >> 2010-08-26 07:18:14,691 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: >> Creating region >> table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399, >> encoded=1510556231 >> 2010-08-26 07:18:21,085 DEBUG >> org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: >> Total=958.38617MB (1004940736), Free=238.2MB (249864000), Max=11
Re: Out Of Memory on region servers upon bulk import
On Thu, Aug 26, 2010 at 9:31 AM, Martin Arnandze wrote: > I provide the answers below. > Thanks! > Martin > > On Aug 26, 2010, at 11:45 AM, Stack wrote: > >> On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze wrote: >>> Hi, >>> I'm doing an experiment on an 8 node cluster, each of which has 6GB of RAM >>> allocated to hbase region server. Basically, doing a bulk import processing >>> large files, >> >> >> How large? > > about 10 million records each a few Kb. > So the files are not large? They are but a few kb each? St.Ack
Re: Out Of Memory on region servers upon bulk import
Hi Martin, Can you paste your conf? Have you by any chance upped your handler count a lot? Each handler takes up an amount of RAM equal to the largest Puts you do. With normal write buffer sizes, you're looking at around 2MB per handler, so while it sounds nice to bump the handler count up to a really high number, you can get OOMEs like you're seeing. Thanks -Todd On Thu, Aug 26, 2010 at 9:31 AM, Martin Arnandze wrote: > I provide the answers below. > Thanks! > Martin > > On Aug 26, 2010, at 11:45 AM, Stack wrote: > > > On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze > wrote: > >> Hi, > >> I'm doing an experiment on an 8 node cluster, each of which has 6GB of > RAM allocated to hbase region server. Basically, doing a bulk import > processing large files, > > > > > > How large? > > about 10 million records each a few Kb. > > > > > Unless very large, it should not be OOMEing. > > > > but some imports require to do gets and scans as well. In the master > > UI I see that the heap used gets very close to the 6GB limit, but I > > know hbase is eager for memory and will use the heap as much as > > possible.I use block caching. Looking at similar posts I see that > > modifying the handler count and memory store upper/ower limits may be > > key to solving this issue. Nevertheless I wanted to ask if there is a > > way to estimate the extra memory used by hbase that makes it crash and > > if there are other configuration settings I should be looking into to > > prevent OOME. The job runs correctly for some time but region servers > > eventually crash. > >> > >> More information about the cluster: > >> > >> - All nodes have 16GM total memory. > >> - 7 nodes running region server (6GB) + datanodes (1GB) + task trackers > (1GB Heap). Map reduce jobs running w/ 756MB tops each. > > > > Good. How many MR child tasks can run on each node concurrently? > > three mappers and two reducers > > > > >> - 1 node running hbase master (2GB Heap allocated), namenode (4GB), > Secondary Namenode (4GB), JobTracker (4GB) and Master (2GB). > >> - 3 of the nodes have zookeeper running with 512MB Heap > >> > >> Many thanks, > >> Martin > >> > > > > > > Can we see the lines before the below is thrown? Also, do a listing > > (ls -r) on this region in hdfs and lets see if anything pops out about > > files sizes, etc. You'll need to manually map the below region name > > to its encoded name to figure the region but the encoded name should > > be earlier in the log. You'll do something like: > > > > bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME > > /usr/lib/hadoop-0.20/bin/hadoop fs -lsr /hbase/table_import/1698505444 > -rw-r--r-- 3 hadoop supergroup 1450 2010-08-25 23:17 > /hbase/table_import/1698505444/.regioninfo > drwxr-xr-x - hadoop supergroup 0 2010-08-26 11:07 > /hbase/table_importl/1698505444/fam > -rw-r--r-- 3 hadoop supergroup4244491 2010-08-26 11:07 > /hbase/table_import/1698505444/fam/5785049964186428982 > -rw-r--r-- 3 hadoop supergroup 180147216 2010-08-26 06:09 > /hbase/table_import/1698505444/fam/705757673046090229 > > > Previous log: > > 010-08-26 07:18:14,691 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region > table_import,f1cbb42c-b6ae-404d-800c-043da5409441-9223370754623831807WkmpwnRDmveKYzWEfw/tb4GpP9yHDl+/G7OCaZWEgrmGcW+XEF131YDTQwDqZsO93tDicdPcOdRq\x0AU7zDBqoxpA==,1282790086498/1451783432 > available; sequence id is 1518500874 > 2010-08-26 07:18:14,691 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: > table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399 > 2010-08-26 07:18:14,691 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: > Creating region > table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399, > encoded=1510556231 > 2010-08-26 07:18:21,085 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: > Total=958.38617MB (1004940736), Free=238.2MB (249864000), Max=1196.675MB > (1254804736), Counts: Blocks=115717, Access=51364517, Hit=231796, > Miss=51132721, Evictions=15, Evicted=218920, Ratios: Hit > Ratio=0.45127649791538715%, Miss Ratio=99.54872131347656%, > Evicted/Run=14594.6669921875 > 2010-08-26 07:18:27,659 DEBUG org.apache.hadoop.hbase.regionserver.Store: > loaded /hbase/table_import/1510556231/fam/2639910770219077750, > isReference=false, sequence id=1518500860, length=200014693, > majorCompaction=false > 2010-08-26 07:18:35,188 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region > table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399/1510556231 > available; sequence id is 1518500861 > 2010-08-26 07:18:35,188 INFO > org.apache.h
Re: Out Of Memory on region servers upon bulk import
I provide the answers below. Thanks! Martin On Aug 26, 2010, at 11:45 AM, Stack wrote: > On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze wrote: >> Hi, >> I'm doing an experiment on an 8 node cluster, each of which has 6GB of RAM >> allocated to hbase region server. Basically, doing a bulk import processing >> large files, > > > How large? about 10 million records each a few Kb. > > Unless very large, it should not be OOMEing. > > but some imports require to do gets and scans as well. In the master > UI I see that the heap used gets very close to the 6GB limit, but I > know hbase is eager for memory and will use the heap as much as > possible.I use block caching. Looking at similar posts I see that > modifying the handler count and memory store upper/ower limits may be > key to solving this issue. Nevertheless I wanted to ask if there is a > way to estimate the extra memory used by hbase that makes it crash and > if there are other configuration settings I should be looking into to > prevent OOME. The job runs correctly for some time but region servers > eventually crash. >> >> More information about the cluster: >> >> - All nodes have 16GM total memory. >> - 7 nodes running region server (6GB) + datanodes (1GB) + task trackers >> (1GB Heap). Map reduce jobs running w/ 756MB tops each. > > Good. How many MR child tasks can run on each node concurrently? three mappers and two reducers > >> - 1 node running hbase master (2GB Heap allocated), namenode (4GB), >> Secondary Namenode (4GB), JobTracker (4GB) and Master (2GB). >> - 3 of the nodes have zookeeper running with 512MB Heap >> >> Many thanks, >> Martin >> > > > Can we see the lines before the below is thrown? Also, do a listing > (ls -r) on this region in hdfs and lets see if anything pops out about > files sizes, etc. You'll need to manually map the below region name > to its encoded name to figure the region but the encoded name should > be earlier in the log. You'll do something like: > > bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME /usr/lib/hadoop-0.20/bin/hadoop fs -lsr /hbase/table_import/1698505444 -rw-r--r-- 3 hadoop supergroup 1450 2010-08-25 23:17 /hbase/table_import/1698505444/.regioninfo drwxr-xr-x - hadoop supergroup 0 2010-08-26 11:07 /hbase/table_importl/1698505444/fam -rw-r--r-- 3 hadoop supergroup4244491 2010-08-26 11:07 /hbase/table_import/1698505444/fam/5785049964186428982 -rw-r--r-- 3 hadoop supergroup 180147216 2010-08-26 06:09 /hbase/table_import/1698505444/fam/705757673046090229 Previous log: 010-08-26 07:18:14,691 INFO org.apache.hadoop.hbase.regionserver.HRegion: region table_import,f1cbb42c-b6ae-404d-800c-043da5409441-9223370754623831807WkmpwnRDmveKYzWEfw/tb4GpP9yHDl+/G7OCaZWEgrmGcW+XEF131YDTQwDqZsO93tDicdPcOdRq\x0AU7zDBqoxpA==,1282790086498/1451783432 available; sequence id is 1518500874 2010-08-26 07:18:14,691 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399 2010-08-26 07:18:14,691 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Creating region table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399, encoded=1510556231 2010-08-26 07:18:21,085 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats: Sizes: Total=958.38617MB (1004940736), Free=238.2MB (249864000), Max=1196.675MB (1254804736), Counts: Blocks=115717, Access=51364517, Hit=231796, Miss=51132721, Evictions=15, Evicted=218920, Ratios: Hit Ratio=0.45127649791538715%, Miss Ratio=99.54872131347656%, Evicted/Run=14594.6669921875 2010-08-26 07:18:27,659 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded /hbase/table_import/1510556231/fam/2639910770219077750, isReference=false, sequence id=1518500860, length=200014693, majorCompaction=false 2010-08-26 07:18:35,188 INFO org.apache.hadoop.hbase.regionserver.HRegion: region table_import,d1e50232ac85a0a965e48647de5dc6ce-92233707546248658079F073MJ/gGEEs6mwkLsY/lLH+QvHGVBhBavAz0HSPEEKY+NrjTTzHUJdPtuJ0lXqz2i2Qs2DmFkz\x0A5P2broA7Gg==,128278399/1510556231 available; sequence id is 1518500861 2010-08-26 07:18:35,188 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: table_import,8ded1642-1c52-444a-bfdc-43521b220714-9223370754627999807UbbWwFDcGatAe8OniLMUXoaVeEdOvSkqiwXfJgUxNlt0aosKXsWevrlra8QDbEvTZelj/jLyux8y\x0AcCBiLeHbqg==,1282792675254 2010-08-26 07:18:35,189 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Creating region table_import,8ded1642-1c52-444a-bfdc-43521b220714-9223370754627999807UbbWwFDcGatAe8OniLMUXoaVeEdOvSkqiwXfJgUxNlt0aosKXsWevrlra8QDbEvTZelj/jLyux8y\x0AcCBiLeHbqg==,1282792675254, encoded=1698505444 2010-08-26 07:19:14,859 ERROR o
Re: Out Of Memory on region servers upon bulk import
On Thu, Aug 26, 2010 at 8:07 AM, Martin Arnandze wrote: > Hi, > I'm doing an experiment on an 8 node cluster, each of which has 6GB of RAM > allocated to hbase region server. Basically, doing a bulk import processing > large files, How large? Unless very large, it should not be OOMEing. but some imports require to do gets and scans as well. In the master UI I see that the heap used gets very close to the 6GB limit, but I know hbase is eager for memory and will use the heap as much as possible.I use block caching. Looking at similar posts I see that modifying the handler count and memory store upper/ower limits may be key to solving this issue. Nevertheless I wanted to ask if there is a way to estimate the extra memory used by hbase that makes it crash and if there are other configuration settings I should be looking into to prevent OOME. The job runs correctly for some time but region servers eventually crash. > > More information about the cluster: > > - All nodes have 16GM total memory. > - 7 nodes running region server (6GB) + datanodes (1GB) + task trackers (1GB > Heap). Map reduce jobs running w/ 756MB tops each. Good. How many MR child tasks can run on each node concurrently? > - 1 node running hbase master (2GB Heap allocated), namenode (4GB), Secondary > Namenode (4GB), JobTracker (4GB) and Master (2GB). > - 3 of the nodes have zookeeper running with 512MB Heap > > Many thanks, > Martin > Can we see the lines before the below is thrown? Also, do a listing (ls -r) on this region in hdfs and lets see if anything pops out about files sizes, etc. You'll need to manually map the below region name to its encoded name to figure the region but the encoded name should be earlier in the log. You'll do something like: bin/hbase fs -lsr /hbase/table_import/REGION_ENCODED_NAME Thanks, St.Ack > > > 2010-08-26 07:19:14,859 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening > table_import,8ded1642-1c52-444a-bfdc-43521b220714-9223370754627999807UbbWwFDcGatAe8OniLMUXoaVeEdOvSkqiwXfJgUxNlt0aosKXsWevrlra8QDbEvTZelj/jLyux8y\x0AcCBiLeHbqg==,1282792675254 > java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.hbase.io.hfile.HFile$BlockIndex.readIndex(HFile.java:1538) > at > org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:806) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273) > at > org.apache.hadoop.hbase.regionserver.StoreFile.(StoreFile.java:129) > at > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410) > at org.apache.hadoop.hbase.regionserver.Store.(Store.java:221) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1571) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1538) > at > org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1458) > at java.lang.Thread.run(Thread.java:619) >