Hi,苏铖. U may need presplit you htable when the load is heavy or there should be some problem in your client code. Just a suggestion.
bruce 2012/10/29 苏铖 <[email protected]> > Hi, everyone. > > I changed the max size of hbase store file and increased region servers. > The > former exception doesn't happen again. > But there is another exception at the client side. > > 2012-10-29 19:06:27:758 WARN [pool-2-thread-2] > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation > | Failed all from > > region=statistic_visit_detail1,,1351508069797.3272dd30817191d9d393d1d6e1b99d > 1b., hostname=hadoop02, port=60020 > java.util.concurrent.ExecutionException: java.lang.RuntimeException: > java.lang.NullPointerException > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > processBatchCallback(HConnectionManager.java:1557) > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > processBatch(HConnectionManager.java:1409) > at > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900) > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:773) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760) > at > com.lietou.datawarehouse.imp.HBaseImporter$ActualHBaseImporter$1. > process(HBaseImporter.java:150) > at > com.lietou.datawarehouse.imp.HBaseImporter$ActualHBaseImporter$1. > process(HBaseImporter.java:133) > at > > com.lietou.datawarehouse.common.range.Repeater.rangeRepeat(Repeater.java:48) > at > > com.lietou.datawarehouse.common.range.Repeater.rangeRepeat(Repeater.java:30) > at > > com.lietou.datawarehouse.imp.HBaseImporter$ActualHBaseImporter.run(HBaseImpo > rter.java:162) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja > va:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 > 08) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.RuntimeException: java.lang.NullPointerException > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > getRegionServerWithoutRetries(HConnectionManager.java:1371) > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$ > 3.call(HConnectionManager.java:1383) > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$ > 3.call(HConnectionManager.java:1381) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > Caused by: java.lang.NullPointerException > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi > ne.java:158) > at $Proxy10.multi(Unknown Source) > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$ > 3$1.call(HConnectionManager.java:1386) > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$ > 3$1.call(HConnectionManager.java:1384) > at > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > getRegionServerWithoutRetries(HConnectionManager.java:1365) > ... 7 more > > > This error happens quite offen. And on the server side, here are some > warnings > > 2012-10-29 19:50:39,748 WARN org.apache.hadoop.ipc.HBaseServer: > (responseTooSlow): > > {"processingtimems":17476,"call":"multi(org.apache.hadoop.hbase.client.Multi > Action@62aaeb8e), rpc version=1, client version=29, > methodsFingerPrint=54742778","client":"192.168.1.70:3237 > ","starttimems":1351 > > 511422270,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method": > "multi"} > > > I have only 5 threads to execute put actions at the same time. > The machine load is not very high. > > top - 19:55:02 up 7:56, 4 users, load average: 1.62, 1.36, 1.11 > > Anyone met this error before? Please help me. > > Thanks. > > > -----邮件原件----- > 发件人: 苏铖 [mailto:[email protected]] > 发送时间: 2012年10月29日 15:53 > 收件人: [email protected] > 主题: 答复: How to adjust hbase settings when too many store files? > > I checked the region server log again, and I found something below: > > 2012-10-28 06:24:24,811 ERROR > org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: > Compaction failed regionName=stati > > stic_visit_detail,20120922|13984|451728,1351376659451.9b2bfae5d77109693a153e > b16fcb7793., storeName=cf1, fileCount=7, fileSize=1.1g ( > 681.7m, 168.6m, 139.9m, 36.2m, 53.0m, 26.5m, 5.9m), priority=0, > time=469259302083252 > java.io.IOException: java.io.IOException: File > > /hbase/statistic_visit_detail/9b2bfae5d77109693a153eb16fcb7793/.tmp/3a3e6ee8 > 8a524659b > 9f9716e5ca21a74 could only be replicated to 0 nodes, instead of 1 > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNam > esystem.java:1531) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:685) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384) > > It seems that, the fileSize already exceed the max size, which is 1G as > default. > > So I add more region servers and enlarge the max file size, to see if it > works. > > Thanks a lot. > > -----邮件原件----- > 发件人: Ramkrishna.S.Vasudevan [mailto:[email protected]] > 发送时间: 2012年10月29日 14:06 > 收件人: [email protected] > 主题: RE: How to adjust hbase settings when too many store files? > > Also check what is your heap size of RS? > When you say hTable.put(), how many such threads are there? > > What is your region size? Is your regions splitting continuously do to > heavy load? > > Regards > Ram > > > > -----Original Message----- > > From: [email protected] [mailto:[email protected]] > > Sent: Monday, October 29, 2012 10:01 AM > > To: [email protected] > > Cc: <[email protected]> > > Subject: Re: How to adjust hbase settings when too many store files? > > > > What version of hbase were you using ? > > Did you pre split the table before loading ? > > > > Thanks > > > > > > > > On Oct 28, 2012, at 8:33 PM, 苏铖 <[email protected]> wrote: > > > > > Hello. I encounter a region server error when I try to put bulk data > > from a > > > java client. > > > > > > The java client extracts data from a relational database and puts > > those data > > > into hbase. > > > > > > When I try to extract data from a large table(say, 1 billion > > records), the > > > error happens. > > > > > > > > > > > > The region server's log says: > > > > > > > > > > > >> 2012-10-28 00:00:02,169 WARN > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region > > > statistic_visit_detail,20120804|72495|8549 > > > > > > 56,1351353594195.ad2592ee7a3610c60c47cf8be77496c8. has too many store > > files; > > > delaying flush up to 90000ms > > > > > >> 2012-10-28 00:00:02,791 DEBUG > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread > > woke up > > > because memory above low wa > > > > > > ter=347.1m > > > > > >> 2012-10-28 00:00:02,791 DEBUG > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Under global > > heap > > > pressure: Region statistic_vis > > > > > > > > it_detail,20120804|72495|854956,1351353594195.ad2592ee7a3610c60c47cf8be > > 77496 > > > c8. has too many store files, but is 141.5m vs best flus > > > > > > hable region's 46.8m. Choosing the bigger. > > > > > >> 2012-10-28 00:00:02,791 INFO > > > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region > > > statistic_visit_detail,20120804|7 > > > > > > 2495|854956,1351353594195.ad2592ee7a3610c60c47cf8be77496c8. due to > > global > > > heap pressure > > > > > > ... > > > > > > > > > > > > And finally, > > > > > > > > > > > >> 2012-10-28 00:00:43,511 INFO > > org.apache.hadoop.hbase.regionserver.HRegion: > > > compaction interrupted by user > > > > > >> java.io.InterruptedIOException: Aborting compaction of store cf1 in > > region > > > statistic_visit_detail,20120804|72495|854956,135135359419 > > > > > > 5.ad2592ee7a3610c60c47cf8be77496c8. because user requested stop. > > > > > > at > > > > > org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1275 > > ) > > > > > > at > > > org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:765) > > > > > > at > > > > > org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1023) > > > > > > at > > > > > org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run( > > Compa > > > ctionRequest.java:177) > > > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecut > > or.ja > > > va:886) > > > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j > > ava:9 > > > 08) > > > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > > > > > > > Then the region server shuts down. > > > > > > > > > > > > It seems that too many store files(due to too many records from > > > relational-db) consumed too many memories, if I'm right. > > > > > > I'm new to hbase, what settings should I adjust? Or even increase > > region > > > servers? > > > > > > I'm going to do some research by myself, and any advise will be > > appreciated. > > > > > > Best regards, > > > > > > > > > > > > Su > > > > > > > > > > > > >
