One more thing, the Hbase table in question is neither enabled, nor disabled:
hbase(main):006:0> is_disabled 'userTable1' false 0 row(s) in 0.0040 seconds hbase(main):007:0> is_enabled 'userTable1' false 0 row(s) in 0.0040 seconds Ameya On Thu, Nov 1, 2012 at 12:02 AM, Ameya Kantikar <[email protected]> wrote: > Hi, > > I am trying to load lot of data (around 1.5 TB) into a single Hbase table. > I have setup region size at 2 GB. I also > set hbase.regionserver.handler.count at 30. > > When I start loading data via MR, after a while, tasks start failing with > following error: > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: NotServingRegionException: 1 time, servers with issues: > smartdeals-hbase8-snc1.snc1:60020, > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1641) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:943) > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:820) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:795) > at > com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:83) > at > com..mr.hbase.LoadUserCacheInHbase$TokenizerMapper.map(LoadUserCacheInHbase.java:33) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645) > at org.apache.hadoop.mapred.MapTask.run(MapTask.j > > On the hbase8 machine I see following in logs: > > ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Error while syncing, > requesting close of hlog > java.io.IOException: Reflection > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:230) > at > org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1109) > at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1213) > at > org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1071) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:228) > ... 4 more > > > I only have 15 map tasks each on a 10 machine cluster (total 150 map tasks > entering data into Hbase table). > > Further, I see 2-3 regions perpetually under "Regions in Transitions" in > Hbase master web console as follows: > > 8dcb3edee4e43faa3dbeac2db4f12274userTable1,[email protected],1351728961461.8dcb3edee4e43faa3dbeac2db4f12274. > state=PENDING_OPEN, ts=Thu Nov 01 06:39:57 UTC 2012 (409s ago), > server=smartdeals-hbase1-snc1.snc1,60020,1351751785514 > > > bb91fd0c855e60dd4159e0ad3fd52cdauserTable1,[email protected],1351728968936.bb91fd0c855e60dd4159e0ad3fd52cda. > state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago), > server=smartdeals-hbase3-snc1.snc1,60020,1351747466016 > bd44334a11464baf85013c97d673e600userTable1,[email protected],1351728952308.bd44334a11464baf85013c97d673e600. > state=PENDING_OPEN, ts=Thu Nov 01 06:42:17 UTC 2012 (269s ago), > server=smartdeals-hbase1-snc1.snc1,60020,1351751785514 > ed1f7e7908fc232f10d78dd1e796a5d7userTable1,[email protected],1351728971232.ed1f7e7908fc232f10d78dd1e796a5d7. > state=PENDING_OPEN, ts=Thu Nov 01 06:37:37 UTC 2012 (549s ago), > server=smartdeals-hbase3-snc1.snc1,60020,1351747466016 > > > Note these are not going away even after 30 minutes. > > Further after running > > hbase hbck -summary I get following: > > Summary: > -ROOT- is okay. > Number of regions: 1 > Deployed on: smartdeals-hbase7-snc1.snc1,60020,1351747458782 > .META. is okay. > Number of regions: 1 > Deployed on: smartdeals-hbase7-snc1.snc1,60020,1351747458782 > test1 is okay. > Number of regions: 1 > Deployed on: smartdeals-hbase2-snc1.snc1,60020,1351747457308 > userTable1 is okay. > Number of regions: 32 > Deployed on: smartdeals-hbase10-snc1.snc1,60020,1351747456776 > smartdeals-hbase2-snc1.snc1,60020,1351747457308 > smartdeals-hbase4-snc1.snc1,60020,1351747455571 > smartdeals-hbase5-snc1.snc1,60020,1351747458579 > smartdeals-hbase6-snc1.snc1,60020,1351747458186 > smartdeals-hbase7-snc1.snc1,60020,1351747458782 > smartdeals-hbase8-snc1.snc1,60020,1351747459112 > smartdeals-hbase9-snc1.snc1,60020,1351747455106 > 24 inconsistencies detected. > Status: INCONSISTENT > > In master logs I am seeing following error: > > ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: > smartdeals-hbase3-snc1.snc1,60020,1351747466016 due to > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: > Received:OPEN for the > region:userTable1,[email protected],1351728968936.bb91fd0c855e60dd4159e0ad3fd52cda. > ,which we are already trying to OPEN. > at > org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:2499) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2457) > at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336) > > > Am I missing something? How do I recover from this? How do I load lot of data > via MR into Hbase Tables? > > > I am running under following setup: > > hadoop:2.0.0-cdh4.0.1 > > hbase: 0.92.1-cdh4.0.1, r > > > Would greatly appreciate any help. > > > Ameya > > > >
