Uhmm... You're copying data from Table A back to Table A? Ok... you really want to disable your caching altogether and make sure each row as you write it is committed to the table.
Try that... it will hurt your performance, but it may keep you afloat. HTH -Mike You've got a scanner and you're running through your table. You're co > Date: Mon, 9 Jan 2012 16:10:25 -0800 > Subject: copy job for mapreduce failing due to large rows > From: tvi...@readypulse.com > To: user@hbase.apache.org > > hi, > I wrote a mapreduce job to copy rows from my table to the same table since > i want to change my row key schema. but the job is failing consistently at > the same point due to presence of large rows. i don't know how to unblock > myself. > > here is the error stack i see. > > attempt_201112151554_0028_m_000120_2: Remapping > 165845033445190:1313884800:weekly:AudEng > attempt_201112151554_0028_m_000120_2: Remapping > 165845033445190:1313884800:weekly:ContentEng > 12/01/10 00:01:01 INFO mapred.JobClient: Task Id : > attempt_201112151554_0028_m_000121_2, Status : FAILED > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed > 1 action: servers with issues: ip-10-68-145-124.ec2.internal:60020, > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1227) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1241) > at > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:826) > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:682) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:667) > at > org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:127) > at > org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:82) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531) > at > org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) > at > com.akanksh.information.hbasetest.HBaseBackuper$BackuperMapper.map(HBaseBackuper.java:68) > at > com.akanksh.information.hbasetest.HBaseBackuper$BackuperMapper.map(HBaseBackuper.java:34) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > > when i open the region server log, i only see a warning here - > > 2012-01-10 00:00:13,745 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Bl > ock cache LRU eviction started; Attempting to free 59.84 MB of total=508.6 > MB > 2012-01-10 00:00:13,793 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Bl > ock cache LRU eviction completed; freed=59.88 MB, total=449.28 MB, > single=130.23 > MB, multi=352.69 MB, memory=21.27 MB > 2012-01-10 00:00:17,230 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LR > U Stats: total=451.46 MB, free=146.87 MB, max=598.34 MB, blocks=9096, > accesses=1 > 663927726, hits=1565631235, hitRatio=94.09%%, cachingAccesses=1638666127, > cachin > gHits=1563292171, cachingHitsRatio=95.40%%, evictions=83895, > evicted=75364860, e > victedPerRun=898.3236694335938 > 2012-01-10 00:00:52,545 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server > handl > er 9 on 60020, responseTooLarge for: next(-5685114053145855194, 50) from > 10.68.1 > 45.124:44423: Size: 121.7m > 2012-01-10 00:01:06,229 DEBUG > org.apache.hadoop.hbase.io.hfile.LruBlockCache: Bl > ock cache LRU eviction started; Attempting to free 59.89 MB of total=508.64 > MB > > i saw a similar thread in the past where your suggestion was to use bulk > load. but i am essentially going through a schema change and doing > migrations. so how do i go about it. i tried decreasing the scan caching > size from 500 to 50. i do setCacheBlocks(false) in my job. > > http://mail-archives.apache.org/mod_mbox/hbase-user/201112.mbox/%3c79df7533-927b-4d30-9ab3-3fe3a1b8c...@gmail.com%3E > > any suggestions? i need to get unblocked asap since this is affecting my > production. > > thanks > vinod
