hi,
I wrote a mapreduce job to copy rows from my table to the same table since
i want to change my row key schema. but the job is failing consistently at
the same point due to presence of large rows. i don't know how to unblock
myself.
here is the error stack i see.
attempt_201112151554_0028_m_000120_2: Remapping
165845033445190:1313884800:weekly:AudEng
attempt_201112151554_0028_m_000120_2: Remapping
165845033445190:1313884800:weekly:ContentEng
12/01/10 00:01:01 INFO mapred.JobClient: Task Id :
attempt_201112151554_0028_m_000121_2, Status : FAILED
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: servers with issues: ip-10-68-145-124.ec2.internal:60020,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1227)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1241)
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:826)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:682)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:667)
at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:127)
at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:82)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
com.akanksh.information.hbasetest.HBaseBackuper$BackuperMapper.map(HBaseBackuper.java:68)
at
com.akanksh.information.hbasetest.HBaseBackuper$BackuperMapper.map(HBaseBackuper.java:34)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
when i open the region server log, i only see a warning here -
2012-01-10 00:00:13,745 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Bl
ock cache LRU eviction started; Attempting to free 59.84 MB of total=508.6
MB
2012-01-10 00:00:13,793 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Bl
ock cache LRU eviction completed; freed=59.88 MB, total=449.28 MB,
single=130.23
MB, multi=352.69 MB, memory=21.27 MB
2012-01-10 00:00:17,230 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LR
U Stats: total=451.46 MB, free=146.87 MB, max=598.34 MB, blocks=9096,
accesses=1
663927726, hits=1565631235, hitRatio=94.09%%, cachingAccesses=1638666127,
cachin
gHits=1563292171, cachingHitsRatio=95.40%%, evictions=83895,
evicted=75364860, e
victedPerRun=898.3236694335938
2012-01-10 00:00:52,545 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handl
er 9 on 60020, responseTooLarge for: next(-5685114053145855194, 50) from
10.68.1
45.124:44423: Size: 121.7m
2012-01-10 00:01:06,229 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Bl
ock cache LRU eviction started; Attempting to free 59.89 MB of total=508.64
MB
i saw a similar thread in the past where your suggestion was to use bulk
load. but i am essentially going through a schema change and doing
migrations. so how do i go about it. i tried decreasing the scan caching
size from 500 to 50. i do setCacheBlocks(false) in my job.
http://mail-archives.apache.org/mod_mbox/hbase-user/201112.mbox/%[email protected]%3E
any suggestions? i need to get unblocked asap since this is affecting my
production.
thanks
vinod
