Hi Andrew, Really thanks. When I update to revision 732203, the problem is gone.
On Wed, Jan 7, 2009 at 11:18 AM, Andrew Purtell <[email protected]> wrote: > Hi Edward, > > You may want to update again. > > If you are running latest trunk as of the timestamp of this > reply to your email, and you continue to see the REE exceptions, > then the REEs are likely a result of your region server ceasing > to function and/or a failure to reassign a region after a > split. Master log entries for the region(s) in question would > be helpful. > > - Andy > > > --- On Tue, 1/6/09, Edward J. Yoon <[email protected]> wrote: > >> From: Edward J. Yoon <[email protected]> >> Subject: Exceptions at HTable.flushCommits() >> To: [email protected] >> Cc: [email protected] >> Date: Tuesday, January 6, 2009, 5:52 PM >> After hbase update, it shows >> 'RetriesExhaustedException' during write >> processing of record as below. What's wrong? >> >> ---- >> Wrote input for Map #1 >> Wrote input for Map #2 >> 09/01/07 10:39:51 WARN mapred.JobClient: Use >> GenericOptionsParser for >> parsing the arguments. Applications should implement Tool >> for the >> same. >> 09/01/07 10:39:51 WARN mapred.JobClient: Use genericOptions >> for the >> option -libjars >> 09/01/07 10:39:51 WARN mapred.JobClient: No job jar file >> set. User >> classes may not be found. See JobConf(Class) or >> JobConf#setJar(String). >> 09/01/07 10:39:51 INFO mapred.FileInputFormat: Total input >> paths to process : 3 >> 09/01/07 10:39:51 INFO mapred.JobClient: Running job: >> job_200901071022_0001 >> 09/01/07 10:39:52 INFO mapred.JobClient: map 0% reduce 0% >> 09/01/07 10:40:28 INFO mapred.JobClient: map 33% reduce 0% >> 09/01/07 10:40:31 INFO mapred.JobClient: map 100% reduce >> 0% >> 09/01/07 10:40:43 INFO mapred.JobClient: map 100% reduce >> 22% >> 09/01/07 10:40:45 INFO mapred.JobClient: map 100% reduce >> 44% >> 09/01/07 10:40:47 INFO mapred.JobClient: map 100% reduce >> 67% >> 09/01/07 10:40:53 INFO mapred.JobClient: map 100% reduce >> 68% >> 09/01/07 10:40:58 INFO mapred.JobClient: map 100% reduce >> 69% >> 09/01/07 10:41:07 INFO mapred.JobClient: map 100% reduce >> 70% >> 09/01/07 10:41:12 INFO mapred.JobClient: map 100% reduce >> 71% >> 09/01/07 10:41:18 INFO mapred.JobClient: map 100% reduce >> 72% >> 09/01/07 10:41:25 INFO mapred.JobClient: map 100% reduce >> 73% >> 09/01/07 10:41:32 INFO mapred.JobClient: map 100% reduce >> 74% >> 09/01/07 10:42:51 INFO mapred.JobClient: map 100% reduce >> 0% >> 09/01/07 10:42:51 INFO mapred.JobClient: Task Id : >> attempt_200901071022_0001_r_000001_0, Status : FAILED >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Trying to >> contact region server Some server for region >> DenseMatrix_randhknkr,,1231292385276, row >> '000000000001168', but >> failed after 10 attempts. >> Exceptions: >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:943) >> at >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1315) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1295) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51) >> at >> org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:63) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:36) >> at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) >> at >> org.apache.hadoop.mapred.Child.main(Child.java:155) >> >> 09/01/07 10:42:51 INFO mapred.JobClient: Task Id : >> attempt_200901071022_0001_r_000000_0, Status : FAILED >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Trying to >> contact region server Some server for region >> DenseMatrix_randhknkr,,1231292385276, row >> '000000000001053', but >> failed after 10 attempts. >> Exceptions: >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:943) >> at >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1315) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1295) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51) >> at >> org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:63) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:36) >> at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) >> at >> org.apache.hadoop.mapred.Child.main(Child.java:155) >> >> 09/01/07 10:42:51 INFO mapred.JobClient: Task Id : >> attempt_200901071022_0001_r_000002_0, Status : FAILED >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Trying to >> contact region server Some server for region >> DenseMatrix_randhknkr,,1231292385276, row >> '000000000001061', but >> failed after 10 attempts. >> Exceptions: >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:943) >> at >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1315) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1295) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51) >> at >> org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:63) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:36) >> at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) >> at >> org.apache.hadoop.mapred.Child.main(Child.java:155) >> >> 09/01/07 10:43:01 INFO mapred.JobClient: map 100% reduce >> 45% >> 09/01/07 10:43:05 INFO mapred.JobClient: map 100% reduce >> 68% >> 09/01/07 10:43:10 INFO mapred.JobClient: map 100% reduce >> 69% >> 09/01/07 10:43:15 INFO mapred.JobClient: map 100% reduce >> 70% >> 09/01/07 10:43:25 INFO mapred.JobClient: map 100% reduce >> 71% >> 09/01/07 10:43:30 INFO mapred.JobClient: map 100% reduce >> 72% >> 09/01/07 10:43:40 INFO mapred.JobClient: map 100% reduce >> 73% >> 09/01/07 10:43:45 INFO mapred.JobClient: map 100% reduce >> 74% >> 09/01/07 10:43:51 INFO mapred.JobClient: map 100% reduce >> 75% >> 09/01/07 10:45:11 INFO mapred.JobClient: Task Id : >> attempt_200901071022_0001_r_000002_1, Status : FAILED >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Trying to >> contact region server Some server for region >> DenseMatrix_randhknkr,000000000000257,1231292488131, row >> '000000000001172', but failed after 10 attempts. >> Exceptions: >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:943) >> at >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1315) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1295) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51) >> at >> org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:63) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:36) >> at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) >> at >> org.apache.hadoop.mapred.Child.main(Child.java:155) >> >> 09/01/07 10:45:11 INFO mapred.JobClient: Task Id : >> attempt_200901071022_0001_r_000001_1, Status : FAILED >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Trying to >> contact region server Some server for region >> DenseMatrix_randhknkr,000000000000257,1231292488131, row >> '000000000001330', but failed after 10 attempts. >> Exceptions: >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:943) >> at >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1315) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1295) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51) >> at >> org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:63) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:36) >> at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) >> at >> org.apache.hadoop.mapred.Child.main(Child.java:155) >> >> 09/01/07 10:45:11 INFO mapred.JobClient: Task Id : >> attempt_200901071022_0001_r_000000_1, Status : FAILED >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Trying to >> contact region server Some server for region >> DenseMatrix_randhknkr,000000000000257,1231292488131, row >> '000000000001344', but failed after 10 attempts. >> Exceptions: >> >> at >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:943) >> at >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1315) >> at >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1295) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:71) >> at >> org.apache.hama.mapred.VectorOutputFormat$TableRecordWriter.write(VectorOutputFormat.java:51) >> at >> org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:405) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:63) >> at >> org.apache.hama.mapred.RandomMatrixReduce.reduce(RandomMatrixReduce.java:36) >> at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) >> at >> org.apache.hadoop.mapred.Child.main(Child.java:155) >> >> 09/01/07 10:45:21 INFO mapred.JobClient: map 100% reduce >> 45% >> 09/01/07 10:45:26 INFO mapred.JobClient: map 100% reduce >> 67% >> 09/01/07 10:45:31 INFO mapred.JobClient: map 100% reduce >> 68% >> 09/01/07 10:45:36 INFO mapred.JobClient: map 100% reduce >> 69% >> 09/01/07 10:45:41 INFO mapred.JobClient: map 100% reduce >> 70% >> 09/01/07 10:47:03 INFO mapred.JobClient: map 100% reduce >> 0% >> 09/01/07 10:47:03 INFO mapred.JobClient: Task Id : >> attempt_200901071022_0001_r_000002_2, Status : FAILED >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Trying to >> contact region server Some server for region >> DenseMatrix_randhknkr,000000000000257,1231292627625, row >> '000000000000635', but failed after 10 attempts. >> Exceptions: >> >> -- >> Best Regards, Edward J. Yoon @ NHN, corp. >> [email protected] >> http://blog.udanax.org > > > > -- Best Regards, Edward J. Yoon @ NHN, corp. [email protected] http://blog.udanax.org
