Here is the syslog.txt. Any clues why I keep getting the
NoServerForRegionException?
Thanks,
Irfan
----- Original Message -----
From: "Irfan Mohammed" <irfan...@gmail.com>
To: hbase-dev@hadoop.apache.org
Sent: Friday, July 3, 2009 9:00:29 AM GMT -05:00 US/Canada Eastern
Subject: Re: performance help
Thanks for the quick responses.
I removed the reduce pass and doing the inserts in the map pass. Reduced the number of Map instances to 10. It is still taking about 12 minutes to complete the inserts.
Any reason why there should be arbitrary NoServerForRegionException?
I am working on writing to hdfs and checking the performance.
09/07/03 08:38:35 INFO mapred.JobClient: Running job: job_200906192236_24166
09/07/03 08:38:36 INFO mapred.JobClient: map 0% reduce 0%
09/07/03 08:38:53 INFO mapred.JobClient: map 1% reduce 0%
09/07/03 08:38:59 INFO mapred.JobClient: map 2% reduce 0%
09/07/03 08:39:02 INFO mapred.JobClient: map 3% reduce 0%
09/07/03 08:39:08 INFO mapred.JobClient: map 4% reduce 0%
09/07/03 08:39:14 INFO mapred.JobClient: map 5% reduce 0%
09/07/03 08:39:20 INFO mapred.JobClient: map 6% reduce 0%
09/07/03 08:39:26 INFO mapred.JobClient: map 7% reduce 0%
09/07/03 08:39:35 INFO mapred.JobClient: map 8% reduce 0%
09/07/03 08:39:41 INFO mapred.JobClient: map 9% reduce 0%
09/07/03 08:39:50 INFO mapred.JobClient: map 10% reduce 0%
09/07/03 08:39:56 INFO mapred.JobClient: map 11% reduce 0%
09/07/03 08:40:05 INFO mapred.JobClient: map 12% reduce 0%
09/07/03 08:40:14 INFO mapred.JobClient: map 13% reduce 0%
09/07/03 08:40:20 INFO mapred.JobClient: map 14% reduce 0%
09/07/03 08:40:26 INFO mapred.JobClient: map 15% reduce 0%
09/07/03 08:40:32 INFO mapred.JobClient: map 16% reduce 0%
09/07/03 08:40:38 INFO mapred.JobClient: map 17% reduce 0%
09/07/03 08:40:44 INFO mapred.JobClient: map 18% reduce 0%
09/07/03 08:40:46 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000007_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000007_0: [2009-07-03 08:40:42.553] failed to
initialize the hbase configuration
09/07/03 08:40:46 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000009_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000009_0: [2009-07-03 08:40:40.061] failed to
initialize the hbase configuration
09/07/03 08:40:47 INFO mapred.JobClient: map 19% reduce 0%
09/07/03 08:40:49 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000008_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000008_0: [2009-07-03 08:40:44.631] failed to
initialize the hbase configuration
09/07/03 08:40:53 INFO mapred.JobClient: map 20% reduce 0%
09/07/03 08:40:56 INFO mapred.JobClient: map 21% reduce 0%
09/07/03 08:41:02 INFO mapred.JobClient: map 22% reduce 0%
09/07/03 08:41:08 INFO mapred.JobClient: map 23% reduce 0%
09/07/03 08:41:17 INFO mapred.JobClient: map 24% reduce 0%
09/07/03 08:41:26 INFO mapred.JobClient: map 25% reduce 0%
09/07/03 08:41:32 INFO mapred.JobClient: map 26% reduce 0%
09/07/03 08:41:38 INFO mapred.JobClient: map 27% reduce 0%
09/07/03 08:41:44 INFO mapred.JobClient: map 28% reduce 0%
09/07/03 08:41:50 INFO mapred.JobClient: map 29% reduce 0%
09/07/03 08:41:53 INFO mapred.JobClient: map 30% reduce 0%
09/07/03 08:42:02 INFO mapred.JobClient: map 31% reduce 0%
09/07/03 08:42:08 INFO mapred.JobClient: map 32% reduce 0%
09/07/03 08:42:11 INFO mapred.JobClient: map 33% reduce 0%
09/07/03 08:42:17 INFO mapred.JobClient: map 34% reduce 0%
09/07/03 08:42:20 INFO mapred.JobClient: map 35% reduce 0%
09/07/03 08:42:26 INFO mapred.JobClient: map 36% reduce 0%
09/07/03 08:42:32 INFO mapred.JobClient: map 37% reduce 0%
09/07/03 08:42:38 INFO mapred.JobClient: map 38% reduce 0%
09/07/03 08:42:44 INFO mapred.JobClient: map 39% reduce 0%
09/07/03 08:42:53 INFO mapred.JobClient: map 40% reduce 0%
09/07/03 08:42:55 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000009_1, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000009_1: [2009-07-03 08:42:50.373] failed to
initialize the hbase configuration
09/07/03 08:42:55 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000007_1, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000007_1: [2009-07-03 08:42:49.181] failed to
initialize the hbase configuration
09/07/03 08:42:55 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000008_1, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000008_1: [2009-07-03 08:42:49.498] failed to
initialize the hbase configuration
09/07/03 08:42:59 INFO mapred.JobClient: map 41% reduce 0%
09/07/03 08:43:08 INFO mapred.JobClient: map 42% reduce 0%
09/07/03 08:43:14 INFO mapred.JobClient: map 43% reduce 0%
09/07/03 08:43:23 INFO mapred.JobClient: map 44% reduce 0%
09/07/03 08:43:32 INFO mapred.JobClient: map 45% reduce 0%
09/07/03 08:43:41 INFO mapred.JobClient: map 46% reduce 0%
09/07/03 08:43:50 INFO mapred.JobClient: map 47% reduce 0%
09/07/03 08:43:56 INFO mapred.JobClient: map 48% reduce 0%
09/07/03 08:44:02 INFO mapred.JobClient: map 49% reduce 0%
09/07/03 08:44:08 INFO mapred.JobClient: map 50% reduce 0%
09/07/03 08:44:14 INFO mapred.JobClient: map 51% reduce 0%
09/07/03 08:44:20 INFO mapred.JobClient: map 52% reduce 0%
09/07/03 08:44:23 INFO mapred.JobClient: map 53% reduce 0%
09/07/03 08:44:29 INFO mapred.JobClient: map 54% reduce 0%
09/07/03 08:44:35 INFO mapred.JobClient: map 55% reduce 0%
09/07/03 08:44:38 INFO mapred.JobClient: map 56% reduce 0%
09/07/03 08:44:47 INFO mapred.JobClient: map 57% reduce 0%
09/07/03 08:44:53 INFO mapred.JobClient: map 58% reduce 0%
09/07/03 08:45:01 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000007_2, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000007_2: [2009-07-03 08:44:55.897] failed to
initialize the hbase configuration
09/07/03 08:45:01 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000009_2, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000009_2: [2009-07-03 08:44:56.296] failed to
initialize the hbase configuration
09/07/03 08:45:02 INFO mapred.JobClient: map 59% reduce 0%
09/07/03 08:45:04 INFO mapred.JobClient: Task Id :
attempt_200906192236_24166_m_000008_2, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_24166_m_000008_2: [2009-07-03 08:44:59.221] failed to
initialize the hbase configuration
09/07/03 08:45:08 INFO mapred.JobClient: map 60% reduce 0%
09/07/03 08:45:17 INFO mapred.JobClient: map 61% reduce 0%
09/07/03 08:45:26 INFO mapred.JobClient: map 62% reduce 0%
09/07/03 08:45:32 INFO mapred.JobClient: map 63% reduce 0%
09/07/03 08:45:38 INFO mapred.JobClient: map 64% reduce 0%
09/07/03 08:45:44 INFO mapred.JobClient: map 65% reduce 0%
09/07/03 08:45:50 INFO mapred.JobClient: map 66% reduce 0%
09/07/03 08:45:56 INFO mapred.JobClient: map 67% reduce 0%
09/07/03 08:46:02 INFO mapred.JobClient: map 68% reduce 0%
09/07/03 08:46:08 INFO mapred.JobClient: map 69% reduce 0%
09/07/03 08:46:15 INFO mapred.JobClient: map 70% reduce 0%
09/07/03 08:46:21 INFO mapred.JobClient: map 71% reduce 0%
09/07/03 08:46:27 INFO mapred.JobClient: map 72% reduce 0%
09/07/03 08:46:36 INFO mapred.JobClient: map 73% reduce 0%
09/07/03 08:46:45 INFO mapred.JobClient: map 74% reduce 0%
09/07/03 08:46:54 INFO mapred.JobClient: map 75% reduce 0%
09/07/03 08:47:03 INFO mapred.JobClient: map 76% reduce 0%
09/07/03 08:47:12 INFO mapred.JobClient: map 77% reduce 0%
09/07/03 08:47:18 INFO mapred.JobClient: map 78% reduce 0%
09/07/03 08:47:24 INFO mapred.JobClient: map 79% reduce 0%
09/07/03 08:47:33 INFO mapred.JobClient: map 80% reduce 0%
09/07/03 08:47:42 INFO mapred.JobClient: map 81% reduce 0%
09/07/03 08:47:51 INFO mapred.JobClient: map 82% reduce 0%
09/07/03 08:48:00 INFO mapred.JobClient: map 83% reduce 0%
09/07/03 08:48:09 INFO mapred.JobClient: map 84% reduce 0%
09/07/03 08:48:15 INFO mapred.JobClient: map 85% reduce 0%
09/07/03 08:48:24 INFO mapred.JobClient: map 86% reduce 0%
09/07/03 08:48:30 INFO mapred.JobClient: map 87% reduce 0%
09/07/03 08:48:39 INFO mapred.JobClient: map 88% reduce 0%
09/07/03 08:48:54 INFO mapred.JobClient: map 89% reduce 0%
09/07/03 08:49:06 INFO mapred.JobClient: map 90% reduce 0%
09/07/03 08:49:15 INFO mapred.JobClient: map 91% reduce 0%
09/07/03 08:49:24 INFO mapred.JobClient: map 92% reduce 0%
09/07/03 08:49:30 INFO mapred.JobClient: map 93% reduce 0%
09/07/03 08:49:36 INFO mapred.JobClient: map 94% reduce 0%
09/07/03 08:49:45 INFO mapred.JobClient: map 95% reduce 0%
09/07/03 08:49:57 INFO mapred.JobClient: map 96% reduce 0%
09/07/03 08:50:08 INFO mapred.JobClient: map 97% reduce 0%
09/07/03 08:50:17 INFO mapred.JobClient: map 98% reduce 0%
09/07/03 08:50:26 INFO mapred.JobClient: map 99% reduce 0%
09/07/03 08:50:35 INFO mapred.JobClient: map 100% reduce 0%
09/07/03 08:50:40 INFO mapred.JobClient: Job complete: job_200906192236_24166
09/07/03 08:50:40 INFO mapred.JobClient: Counters: 7
09/07/03 08:50:40 INFO mapred.JobClient: Job Counters
09/07/03 08:50:40 INFO mapred.JobClient: Launched map tasks=19
09/07/03 08:50:40 INFO mapred.JobClient: Data-local map tasks=19
09/07/03 08:50:40 INFO mapred.JobClient: FileSystemCounters
09/07/03 08:50:40 INFO mapred.JobClient: HDFS_BYTES_READ=57966580
09/07/03 08:50:40 INFO mapred.JobClient: Map-Reduce Framework
09/07/03 08:50:40 INFO mapred.JobClient: Map input records=294786
09/07/03 08:50:40 INFO mapred.JobClient: Spilled Records=0
09/07/03 08:50:40 INFO mapred.JobClient: Map input bytes=57966580
09/07/03 08:50:40 INFO mapred.JobClient: Map output records=0
----- Original Message -----
From: "stack" <st...@duboce.net>
To: hbase-dev@hadoop.apache.org
Sent: Thursday, July 2, 2009 6:12:29 PM GMT -05:00 US/Canada Eastern
Subject: Re: performance help
Why 4 tables? Why not one table and four column families, one for each
metric? (Looking in excel spreadsheet, each row has same key). Then you'd
be doing one insert against a single table rather than four separate ones.
Looking at your MR output below, it looks like it takes 40 seconds to
complete the map tasks. The report says that there 294786 inputs. Says
that the mapper outputs 17M records. Is that expected?
A few of your reducers failed and were done over again. The redos were
probably significant part of the overall elapsed time. The failures are
trying to find root region. Root region is in zk. Odd it can't be found
there.
The fetching of map data and sort is taking a considerable amount of the
overall time. Do you need to reduce step (Couldn't tell from the excel
spreadsheet -- there didn't seem to be any summing going on). If not, this
could make for savings too.
You might try outputting to hdfs first to see how fast the job runs with no
hbase involved. See how long that takes. Tune this part of the job first.
Then add in hbase and see how much it slows things.
Looking at your code, nothing obviously onerous.
St.Ack
On Thu, Jul 2, 2009 at 1:22 PM, Irfan Mohammed <irfan...@gmail.com> wrote:
Hi,
Hbase/Hadoop Setup:
1. 3 regionservers
2. Run the task using 20 Map Tasks and 20 Reduce Tasks.
3. Using an older hbase version from the trunk [ Version: 0.20.0-dev,
r786695, Sat Jun 20 18:01:17 EDT 2009 ]
4. Using hadoop [ 0.20.0 ]
Test Data:
1. The input is a CSV file with a 1M rows and about 20 columns and 4
metrics.
2. Output is 4 hbase tables "txn_m1", "txn_m2", "txn_m3", "txn_m4".
The task is to parse through the CSV file and for each metric m1 create an
entry into the hbase table "txn_m1" with the columns as needed. Attached is
an pdf [from an excel] which explains how a single row in the CSV is
converted into hbase data in the mapper and reducer stage. Attached is the
code as well.
For processing a 1M records, it is taking about 38 minutes. I am using
HTable.incrementColumnValue() in the reduce pass to create the records in
the hbase tables.
Is there anything I should be doing differently or inherently incorrect? I
would like run this task in 1 minute.
Thanks for the help,
Irfan
Here is the output of the process. Let me know if I should attach any other
log.
09/07/02 15:19:11 INFO mapred.JobClient: Running job: job_200906192236_5114
09/07/02 15:19:12 INFO mapred.JobClient: map 0% reduce 0%
09/07/02 15:19:29 INFO mapred.JobClient: map 30% reduce 0%
09/07/02 15:19:32 INFO mapred.JobClient: map 46% reduce 0%
09/07/02 15:19:35 INFO mapred.JobClient: map 64% reduce 0%
09/07/02 15:19:38 INFO mapred.JobClient: map 75% reduce 0%
09/07/02 15:19:44 INFO mapred.JobClient: map 76% reduce 0%
09/07/02 15:19:47 INFO mapred.JobClient: map 99% reduce 1%
09/07/02 15:19:50 INFO mapred.JobClient: map 100% reduce 3%
09/07/02 15:19:53 INFO mapred.JobClient: map 100% reduce 4%
09/07/02 15:19:56 INFO mapred.JobClient: map 100% reduce 10%
09/07/02 15:19:59 INFO mapred.JobClient: map 100% reduce 12%
09/07/02 15:20:02 INFO mapred.JobClient: map 100% reduce 16%
09/07/02 15:20:05 INFO mapred.JobClient: map 100% reduce 25%
09/07/02 15:20:08 INFO mapred.JobClient: map 100% reduce 33%
09/07/02 15:20:11 INFO mapred.JobClient: map 100% reduce 36%
09/07/02 15:20:14 INFO mapred.JobClient: map 100% reduce 39%
09/07/02 15:20:17 INFO mapred.JobClient: map 100% reduce 41%
09/07/02 15:20:29 INFO mapred.JobClient: map 100% reduce 42%
09/07/02 15:20:32 INFO mapred.JobClient: map 100% reduce 44%
09/07/02 15:20:38 INFO mapred.JobClient: map 100% reduce 46%
09/07/02 15:20:49 INFO mapred.JobClient: map 100% reduce 47%
09/07/02 15:20:55 INFO mapred.JobClient: map 100% reduce 50%
09/07/02 15:21:01 INFO mapred.JobClient: map 100% reduce 51%
09/07/02 15:21:34 INFO mapred.JobClient: map 100% reduce 52%
09/07/02 15:21:39 INFO mapred.JobClient: map 100% reduce 53%
09/07/02 15:22:06 INFO mapred.JobClient: map 100% reduce 54%
09/07/02 15:22:28 INFO mapred.JobClient: map 100% reduce 55%
09/07/02 15:22:44 INFO mapred.JobClient: map 100% reduce 56%
09/07/02 15:23:02 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000002_0, Status : FAILED
attempt_200906192236_5114_r_000002_0: [2009-07-02 15:20:27.230] fetching
new record writer ...
attempt_200906192236_5114_r_000002_0: [2009-07-02 15:22:51.429] failed to
initialize the hbase configuration
09/07/02 15:23:08 INFO mapred.JobClient: map 100% reduce 53%
09/07/02 15:23:08 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000013_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
to locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_5114_r_000013_0: [2009-07-02 15:20:33.183] fetching
new record writer ...
attempt_200906192236_5114_r_000013_0: [2009-07-02 15:23:04.369] failed to
initialize the hbase configuration
09/07/02 15:23:09 INFO mapred.JobClient: map 100% reduce 50%
09/07/02 15:23:14 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000012_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
to locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_5114_r_000012_0: [2009-07-02 15:20:48.434] fetching
new record writer ...
attempt_200906192236_5114_r_000012_0: [2009-07-02 15:23:10.185] failed to
initialize the hbase configuration
09/07/02 15:23:15 INFO mapred.JobClient: map 100% reduce 48%
09/07/02 15:23:17 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000014_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying
to locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_5114_r_000014_0: [2009-07-02 15:20:47.442] fetching
new record writer ...
attempt_200906192236_5114_r_000014_0: [2009-07-02 15:23:13.285] failed to
initialize the hbase configuration
09/07/02 15:23:18 INFO mapred.JobClient: map 100% reduce 45%
09/07/02 15:23:21 INFO mapred.JobClient: map 100% reduce 46%
09/07/02 15:23:29 INFO mapred.JobClient: map 100% reduce 47%
09/07/02 15:23:32 INFO mapred.JobClient: map 100% reduce 48%
09/07/02 15:23:36 INFO mapred.JobClient: map 100% reduce 49%
09/07/02 15:23:39 INFO mapred.JobClient: map 100% reduce 51%
09/07/02 15:23:42 INFO mapred.JobClient: map 100% reduce 56%
09/07/02 15:23:45 INFO mapred.JobClient: map 100% reduce 58%
09/07/02 15:24:20 INFO mapred.JobClient: map 100% reduce 59%
09/07/02 15:25:11 INFO mapred.JobClient: map 100% reduce 60%
09/07/02 15:25:17 INFO mapred.JobClient: map 100% reduce 61%
09/07/02 15:25:26 INFO mapred.JobClient: map 100% reduce 62%
09/07/02 15:25:32 INFO mapred.JobClient: map 100% reduce 64%
09/07/02 15:25:38 INFO mapred.JobClient: map 100% reduce 65%
09/07/02 15:26:20 INFO mapred.JobClient: map 100% reduce 66%
09/07/02 15:26:40 INFO mapred.JobClient: map 100% reduce 67%
09/07/02 15:26:48 INFO mapred.JobClient: map 100% reduce 68%
09/07/02 15:27:16 INFO mapred.JobClient: map 100% reduce 69%
09/07/02 15:27:21 INFO mapred.JobClient: map 100% reduce 70%
09/07/02 15:27:46 INFO mapred.JobClient: map 100% reduce 71%
09/07/02 15:28:25 INFO mapred.JobClient: map 100% reduce 72%
09/07/02 15:28:46 INFO mapred.JobClient: map 100% reduce 73%
09/07/02 15:29:08 INFO mapred.JobClient: map 100% reduce 74%
09/07/02 15:29:45 INFO mapred.JobClient: map 100% reduce 76%
09/07/02 15:30:42 INFO mapred.JobClient: map 100% reduce 77%
09/07/02 15:31:06 INFO mapred.JobClient: map 100% reduce 78%
09/07/02 15:31:12 INFO mapred.JobClient: map 100% reduce 79%
09/07/02 15:31:36 INFO mapred.JobClient: map 100% reduce 81%
09/07/02 15:31:37 INFO mapred.JobClient: map 100% reduce 82%
09/07/02 15:32:00 INFO mapred.JobClient: map 100% reduce 83%
09/07/02 15:32:09 INFO mapred.JobClient: map 100% reduce 84%
09/07/02 15:32:30 INFO mapred.JobClient: map 100% reduce 86%
09/07/02 15:38:42 INFO mapred.JobClient: map 100% reduce 88%
09/07/02 15:39:49 INFO mapred.JobClient: map 100% reduce 89%
09/07/02 15:41:13 INFO mapred.JobClient: map 100% reduce 90%
09/07/02 15:41:16 INFO mapred.JobClient: map 100% reduce 91%
09/07/02 15:41:28 INFO mapred.JobClient: map 100% reduce 93%
09/07/02 15:44:34 INFO mapred.JobClient: map 100% reduce 94%
09/07/02 15:45:41 INFO mapred.JobClient: map 100% reduce 95%
09/07/02 15:45:50 INFO mapred.JobClient: map 100% reduce 96%
09/07/02 15:46:17 INFO mapred.JobClient: map 100% reduce 98%
09/07/02 15:55:29 INFO mapred.JobClient: map 100% reduce 99%
09/07/02 15:57:08 INFO mapred.JobClient: map 100% reduce 100%
09/07/02 15:57:14 INFO mapred.JobClient: Job complete:
job_200906192236_5114
09/07/02 15:57:14 INFO mapred.JobClient: Counters: 18
09/07/02 15:57:14 INFO mapred.JobClient: Job Counters
09/07/02 15:57:14 INFO mapred.JobClient: Launched reduce tasks=24
09/07/02 15:57:14 INFO mapred.JobClient: Rack-local map tasks=2
09/07/02 15:57:14 INFO mapred.JobClient: Launched map tasks=20
09/07/02 15:57:14 INFO mapred.JobClient: Data-local map tasks=18
09/07/02 15:57:14 INFO mapred.JobClient: FileSystemCounters
09/07/02 15:57:14 INFO mapred.JobClient: FILE_BYTES_READ=1848609562
09/07/02 15:57:14 INFO mapred.JobClient: HDFS_BYTES_READ=57982980
09/07/02 15:57:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2768325646
09/07/02 15:57:14 INFO mapred.JobClient: Map-Reduce Framework
09/07/02 15:57:14 INFO mapred.JobClient: Reduce input groups=4863
09/07/02 15:57:14 INFO mapred.JobClient: Combine output records=0
09/07/02 15:57:14 INFO mapred.JobClient: Map input records=294786
09/07/02 15:57:14 INFO mapred.JobClient: Reduce shuffle bytes=883803390
09/07/02 15:57:14 INFO mapred.JobClient: Reduce output records=0
09/07/02 15:57:14 INFO mapred.JobClient: Spilled Records=50956464
09/07/02 15:57:14 INFO mapred.JobClient: Map output bytes=888797024
09/07/02 15:57:14 INFO mapred.JobClient: Map input bytes=57966580
09/07/02 15:57:14 INFO mapred.JobClient: Combine input records=0
09/07/02 15:57:14 INFO mapred.JobClient: Map output records=16985488
09/07/02 15:57:14 INFO mapred.JobClient: Reduce input records=16985488
