Are you having HDFS issues? Did a RegionServer go down at all? Not
sure what's up with the root location issue.
With only 3 nodes and 20 reducers you are pushing things pretty hard...
The best way to investigate how to improve your performance is to look
at the load characteristics of the nodes. I definitely recommend
setting up Ganglia as a first step.
Irfan Mohammed wrote:
Hi,
Hbase/Hadoop Setup:
1. 3 regionservers
2. Run the task using 20 Map Tasks and 20 Reduce Tasks.
3. Using an older hbase version from the trunk [ Version: 0.20.0-dev, r786695, Sat Jun 20 18:01:17 EDT 2009 ]
4. Using hadoop [ 0.20.0 ]
Test Data:
1. The input is a CSV file with a 1M rows and about 20 columns and 4 metrics.
2. Output is 4 hbase tables "txn_m1", "txn_m2", "txn_m3", "txn_m4".
The task is to parse through the CSV file and for each metric m1 create an entry into the
hbase table "txn_m1" with the columns as needed. Attached is an pdf [from an
excel] which explains how a single row in the CSV is converted into hbase data in the
mapper and reducer stage. Attached is the code as well.
For processing a 1M records, it is taking about 38 minutes. I am using HTable.incrementColumnValue() in the reduce pass to create the records in the hbase tables.
Is there anything I should be doing differently or inherently incorrect? I
would like run this task in 1 minute.
Thanks for the help,
Irfan
Here is the output of the process. Let me know if I should attach any other log.
09/07/02 15:19:11 INFO mapred.JobClient: Running job: job_200906192236_5114
09/07/02 15:19:12 INFO mapred.JobClient: map 0% reduce 0%
09/07/02 15:19:29 INFO mapred.JobClient: map 30% reduce 0%
09/07/02 15:19:32 INFO mapred.JobClient: map 46% reduce 0%
09/07/02 15:19:35 INFO mapred.JobClient: map 64% reduce 0%
09/07/02 15:19:38 INFO mapred.JobClient: map 75% reduce 0%
09/07/02 15:19:44 INFO mapred.JobClient: map 76% reduce 0%
09/07/02 15:19:47 INFO mapred.JobClient: map 99% reduce 1%
09/07/02 15:19:50 INFO mapred.JobClient: map 100% reduce 3%
09/07/02 15:19:53 INFO mapred.JobClient: map 100% reduce 4%
09/07/02 15:19:56 INFO mapred.JobClient: map 100% reduce 10%
09/07/02 15:19:59 INFO mapred.JobClient: map 100% reduce 12%
09/07/02 15:20:02 INFO mapred.JobClient: map 100% reduce 16%
09/07/02 15:20:05 INFO mapred.JobClient: map 100% reduce 25%
09/07/02 15:20:08 INFO mapred.JobClient: map 100% reduce 33%
09/07/02 15:20:11 INFO mapred.JobClient: map 100% reduce 36%
09/07/02 15:20:14 INFO mapred.JobClient: map 100% reduce 39%
09/07/02 15:20:17 INFO mapred.JobClient: map 100% reduce 41%
09/07/02 15:20:29 INFO mapred.JobClient: map 100% reduce 42%
09/07/02 15:20:32 INFO mapred.JobClient: map 100% reduce 44%
09/07/02 15:20:38 INFO mapred.JobClient: map 100% reduce 46%
09/07/02 15:20:49 INFO mapred.JobClient: map 100% reduce 47%
09/07/02 15:20:55 INFO mapred.JobClient: map 100% reduce 50%
09/07/02 15:21:01 INFO mapred.JobClient: map 100% reduce 51%
09/07/02 15:21:34 INFO mapred.JobClient: map 100% reduce 52%
09/07/02 15:21:39 INFO mapred.JobClient: map 100% reduce 53%
09/07/02 15:22:06 INFO mapred.JobClient: map 100% reduce 54%
09/07/02 15:22:28 INFO mapred.JobClient: map 100% reduce 55%
09/07/02 15:22:44 INFO mapred.JobClient: map 100% reduce 56%
09/07/02 15:23:02 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000002_0, Status : FAILED
attempt_200906192236_5114_r_000002_0: [2009-07-02 15:20:27.230] fetching new
record writer ...
attempt_200906192236_5114_r_000002_0: [2009-07-02 15:22:51.429] failed to
initialize the hbase configuration
09/07/02 15:23:08 INFO mapred.JobClient: map 100% reduce 53%
09/07/02 15:23:08 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000013_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_5114_r_000013_0: [2009-07-02 15:20:33.183] fetching new
record writer ...
attempt_200906192236_5114_r_000013_0: [2009-07-02 15:23:04.369] failed to
initialize the hbase configuration
09/07/02 15:23:09 INFO mapred.JobClient: map 100% reduce 50%
09/07/02 15:23:14 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000012_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_5114_r_000012_0: [2009-07-02 15:20:48.434] fetching new
record writer ...
attempt_200906192236_5114_r_000012_0: [2009-07-02 15:23:10.185] failed to
initialize the hbase configuration
09/07/02 15:23:15 INFO mapred.JobClient: map 100% reduce 48%
09/07/02 15:23:17 INFO mapred.JobClient: Task Id :
attempt_200906192236_5114_r_000014_0, Status : FAILED
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to
locate root region
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
at
com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
attempt_200906192236_5114_r_000014_0: [2009-07-02 15:20:47.442] fetching new
record writer ...
attempt_200906192236_5114_r_000014_0: [2009-07-02 15:23:13.285] failed to
initialize the hbase configuration
09/07/02 15:23:18 INFO mapred.JobClient: map 100% reduce 45%
09/07/02 15:23:21 INFO mapred.JobClient: map 100% reduce 46%
09/07/02 15:23:29 INFO mapred.JobClient: map 100% reduce 47%
09/07/02 15:23:32 INFO mapred.JobClient: map 100% reduce 48%
09/07/02 15:23:36 INFO mapred.JobClient: map 100% reduce 49%
09/07/02 15:23:39 INFO mapred.JobClient: map 100% reduce 51%
09/07/02 15:23:42 INFO mapred.JobClient: map 100% reduce 56%
09/07/02 15:23:45 INFO mapred.JobClient: map 100% reduce 58%
09/07/02 15:24:20 INFO mapred.JobClient: map 100% reduce 59%
09/07/02 15:25:11 INFO mapred.JobClient: map 100% reduce 60%
09/07/02 15:25:17 INFO mapred.JobClient: map 100% reduce 61%
09/07/02 15:25:26 INFO mapred.JobClient: map 100% reduce 62%
09/07/02 15:25:32 INFO mapred.JobClient: map 100% reduce 64%
09/07/02 15:25:38 INFO mapred.JobClient: map 100% reduce 65%
09/07/02 15:26:20 INFO mapred.JobClient: map 100% reduce 66%
09/07/02 15:26:40 INFO mapred.JobClient: map 100% reduce 67%
09/07/02 15:26:48 INFO mapred.JobClient: map 100% reduce 68%
09/07/02 15:27:16 INFO mapred.JobClient: map 100% reduce 69%
09/07/02 15:27:21 INFO mapred.JobClient: map 100% reduce 70%
09/07/02 15:27:46 INFO mapred.JobClient: map 100% reduce 71%
09/07/02 15:28:25 INFO mapred.JobClient: map 100% reduce 72%
09/07/02 15:28:46 INFO mapred.JobClient: map 100% reduce 73%
09/07/02 15:29:08 INFO mapred.JobClient: map 100% reduce 74%
09/07/02 15:29:45 INFO mapred.JobClient: map 100% reduce 76%
09/07/02 15:30:42 INFO mapred.JobClient: map 100% reduce 77%
09/07/02 15:31:06 INFO mapred.JobClient: map 100% reduce 78%
09/07/02 15:31:12 INFO mapred.JobClient: map 100% reduce 79%
09/07/02 15:31:36 INFO mapred.JobClient: map 100% reduce 81%
09/07/02 15:31:37 INFO mapred.JobClient: map 100% reduce 82%
09/07/02 15:32:00 INFO mapred.JobClient: map 100% reduce 83%
09/07/02 15:32:09 INFO mapred.JobClient: map 100% reduce 84%
09/07/02 15:32:30 INFO mapred.JobClient: map 100% reduce 86%
09/07/02 15:38:42 INFO mapred.JobClient: map 100% reduce 88%
09/07/02 15:39:49 INFO mapred.JobClient: map 100% reduce 89%
09/07/02 15:41:13 INFO mapred.JobClient: map 100% reduce 90%
09/07/02 15:41:16 INFO mapred.JobClient: map 100% reduce 91%
09/07/02 15:41:28 INFO mapred.JobClient: map 100% reduce 93%
09/07/02 15:44:34 INFO mapred.JobClient: map 100% reduce 94%
09/07/02 15:45:41 INFO mapred.JobClient: map 100% reduce 95%
09/07/02 15:45:50 INFO mapred.JobClient: map 100% reduce 96%
09/07/02 15:46:17 INFO mapred.JobClient: map 100% reduce 98%
09/07/02 15:55:29 INFO mapred.JobClient: map 100% reduce 99%
09/07/02 15:57:08 INFO mapred.JobClient: map 100% reduce 100%
09/07/02 15:57:14 INFO mapred.JobClient: Job complete: job_200906192236_5114
09/07/02 15:57:14 INFO mapred.JobClient: Counters: 18
09/07/02 15:57:14 INFO mapred.JobClient: Job Counters
09/07/02 15:57:14 INFO mapred.JobClient: Launched reduce tasks=24
09/07/02 15:57:14 INFO mapred.JobClient: Rack-local map tasks=2
09/07/02 15:57:14 INFO mapred.JobClient: Launched map tasks=20
09/07/02 15:57:14 INFO mapred.JobClient: Data-local map tasks=18
09/07/02 15:57:14 INFO mapred.JobClient: FileSystemCounters
09/07/02 15:57:14 INFO mapred.JobClient: FILE_BYTES_READ=1848609562
09/07/02 15:57:14 INFO mapred.JobClient: HDFS_BYTES_READ=57982980
09/07/02 15:57:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=2768325646
09/07/02 15:57:14 INFO mapred.JobClient: Map-Reduce Framework
09/07/02 15:57:14 INFO mapred.JobClient: Reduce input groups=4863
09/07/02 15:57:14 INFO mapred.JobClient: Combine output records=0
09/07/02 15:57:14 INFO mapred.JobClient: Map input records=294786
09/07/02 15:57:14 INFO mapred.JobClient: Reduce shuffle bytes=883803390
09/07/02 15:57:14 INFO mapred.JobClient: Reduce output records=0
09/07/02 15:57:14 INFO mapred.JobClient: Spilled Records=50956464
09/07/02 15:57:14 INFO mapred.JobClient: Map output bytes=888797024
09/07/02 15:57:14 INFO mapred.JobClient: Map input bytes=57966580
09/07/02 15:57:14 INFO mapred.JobClient: Combine input records=0
09/07/02 15:57:14 INFO mapred.JobClient: Map output records=16985488
09/07/02 15:57:14 INFO mapred.JobClient: Reduce input records=16985488
