I am working on writing to HDFS files. Will update you by end of day today.
There are always 10 concurrent mappers running. I keep setting the
setNumMaps(5) and also the following properties in mapred-site.xml to 3 but
still end up running 10 concurrent maps.
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>3</value>
<description>The maximum number of map tasks that will be run
simultaneously by a task tracker.
</description>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>3</value>
<description>The maximum number of reduce tasks that will be run
simultaneously by a task tracker.
</description>
</property>
There are 5 regionservers and the online regions are as follows :
m1 : -ROOT-,,0
m2 : txn_m1,,1245462904101
m3 : txn_m4,,1245462942282
m4 : txn_m2,,1245462890248
m5 : .META.,,1
txn_m3,,1245460727203
I have setAutoFlush(false) and also writeToWal(false) with the same behaviour.
----- Original Message -----
From: "stack" <st...@duboce.net>
To: hbase-dev@hadoop.apache.org
Sent: Monday, July 6, 2009 11:34:07 AM GMT -05:00 US/Canada Eastern
Subject: Re: performance help
So, no difference in overall elapsed time nearly doubling the number of
servers writing 7M updates? Updating 4 tables takes same time as updating
one table? Have you tried writing to files in HDFS to see if time is any
faster to verify that hbase is whats holding up your job?
So, you have 10 maps to complete. How many concurrent mappers do you have
running? 2 per node?
Regards whether splits are happening, are number of regions going up as the
job runs? (You can see in the UI).
Are you batching your updates?
http://hadoop.apache.org/hbase/docs/r0.19.3/api/org/apache/hadoop/hbase/client/HTable.html#setAutoFlush(boolean)
You could try setting Put#writeToWAL to false to see what difference that
makes in your upload.
St.Ack
On Mon, Jul 6, 2009 at 8:09 AM, Irfan Mohammed <irfan...@gmail.com> wrote:
> I added 2 more regionservers and now have 5 regionservers but the inserts
> times are pretty constant around 10-12 minutes. As far I can see the tasks
> are distributed across the 5 regionservers and all [ 10 map tasks ] of them
> start at the same time and complete in ~ 12 minutes.
>
> How and Where can I check whether the update splits are happening and which
> ones are taking long time?
>
> I checked with a single table and four tables and the results are pretty
> consistent of about 12 minutes.
>
> Thanks.
>
> ----- Original Message -----
> From: "stack" <st...@duboce.net>
> To: hbase-dev@hadoop.apache.org
> Sent: Sunday, July 5, 2009 5:31:45 PM GMT -05:00 US/Canada Eastern
> Subject: Re: performance help
>
> On Sat, Jul 4, 2009 at 8:51 PM, Irfan Mohammed <irfan...@gmail.com> wrote:
>
> > my zookeeper quorum had just one server and after jon gray's suggestion
> > added two more to the quorom and the task did not have any failures.
> >
> >
> That is good to know though I think that if a single zk instance is not
> able
> to handle loading of 3 nodes, I think there's something up with it. We'll
> take a look into it.
>
>
>
> > but still took 10 minutes for it to finish in my 3 nodes cluster. i am
> > trying to add more nodes to the cluster and see if i get a better
> > performance.
> >
>
> Yeah, this would be good to know.
>
> So you are doing all in the map now but still updating 4 tables on each
> update? (200k rows in become 7M rows out)? What do you see if you study
> the
> UI? Are the updates split evenly across all 3 servers or are they marching
> lockstep across the table's regions? (i.e. are updates spread across all
> servers or do we bang on one at a time?)
>
>
>
> >
> > regarding the question of # of columns per family, we are looking at the
> > most of 20 families and the # of columns per family varies from
> 100-10000.
> > would that be a problem in hbase?
>
>
>
> According to Jon Gray who tested how hbase does with many columns, only
> real
> issue will be memory; returning 10k columns on one row all in the one go,
> especially if they are of any significant size, could put pressure on
> server+client memory. Otherwise, it should work fine (There are
> optimizations we need to do to make it faster than it is, but its for sure
> way better than it was in 0.19.x).
>
> St.Ack
>
>
> >
> > ----- Original Message -----
> > From: "stack" <st...@duboce.net>
> > To: hbase-dev@hadoop.apache.org
> > Sent: Friday, July 3, 2009 5:43:45 PM GMT -05:00 US/Canada Eastern
> > Subject: Re: performance help
> >
> > Those NoServerForRegionException are probably putting a stake through
> > throughput especially when they are complaining root is unobtainable.
> Lets
> > try and figure whats up here (Jon Gray has a good suggestion in this
> > regard).
> >
> > On schema, how many columns do you think you'll have per family? The
> > number
> > of columns story has improved by a bunch in hbase 0.20.0. Should be able
> > to
> > do thousands if not more (per column family).
> >
> > St.Ack
> >
> >
> > On Fri, Jul 3, 2009 at 6:00 AM, Irfan Mohammed <irfan...@gmail.com>
> wrote:
> >
> > > Thanks for the quick responses.
> > >
> > > I removed the reduce pass and doing the inserts in the map pass.
> Reduced
> > > the number of Map instances to 10. It is still taking about 12 minutes
> to
> > > complete the inserts.
> > >
> > > Any reason why there should be arbitrary NoServerForRegionException?
> > >
> > > I am working on writing to hdfs and checking the performance.
> > >
> > > 09/07/03 08:38:35 INFO mapred.JobClient: Running job:
> > > job_200906192236_24166
> > > 09/07/03 08:38:36 INFO mapred.JobClient: map 0% reduce 0%
> > > 09/07/03 08:38:53 INFO mapred.JobClient: map 1% reduce 0%
> > > 09/07/03 08:38:59 INFO mapred.JobClient: map 2% reduce 0%
> > > 09/07/03 08:39:02 INFO mapred.JobClient: map 3% reduce 0%
> > > 09/07/03 08:39:08 INFO mapred.JobClient: map 4% reduce 0%
> > > 09/07/03 08:39:14 INFO mapred.JobClient: map 5% reduce 0%
> > > 09/07/03 08:39:20 INFO mapred.JobClient: map 6% reduce 0%
> > > 09/07/03 08:39:26 INFO mapred.JobClient: map 7% reduce 0%
> > > 09/07/03 08:39:35 INFO mapred.JobClient: map 8% reduce 0%
> > > 09/07/03 08:39:41 INFO mapred.JobClient: map 9% reduce 0%
> > > 09/07/03 08:39:50 INFO mapred.JobClient: map 10% reduce 0%
> > > 09/07/03 08:39:56 INFO mapred.JobClient: map 11% reduce 0%
> > > 09/07/03 08:40:05 INFO mapred.JobClient: map 12% reduce 0%
> > > 09/07/03 08:40:14 INFO mapred.JobClient: map 13% reduce 0%
> > > 09/07/03 08:40:20 INFO mapred.JobClient: map 14% reduce 0%
> > > 09/07/03 08:40:26 INFO mapred.JobClient: map 15% reduce 0%
> > > 09/07/03 08:40:32 INFO mapred.JobClient: map 16% reduce 0%
> > > 09/07/03 08:40:38 INFO mapred.JobClient: map 17% reduce 0%
> > > 09/07/03 08:40:44 INFO mapred.JobClient: map 18% reduce 0%
> > > 09/07/03 08:40:46 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000007_0, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000007_0: [2009-07-03 08:40:42.553] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:40:46 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000009_0, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000009_0: [2009-07-03 08:40:40.061] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:40:47 INFO mapred.JobClient: map 19% reduce 0%
> > > 09/07/03 08:40:49 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000008_0, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000008_0: [2009-07-03 08:40:44.631] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:40:53 INFO mapred.JobClient: map 20% reduce 0%
> > > 09/07/03 08:40:56 INFO mapred.JobClient: map 21% reduce 0%
> > > 09/07/03 08:41:02 INFO mapred.JobClient: map 22% reduce 0%
> > > 09/07/03 08:41:08 INFO mapred.JobClient: map 23% reduce 0%
> > > 09/07/03 08:41:17 INFO mapred.JobClient: map 24% reduce 0%
> > > 09/07/03 08:41:26 INFO mapred.JobClient: map 25% reduce 0%
> > > 09/07/03 08:41:32 INFO mapred.JobClient: map 26% reduce 0%
> > > 09/07/03 08:41:38 INFO mapred.JobClient: map 27% reduce 0%
> > > 09/07/03 08:41:44 INFO mapred.JobClient: map 28% reduce 0%
> > > 09/07/03 08:41:50 INFO mapred.JobClient: map 29% reduce 0%
> > > 09/07/03 08:41:53 INFO mapred.JobClient: map 30% reduce 0%
> > > 09/07/03 08:42:02 INFO mapred.JobClient: map 31% reduce 0%
> > > 09/07/03 08:42:08 INFO mapred.JobClient: map 32% reduce 0%
> > > 09/07/03 08:42:11 INFO mapred.JobClient: map 33% reduce 0%
> > > 09/07/03 08:42:17 INFO mapred.JobClient: map 34% reduce 0%
> > > 09/07/03 08:42:20 INFO mapred.JobClient: map 35% reduce 0%
> > > 09/07/03 08:42:26 INFO mapred.JobClient: map 36% reduce 0%
> > > 09/07/03 08:42:32 INFO mapred.JobClient: map 37% reduce 0%
> > > 09/07/03 08:42:38 INFO mapred.JobClient: map 38% reduce 0%
> > > 09/07/03 08:42:44 INFO mapred.JobClient: map 39% reduce 0%
> > > 09/07/03 08:42:53 INFO mapred.JobClient: map 40% reduce 0%
> > > 09/07/03 08:42:55 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000009_1, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000009_1: [2009-07-03 08:42:50.373] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:42:55 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000007_1, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000007_1: [2009-07-03 08:42:49.181] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:42:55 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000008_1, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000008_1: [2009-07-03 08:42:49.498] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:42:59 INFO mapred.JobClient: map 41% reduce 0%
> > > 09/07/03 08:43:08 INFO mapred.JobClient: map 42% reduce 0%
> > > 09/07/03 08:43:14 INFO mapred.JobClient: map 43% reduce 0%
> > > 09/07/03 08:43:23 INFO mapred.JobClient: map 44% reduce 0%
> > > 09/07/03 08:43:32 INFO mapred.JobClient: map 45% reduce 0%
> > > 09/07/03 08:43:41 INFO mapred.JobClient: map 46% reduce 0%
> > > 09/07/03 08:43:50 INFO mapred.JobClient: map 47% reduce 0%
> > > 09/07/03 08:43:56 INFO mapred.JobClient: map 48% reduce 0%
> > > 09/07/03 08:44:02 INFO mapred.JobClient: map 49% reduce 0%
> > > 09/07/03 08:44:08 INFO mapred.JobClient: map 50% reduce 0%
> > > 09/07/03 08:44:14 INFO mapred.JobClient: map 51% reduce 0%
> > > 09/07/03 08:44:20 INFO mapred.JobClient: map 52% reduce 0%
> > > 09/07/03 08:44:23 INFO mapred.JobClient: map 53% reduce 0%
> > > 09/07/03 08:44:29 INFO mapred.JobClient: map 54% reduce 0%
> > > 09/07/03 08:44:35 INFO mapred.JobClient: map 55% reduce 0%
> > > 09/07/03 08:44:38 INFO mapred.JobClient: map 56% reduce 0%
> > > 09/07/03 08:44:47 INFO mapred.JobClient: map 57% reduce 0%
> > > 09/07/03 08:44:53 INFO mapred.JobClient: map 58% reduce 0%
> > > 09/07/03 08:45:01 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000007_2, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000007_2: [2009-07-03 08:44:55.897] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:45:01 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000009_2, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000009_2: [2009-07-03 08:44:56.296] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:45:02 INFO mapred.JobClient: map 59% reduce 0%
> > > 09/07/03 08:45:04 INFO mapred.JobClient: Task Id :
> > > attempt_200906192236_24166_m_000008_2, Status : FAILED
> > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > trying
> > > to locate root region
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > at
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > at
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:449)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.<init>(MapTask.java:558)
> > > at
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350)
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > > attempt_200906192236_24166_m_000008_2: [2009-07-03 08:44:59.221] failed
> > to
> > > initialize the hbase configuration
> > > 09/07/03 08:45:08 INFO mapred.JobClient: map 60% reduce 0%
> > > 09/07/03 08:45:17 INFO mapred.JobClient: map 61% reduce 0%
> > > 09/07/03 08:45:26 INFO mapred.JobClient: map 62% reduce 0%
> > > 09/07/03 08:45:32 INFO mapred.JobClient: map 63% reduce 0%
> > > 09/07/03 08:45:38 INFO mapred.JobClient: map 64% reduce 0%
> > > 09/07/03 08:45:44 INFO mapred.JobClient: map 65% reduce 0%
> > > 09/07/03 08:45:50 INFO mapred.JobClient: map 66% reduce 0%
> > > 09/07/03 08:45:56 INFO mapred.JobClient: map 67% reduce 0%
> > > 09/07/03 08:46:02 INFO mapred.JobClient: map 68% reduce 0%
> > > 09/07/03 08:46:08 INFO mapred.JobClient: map 69% reduce 0%
> > > 09/07/03 08:46:15 INFO mapred.JobClient: map 70% reduce 0%
> > > 09/07/03 08:46:21 INFO mapred.JobClient: map 71% reduce 0%
> > > 09/07/03 08:46:27 INFO mapred.JobClient: map 72% reduce 0%
> > > 09/07/03 08:46:36 INFO mapred.JobClient: map 73% reduce 0%
> > > 09/07/03 08:46:45 INFO mapred.JobClient: map 74% reduce 0%
> > > 09/07/03 08:46:54 INFO mapred.JobClient: map 75% reduce 0%
> > > 09/07/03 08:47:03 INFO mapred.JobClient: map 76% reduce 0%
> > > 09/07/03 08:47:12 INFO mapred.JobClient: map 77% reduce 0%
> > > 09/07/03 08:47:18 INFO mapred.JobClient: map 78% reduce 0%
> > > 09/07/03 08:47:24 INFO mapred.JobClient: map 79% reduce 0%
> > > 09/07/03 08:47:33 INFO mapred.JobClient: map 80% reduce 0%
> > > 09/07/03 08:47:42 INFO mapred.JobClient: map 81% reduce 0%
> > > 09/07/03 08:47:51 INFO mapred.JobClient: map 82% reduce 0%
> > > 09/07/03 08:48:00 INFO mapred.JobClient: map 83% reduce 0%
> > > 09/07/03 08:48:09 INFO mapred.JobClient: map 84% reduce 0%
> > > 09/07/03 08:48:15 INFO mapred.JobClient: map 85% reduce 0%
> > > 09/07/03 08:48:24 INFO mapred.JobClient: map 86% reduce 0%
> > > 09/07/03 08:48:30 INFO mapred.JobClient: map 87% reduce 0%
> > > 09/07/03 08:48:39 INFO mapred.JobClient: map 88% reduce 0%
> > > 09/07/03 08:48:54 INFO mapred.JobClient: map 89% reduce 0%
> > > 09/07/03 08:49:06 INFO mapred.JobClient: map 90% reduce 0%
> > > 09/07/03 08:49:15 INFO mapred.JobClient: map 91% reduce 0%
> > > 09/07/03 08:49:24 INFO mapred.JobClient: map 92% reduce 0%
> > > 09/07/03 08:49:30 INFO mapred.JobClient: map 93% reduce 0%
> > > 09/07/03 08:49:36 INFO mapred.JobClient: map 94% reduce 0%
> > > 09/07/03 08:49:45 INFO mapred.JobClient: map 95% reduce 0%
> > > 09/07/03 08:49:57 INFO mapred.JobClient: map 96% reduce 0%
> > > 09/07/03 08:50:08 INFO mapred.JobClient: map 97% reduce 0%
> > > 09/07/03 08:50:17 INFO mapred.JobClient: map 98% reduce 0%
> > > 09/07/03 08:50:26 INFO mapred.JobClient: map 99% reduce 0%
> > > 09/07/03 08:50:35 INFO mapred.JobClient: map 100% reduce 0%
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Job complete:
> > > job_200906192236_24166
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Counters: 7
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Job Counters
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Launched map tasks=19
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Data-local map tasks=19
> > > 09/07/03 08:50:40 INFO mapred.JobClient: FileSystemCounters
> > > 09/07/03 08:50:40 INFO mapred.JobClient: HDFS_BYTES_READ=57966580
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Map-Reduce Framework
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Map input records=294786
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Spilled Records=0
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Map input bytes=57966580
> > > 09/07/03 08:50:40 INFO mapred.JobClient: Map output records=0
> > >
> > >
> > > ----- Original Message -----
> > > From: "stack" <st...@duboce.net>
> > > To: hbase-dev@hadoop.apache.org
> > > Sent: Thursday, July 2, 2009 6:12:29 PM GMT -05:00 US/Canada Eastern
> > > Subject: Re: performance help
> > >
> > > Why 4 tables? Why not one table and four column families, one for each
> > > metric? (Looking in excel spreadsheet, each row has same key). Then
> > you'd
> > > be doing one insert against a single table rather than four separate
> > ones.
> > >
> > > Looking at your MR output below, it looks like it takes 40 seconds to
> > > complete the map tasks. The report says that there 294786 inputs.
> Says
> > > that the mapper outputs 17M records. Is that expected?
> > >
> > > A few of your reducers failed and were done over again. The redos were
> > > probably significant part of the overall elapsed time. The failures
> are
> > > trying to find root region. Root region is in zk. Odd it can't be
> found
> > > there.
> > >
> > > The fetching of map data and sort is taking a considerable amount of
> the
> > > overall time. Do you need to reduce step (Couldn't tell from the excel
> > > spreadsheet -- there didn't seem to be any summing going on). If not,
> > this
> > > could make for savings too.
> > >
> > > You might try outputting to hdfs first to see how fast the job runs
> with
> > no
> > > hbase involved. See how long that takes. Tune this part of the job
> > first.
> > > Then add in hbase and see how much it slows things.
> > >
> > > Looking at your code, nothing obviously onerous.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Jul 2, 2009 at 1:22 PM, Irfan Mohammed <irfan...@gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Hbase/Hadoop Setup:
> > > > 1. 3 regionservers
> > > > 2. Run the task using 20 Map Tasks and 20 Reduce Tasks.
> > > > 3. Using an older hbase version from the trunk [ Version: 0.20.0-dev,
> > > > r786695, Sat Jun 20 18:01:17 EDT 2009 ]
> > > > 4. Using hadoop [ 0.20.0 ]
> > > >
> > > > Test Data:
> > > > 1. The input is a CSV file with a 1M rows and about 20 columns and 4
> > > > metrics.
> > > > 2. Output is 4 hbase tables "txn_m1", "txn_m2", "txn_m3", "txn_m4".
> > > >
> > > > The task is to parse through the CSV file and for each metric m1
> create
> > > an
> > > > entry into the hbase table "txn_m1" with the columns as needed.
> > Attached
> > > is
> > > > an pdf [from an excel] which explains how a single row in the CSV is
> > > > converted into hbase data in the mapper and reducer stage. Attached
> is
> > > the
> > > > code as well.
> > > >
> > > > For processing a 1M records, it is taking about 38 minutes. I am
> using
> > > > HTable.incrementColumnValue() in the reduce pass to create the
> records
> > in
> > > > the hbase tables.
> > > >
> > > > Is there anything I should be doing differently or inherently
> > incorrect?
> > > I
> > > > would like run this task in 1 minute.
> > > >
> > > > Thanks for the help,
> > > > Irfan
> > > >
> > > > Here is the output of the process. Let me know if I should attach any
> > > other
> > > > log.
> > > >
> > > > 09/07/02 15:19:11 INFO mapred.JobClient: Running job:
> > > job_200906192236_5114
> > > > 09/07/02 15:19:12 INFO mapred.JobClient: map 0% reduce 0%
> > > > 09/07/02 15:19:29 INFO mapred.JobClient: map 30% reduce 0%
> > > > 09/07/02 15:19:32 INFO mapred.JobClient: map 46% reduce 0%
> > > > 09/07/02 15:19:35 INFO mapred.JobClient: map 64% reduce 0%
> > > > 09/07/02 15:19:38 INFO mapred.JobClient: map 75% reduce 0%
> > > > 09/07/02 15:19:44 INFO mapred.JobClient: map 76% reduce 0%
> > > > 09/07/02 15:19:47 INFO mapred.JobClient: map 99% reduce 1%
> > > > 09/07/02 15:19:50 INFO mapred.JobClient: map 100% reduce 3%
> > > > 09/07/02 15:19:53 INFO mapred.JobClient: map 100% reduce 4%
> > > > 09/07/02 15:19:56 INFO mapred.JobClient: map 100% reduce 10%
> > > > 09/07/02 15:19:59 INFO mapred.JobClient: map 100% reduce 12%
> > > > 09/07/02 15:20:02 INFO mapred.JobClient: map 100% reduce 16%
> > > > 09/07/02 15:20:05 INFO mapred.JobClient: map 100% reduce 25%
> > > > 09/07/02 15:20:08 INFO mapred.JobClient: map 100% reduce 33%
> > > > 09/07/02 15:20:11 INFO mapred.JobClient: map 100% reduce 36%
> > > > 09/07/02 15:20:14 INFO mapred.JobClient: map 100% reduce 39%
> > > > 09/07/02 15:20:17 INFO mapred.JobClient: map 100% reduce 41%
> > > > 09/07/02 15:20:29 INFO mapred.JobClient: map 100% reduce 42%
> > > > 09/07/02 15:20:32 INFO mapred.JobClient: map 100% reduce 44%
> > > > 09/07/02 15:20:38 INFO mapred.JobClient: map 100% reduce 46%
> > > > 09/07/02 15:20:49 INFO mapred.JobClient: map 100% reduce 47%
> > > > 09/07/02 15:20:55 INFO mapred.JobClient: map 100% reduce 50%
> > > > 09/07/02 15:21:01 INFO mapred.JobClient: map 100% reduce 51%
> > > > 09/07/02 15:21:34 INFO mapred.JobClient: map 100% reduce 52%
> > > > 09/07/02 15:21:39 INFO mapred.JobClient: map 100% reduce 53%
> > > > 09/07/02 15:22:06 INFO mapred.JobClient: map 100% reduce 54%
> > > > 09/07/02 15:22:28 INFO mapred.JobClient: map 100% reduce 55%
> > > > 09/07/02 15:22:44 INFO mapred.JobClient: map 100% reduce 56%
> > > > 09/07/02 15:23:02 INFO mapred.JobClient: Task Id :
> > > > attempt_200906192236_5114_r_000002_0, Status : FAILED
> > > > attempt_200906192236_5114_r_000002_0: [2009-07-02 15:20:27.230]
> > fetching
> > > > new record writer ...
> > > > attempt_200906192236_5114_r_000002_0: [2009-07-02 15:22:51.429]
> failed
> > to
> > > > initialize the hbase configuration
> > > > 09/07/02 15:23:08 INFO mapred.JobClient: map 100% reduce 53%
> > > > 09/07/02 15:23:08 INFO mapred.JobClient: Task Id :
> > > > attempt_200906192236_5114_r_000013_0, Status : FAILED
> > > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > > trying
> > > > to locate root region
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > > at
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > > at
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > > at
> > > >
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
> > > > at
> > > >
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
> > > > at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > >
> > > > attempt_200906192236_5114_r_000013_0: [2009-07-02 15:20:33.183]
> > fetching
> > > > new record writer ...
> > > > attempt_200906192236_5114_r_000013_0: [2009-07-02 15:23:04.369]
> failed
> > to
> > > > initialize the hbase configuration
> > > > 09/07/02 15:23:09 INFO mapred.JobClient: map 100% reduce 50%
> > > > 09/07/02 15:23:14 INFO mapred.JobClient: Task Id :
> > > > attempt_200906192236_5114_r_000012_0, Status : FAILED
> > > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > > trying
> > > > to locate root region
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > > at
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > > at
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > > at
> > > >
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
> > > > at
> > > >
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
> > > > at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > >
> > > > attempt_200906192236_5114_r_000012_0: [2009-07-02 15:20:48.434]
> > fetching
> > > > new record writer ...
> > > > attempt_200906192236_5114_r_000012_0: [2009-07-02 15:23:10.185]
> failed
> > to
> > > > initialize the hbase configuration
> > > > 09/07/02 15:23:15 INFO mapred.JobClient: map 100% reduce 48%
> > > > 09/07/02 15:23:17 INFO mapred.JobClient: Task Id :
> > > > attempt_200906192236_5114_r_000014_0, Status : FAILED
> > > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out
> > > trying
> > > > to locate root region
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:863)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:514)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:523)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:496)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:628)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:527)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:490)
> > > > at
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:124)
> > > > at
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:107)
> > > > at
> > > >
> > >
> >
> com.qwapi.txnload.LoadMultipleCubes$CubeOutputFormat.getRecordWriter(LoadMultipleCubes.java:442)
> > > > at
> > > >
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:435)
> > > > at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > >
> > > > attempt_200906192236_5114_r_000014_0: [2009-07-02 15:20:47.442]
> > fetching
> > > > new record writer ...
> > > > attempt_200906192236_5114_r_000014_0: [2009-07-02 15:23:13.285]
> failed
> > to
> > > > initialize the hbase configuration
> > > > 09/07/02 15:23:18 INFO mapred.JobClient: map 100% reduce 45%
> > > > 09/07/02 15:23:21 INFO mapred.JobClient: map 100% reduce 46%
> > > > 09/07/02 15:23:29 INFO mapred.JobClient: map 100% reduce 47%
> > > > 09/07/02 15:23:32 INFO mapred.JobClient: map 100% reduce 48%
> > > > 09/07/02 15:23:36 INFO mapred.JobClient: map 100% reduce 49%
> > > > 09/07/02 15:23:39 INFO mapred.JobClient: map 100% reduce 51%
> > > > 09/07/02 15:23:42 INFO mapred.JobClient: map 100% reduce 56%
> > > > 09/07/02 15:23:45 INFO mapred.JobClient: map 100% reduce 58%
> > > > 09/07/02 15:24:20 INFO mapred.JobClient: map 100% reduce 59%
> > > > 09/07/02 15:25:11 INFO mapred.JobClient: map 100% reduce 60%
> > > > 09/07/02 15:25:17 INFO mapred.JobClient: map 100% reduce 61%
> > > > 09/07/02 15:25:26 INFO mapred.JobClient: map 100% reduce 62%
> > > > 09/07/02 15:25:32 INFO mapred.JobClient: map 100% reduce 64%
> > > > 09/07/02 15:25:38 INFO mapred.JobClient: map 100% reduce 65%
> > > > 09/07/02 15:26:20 INFO mapred.JobClient: map 100% reduce 66%
> > > > 09/07/02 15:26:40 INFO mapred.JobClient: map 100% reduce 67%
> > > > 09/07/02 15:26:48 INFO mapred.JobClient: map 100% reduce 68%
> > > > 09/07/02 15:27:16 INFO mapred.JobClient: map 100% reduce 69%
> > > > 09/07/02 15:27:21 INFO mapred.JobClient: map 100% reduce 70%
> > > > 09/07/02 15:27:46 INFO mapred.JobClient: map 100% reduce 71%
> > > > 09/07/02 15:28:25 INFO mapred.JobClient: map 100% reduce 72%
> > > > 09/07/02 15:28:46 INFO mapred.JobClient: map 100% reduce 73%
> > > > 09/07/02 15:29:08 INFO mapred.JobClient: map 100% reduce 74%
> > > > 09/07/02 15:29:45 INFO mapred.JobClient: map 100% reduce 76%
> > > > 09/07/02 15:30:42 INFO mapred.JobClient: map 100% reduce 77%
> > > > 09/07/02 15:31:06 INFO mapred.JobClient: map 100% reduce 78%
> > > > 09/07/02 15:31:12 INFO mapred.JobClient: map 100% reduce 79%
> > > > 09/07/02 15:31:36 INFO mapred.JobClient: map 100% reduce 81%
> > > > 09/07/02 15:31:37 INFO mapred.JobClient: map 100% reduce 82%
> > > > 09/07/02 15:32:00 INFO mapred.JobClient: map 100% reduce 83%
> > > > 09/07/02 15:32:09 INFO mapred.JobClient: map 100% reduce 84%
> > > > 09/07/02 15:32:30 INFO mapred.JobClient: map 100% reduce 86%
> > > > 09/07/02 15:38:42 INFO mapred.JobClient: map 100% reduce 88%
> > > > 09/07/02 15:39:49 INFO mapred.JobClient: map 100% reduce 89%
> > > > 09/07/02 15:41:13 INFO mapred.JobClient: map 100% reduce 90%
> > > > 09/07/02 15:41:16 INFO mapred.JobClient: map 100% reduce 91%
> > > > 09/07/02 15:41:28 INFO mapred.JobClient: map 100% reduce 93%
> > > > 09/07/02 15:44:34 INFO mapred.JobClient: map 100% reduce 94%
> > > > 09/07/02 15:45:41 INFO mapred.JobClient: map 100% reduce 95%
> > > > 09/07/02 15:45:50 INFO mapred.JobClient: map 100% reduce 96%
> > > > 09/07/02 15:46:17 INFO mapred.JobClient: map 100% reduce 98%
> > > > 09/07/02 15:55:29 INFO mapred.JobClient: map 100% reduce 99%
> > > > 09/07/02 15:57:08 INFO mapred.JobClient: map 100% reduce 100%
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Job complete:
> > > > job_200906192236_5114
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Counters: 18
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Job Counters
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Launched reduce tasks=24
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Rack-local map tasks=2
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Launched map tasks=20
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Data-local map tasks=18
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: FileSystemCounters
> > > > 09/07/02 15:57:14 INFO mapred.JobClient:
> FILE_BYTES_READ=1848609562
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: HDFS_BYTES_READ=57982980
> > > > 09/07/02 15:57:14 INFO mapred.JobClient:
> > > FILE_BYTES_WRITTEN=2768325646
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Map-Reduce Framework
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Reduce input groups=4863
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Combine output records=0
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Map input records=294786
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Reduce shuffle
> > > bytes=883803390
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Reduce output records=0
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Spilled Records=50956464
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Map output
> bytes=888797024
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Map input bytes=57966580
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Combine input records=0
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Map output
> > records=16985488
> > > > 09/07/02 15:57:14 INFO mapred.JobClient: Reduce input
> > > records=16985488
> > > >
> > > >
> > >
> >
>
