On Wed, Oct 21, 2009 at 8:22 AM, Mark Vigeant <[email protected]>wrote:
> Ok, so first in response to St. Ack, nothing fishy appears to be happening > in the logs: data is being written to all regionservesrs. > > And it's not hovering around 100% done, it just has sent about 118 map > jobs, or "Task attempts" > > I saw this in your first posting: 10/21/09 10:22:52 INFO mapred.JobClient: map 100% reduce 0%. Is your job writing hbase in the map task or in reducer? Are you using TableOutputFormat? > I'm using Hadoop 0.20.1 and HBase 0.20.0 > > Each node is a virtual machine with 2 CPU, 4 GB host memory and 100 GB > storage. > > You are running DN, TT, HBase, and ZK on above? One disk shared by all? > I don't know what you meant by slots per TT... > Children running at any one time on a TaskTracker. You should start with one only since you have such an anemic platform. > > And the heapsize is the default of 1000 MB. That is probably a huge > problem, now that I think about it, heh. > > And there is absolutely no special configuration that I'm using. I have > Hbase running my zookeeper quorum on 2 machines, but that's about it. > You've upped filedescriptors and xceivers, all the stuff in 'Getting Started'? St>Ack > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of > Jean-Daniel Cryans > Sent: Wednesday, October 21, 2009 11:04 AM > To: [email protected] > Subject: Re: Table Upload Optimization > > Well the XMLStreamingInputFormat lets you map XML files which is neat > but it has a problem and always needs to be patched. I wondered if > that was missing but in your case it's not the problem. > > Did you check the logs of the master and region servers? Also I'd like to > know > > - Version of Hadoop and HBase > - Nodes's hardware > - How many map slots per TT > - HBASE_HEAPSIZE from conf/hbase-env.sh > - Special configuration you use > > Thx, > > J-D > > On Wed, Oct 21, 2009 at 7:57 AM, Mark Vigeant > <[email protected]> wrote: > > No. Should I? > > > > -----Original Message----- > > From: [email protected] [mailto:[email protected]] On Behalf Of > Jean-Daniel Cryans > > Sent: Wednesday, October 21, 2009 10:55 AM > > To: [email protected] > > Subject: Re: Table Upload Optimization > > > > Are you using the Hadoop Streaming API? > > > > J-D > > > > On Wed, Oct 21, 2009 at 7:52 AM, Mark Vigeant > > <[email protected]> wrote: > >> Hey > >> > >> So I want to upload a lot of XML data into an HTable. I have a class > that successfully maps up to about 500 MB of data or so (on one > regionserver) into a table, but if I go for much bigger than that it takes > forever and eventually just stops. I tried uploading a big XML file into my > 4 regionserver cluster (about 7 GB) and it's been a day and it's still going > at it. > >> > >> What I get when I run the job on the 4 node cluster is: > >> 10/21/09 10:22:35 INFO mapred.LocalJobRunner: > >> 10/21/09 10:22:38 INFO mapred.LocalJobRunner: > >> (then it does that for a while until...) > >> 10/21/09 10:22:52 INFO mapred.TaskRunner: Task > attempt_local_0001_m_000117_0 is done. And is in the process of committing > >> 10/21/09 10:22:52 INFO mapred.LocalJobRunner: > >> 10/21/09 10:22:52 mapred.TaskRunner: Task > 'attempt_local_0001_m_000117_0' is done. > >> 10/21/09 10:22:52 INFO mapred.JobClient: map 100% reduce 0% > >> 10/21/09 10:22:58 INFO mapred.LocalJobRunner: > >> 10/21/09 10:22:59 INFO mapred.JobClient: map 99% reduce 0% > >> > >> > >> I'm convinced I'm not configuring hbase or hadoop correctly. Any > suggestions? > >> > >> Mark Vigeant > >> RiskMetrics Group, Inc. > >> > > >
