Oh no. I just realized this was all posted to hdfs, not hbase - I guess I really needed to sleep. Sorry everyone! Long weekend.
Best, -stu Sent from my Verizon Wireless BlackBerry -----Original Message----- From: stu24m...@yahoo.com Date: Wed, 7 Jul 2010 03:23:52 To: <hdfs-user@hadoop.apache.org> Reply-To: hdfs-user@hadoop.apache.org Subject: Re: Diagnosing can't find mapper errors Just to follow up: I've confirmed changing the eclipse jar export options results in map tasks that can run to completion. Take care, -stu Sent from my Verizon Wireless BlackBerry -----Original Message----- From: Stuart Smith <stu24m...@yahoo.com> Date: Tue, 6 Jul 2010 17:12:52 To: <hdfs-user@hadoop.apache.org> Reply-To: hdfs-user@hadoop.apache.org Subject: Re: Diagnosing can't find mapper errors Ok, I think I figured out this specific issue (while I still don't know how to improve the experience of "can't find mapper" in general). On Eclipse, when I said "export to jar", I checked: "Extract required libraries into generated jar" instead of: "Package required libraries into generated jar" I really have no idea what the difference is, TBH. The distinction between Extracting and Packaging seems rather cryptic (I did try this: http://www.google.com/search?client=ubuntu&channel=fs&q=eclipse+extracting+packaging+jar&ie=utf-8&oe=utf-8, but it was unhelpful). However, I say "think" because now my map/reduce is just stalled: it spun up 256 map jobs on my 4 node cluster, which just brought everything to a crawl. This seems odd, since most complaints seem to be in the opposite direction: not enough tasks due to the map task per region limitation (from what I can tell). I'll keep digging around, but any advice on optimizing M/R tasks over an hbase table is welcome.. Take care, -stu --- On Tue, 7/6/10, Stuart Smith <stu24m...@yahoo.com> wrote: > From: Stuart Smith <stu24m...@yahoo.com> > Subject: Diagnosing can't find mapper errors > To: hdfs-user@hadoop.apache.org > Date: Tuesday, July 6, 2010, 5:16 PM > Hello, > > It seems like everytime I'm writing a map class, I > make some random mistake that triggers a generically > unhelpful ClassNotFoundException that results in a large > amount of frustration from forgetting the static keyword or > something silly. > > Are there any lint tools to check your class beforehand? Is > there a way to make hadoop give useful error messages when > something goes wrong? To catch things at compile time? > > My current trouble is with a class like so: > > package hbaseScanner; > > ... > > public class Validator > { > ... > public static class ValidateMapper > extends TableMapper<Text,IntWritable> > { > > @Override > public void map( > ImmutableBytesWritable key, Result value, Context context ) > throws IOException, > InterruptedException > { > ... > public static class Reduce extends > TableReducer<Text,IntWritable,Text> > { > @Override > public void reduce( > Text key, Iterable<IntWritable> Values, Context > context ) > throws IOException, > InterruptedException > { > > ... > public static void main(String[] args) > throws Exception > { > Configuration > configuration = new Configuration(); > Job job = new > Job(configuration); > > //don't add anything, > and it will scan everything (according to docs) > Scan scan = new > Scan(); > scan.addColumn( > Bytes.toBytes("content"), Bytes.toBytes("file") ); > > > TableMapReduceUtil.initTableMapperJob("filestore", scan, > ValidateMapper.class, Text.class, IntWritable.class, job); > > TableMapReduceUtil.initTableReducerJob("statistics", > Reduce.class, job); > > > job.waitForCompletion(true); > } > > } > > Hadoop will only say: > us : FAILED > 10/07/06 14:09:54 INFO mapred.JobClient: Task Id : > attempt_201006211228_0010_m_000010_0, Status : FAILED > java.lang.RuntimeException: > java.lang.ClassNotFoundException: > hbaseScanner.Validator$Map > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809) > at > org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:157) > at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569) > > I'm running like so: > > ~/hadoop/bin$ ./hadoop jar ./ScannerValidator.jar > hbaseScanner.Validator > > *sigh*. So frustrating. > > > Take care, > -stu > > > > >
