Hi,
It looks like es-hadoop is not part of your classpath (hence the NCDFE). This might be either due to some
misconfiguration of your classpath or due to the way
the Configuration object is used. It looks like you are using it correctly though typically I use Job(Configuration)
instead of getInstance() (static factory methods
are always bad).
Potentially you can use other variants like LIBJARS or HADOOP_CLASSPATH env variables and make notice of what type of
separator you use between your jars (, vs : vs ;).
Try to debug the classpath and see what you get - see the jar that is created and uploaded to HDFS, turn on logging on
the hadoop side - potentially use distributed cache
Embedding the libraries under lib/ also works (see [2]).
All of the have pros and cons, the idea is to get your sample running and then
debug your env to see what's the issue.
Cheers,
[1]
http://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Example%3A+WordCount+v2.0
[2]
http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
On 12/14/14 3:32 PM, CAI Longqi wrote:
Hello, I’m using elasticsearch-hadoop-2.0.2.jar, and meet the problem:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/elasticsearch/hadoop/mr/EsOutputFormat
at com.clqb.app.ElasticSearch.run(ElasticSearch.java:46)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at com.clqb.app.ElasticSearch.main(ElasticSearch.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.hadoop.mr.EsOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 9 more
Here’s my program:
public class ElasticSearch extends Configured implements Tool {
public static class AwesomeMapper extends Mapper<LongWritable, Text,
NullWritable, MapWritable> {
@Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
context.write(NullWritable.get(),
XmlUtils.xmlTextToMapWritable(value)); // XmlUtils is not shown here
}
}
public static class AwesomeReducer extends Reducer<NullWritable, MapWritable,
NullWritable, NullWritable> {
}
public int run(String[] args) throws Exception {
Configuration conf = getConf();
conf.set("xmlinput.start", "<page>");
conf.set("xmlinput.end", "</page>");
conf.setBoolean("mapred.map.tasks.speculative.execution", false);
conf.setBoolean("mapred.reduce.tasks.speculative.execution", false);
conf.set("es.nodes", "localhost:9200");
conf.set("es.resource", "radio/artists");
Job job = Job.getInstance(conf);
job.setJarByClass(ElasticSearch.class);
job.setInputFormatClass(XmlInputFormat.class);
job.setOutputFormatClass(EsOutputFormat.class);
job.setMapOutputValueClass(MapWritable.class);
job.setMapperClass(AwesomeMapper.class);
job.setReducerClass(AwesomeReducer.class);
Path outputPath = new Path(args[1]);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, outputPath);
outputPath.getFileSystem(conf).delete(outputPath, true);
return job.waitForCompletion(true) ? 0 : 1;
}
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new ElasticSearch(), args);
System.exit(exitCode);
}
}
p.s. *I also make sure that I have included `elasticsearch-hadoop-2.0.2.jar` in
my `-libjars`*. Any suggestions?
Thanks
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/762794c8-0bd0-4c16-b1dd-9c914a29a710%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/762794c8-0bd0-4c16-b1dd-9c914a29a710%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/548DEE3C.3080609%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
