If by error you mean the ClassNotFoundException, you need to check again your classpath. Also be sure to add es-hadoop
to your job classpath (typically pack it with the jar) - the documentation
describes some of the options available [1]
[1]
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/2.1.Beta/mapreduce.html#_installation
On 9/9/14 10:26 PM, gaurav redkar wrote:
Hi Costin,
I had downloaded the elasticsearch-hadoop-2.1.0.Beta1.zip file and used all the
jars from that for the program. Later I
even tried replacing all the jars in my program with jars from with
elasticsearch-hadoop-2.0.0.zip file, but still
facing the same error.
On Tue, Sep 9, 2014 at 6:52 PM, Costin Leau <[email protected]
<mailto:[email protected]>> wrote:
Most likely you have a classpath conflict caused by multiple versions of
es-hadoop. Can you double check you only
have one version (2.1.0.Beta1) available?
Based on the error, I'm guessing you have some 1.3 Mx or the RC somewhere
in there...
On 9/9/14 4:06 PM, gaurav redkar wrote:
Hi Costin,
Thanks for the heads up regarding gist. I will try to follow the
guidelines in the future. As for my program, I
am using
Elasticsearch Hadoop v2.1.0.Beta1 . I tried your suggestion and changed
the output value class to
LinkedMapWritable. but
now I am getting the following error.
https://gist.github.com/__gauravub/7d55bc6b10cb63935eb8
<https://gist.github.com/gauravub/7d55bc6b10cb63935eb8>
Any idea why is this happening ? I even tried using the v2.0.0 of
es-hadoop but am still getting the same error.
On Tue, Sep 9, 2014 at 4:02 PM, Costin Leau <[email protected]
<mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>__> wrote:
Hi,
What version of es-hadoop are you using? The problem stems from
the difference in the types mentioned on your
Mapper, namely the output value class:
conf.setMapOutputValueClass(____MapWritable.class);
to MapWritable while LinkedMapWritable is returned. The latest
versions automatically detect this and use
the proper
type so I recommend upgrading.
If that's not an option, use LinkedMapWritable.
Cheers,
P.S. Please don't post code and stracktraces on the mailing list
since it highly reduces the readability of
your
email. Instead use gist or any other service
to post the code as indicated in the docs [1]. Thanks
[1]
http://www.elasticsearch.org/____guide/en/elasticsearch/__hadoop/__2.1.Beta/__troubleshooting.html#___where___do_i_post_my___information
<http://www.elasticsearch.org/__guide/en/elasticsearch/hadoop/__2.1.Beta/troubleshooting.html#___where_do_i_post_my___information>
<http://www.elasticsearch.org/__guide/en/elasticsearch/hadoop/__2.1.Beta/troubleshooting.html#___where_do_i_post_my___information
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/2.1.Beta/troubleshooting.html#_where_do_i_post_my_information>>
On 9/9/14 11:59 AM, gaurav redkar wrote:
Hi, I was following the example given on official
elasticsearch documentation to read data from
elasticsearch using
hadoop but i am getting the following error.
java.lang.Exception: java.io.IOException: Type mismatch in
value from map: expected
org.apache.hadoop.io <http://org.apache.hadoop.io>
<http://org.apache.hadoop.io>.____MapWritable,
received org.elasticsearch.hadoop.mr
<http://org.elasticsearch.hadoop.mr>
<http://org.elasticsearch.__hadoop.mr
<http://org.elasticsearch.hadoop.mr>>.__LinkedMapWritable
at
org.apache.hadoop.mapred.____LocalJobRunner$Job.runTasks(____LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.____LocalJobRunner$Job.run(____LocalJobRunner.java:522)
Caused by: java.io.IOException: Type mismatch in value from
map: expected org.apache.hadoop.io
<http://org.apache.hadoop.io>
<http://org.apache.hadoop.io>.____MapWritable, received
org.elasticsearch.hadoop.mr <http://org.elasticsearch.hadoop.mr>
<http://org.elasticsearch.__hadoop.mr
<http://org.elasticsearch.hadoop.mr>>.__LinkedMapWritable
at
org.apache.hadoop.mapred.____MapTask$MapOutputBuffer.____collect(MapTask.java:1054)
at
org.apache.hadoop.mapred.____MapTask$OldOutputCollector.____collect(MapTask.java:587)
at es2.mapper1.map(mapper1.java:____29)
at
org.apache.hadoop.mapred.____MapRunner.run(MapRunner.java:____54)
at
org.apache.hadoop.mapred.____MapTask.runOldMapper(MapTask.____java:430)
at org.apache.hadoop.mapred.____MapTask.run(MapTask.java:342)
at
org.apache.hadoop.mapred.____LocalJobRunner$Job$____MapTaskRunnable.run(____LocalJobRunner.java:243)
at
java.util.concurrent.____Executors$RunnableAdapter.____call(Executors.java:471)
at
java.util.concurrent.____FutureTask$Sync.innerRun(____FutureTask.java:334)
at
java.util.concurrent.____FutureTask.run(FutureTask.____java:166)
at
java.util.concurrent.____ThreadPoolExecutor.runWorker(____ThreadPoolExecutor.java:1145)
at
java.util.concurrent.____ThreadPoolExecutor$Worker.run(____ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.____java:724)
14/09/09 13:25:51 INFO mapreduce.Job: Job
job_local994189535_0001 failed with state FAILED due to: NA
14/09/09 13:25:51 INFO mapreduce.Job: Counters: 0
Exception in thread "main" java.io.IOException: Job failed!
at
org.apache.hadoop.mapred.____JobClient.runJob(JobClient.____java:836)
at es2.Es2.main(Es2.java:48)
at sun.reflect.____NativeMethodAccessorImpl.____invoke0(Native
Method)
at
sun.reflect.____NativeMethodAccessorImpl.____invoke(____NativeMethodAccessorImpl.java:____57)
at
sun.reflect.____DelegatingMethodAccessorImpl.____invoke(____DelegatingMethodAccessorImpl.____java:43)
at java.lang.reflect.Method.____invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.____main(RunJar.java:212)
14/09/09 13:25:51 INFO mapred.LocalJobRunner: ShardInputSplit
[node=[QMYbfSRMTsyXVVgfFweMkA/____Conceptnet|192.168.1.202:__9200
<http://192.168.1.202:9200>
<http://192.168.1.202:9200>]____,shard=0] > map
The code for my main runner class is as follows:-
public class Es2 {
static private final Path TMP_DIR = new
Path(Es2.class.getSimpleName()
+ "_TMP_1");
/**
* @param args the command line arguments
*/
public static void main(String[] args) throws
IOException{
//#String node="192.168.1.202";
JobConf conf = new JobConf();
conf.set("es.resource", "data/content");
conf.set("es.nodes", "192.168.1.202:9200
<http://192.168.1.202:9200> <http://192.168.1.202:9200>");
conf.set("es.query", "?q=s_start:java");
conf.setInputFormat(____EsInputFormat.class);
conf.setMapOutputKeyClass(____Text.class);
conf.setMapOutputValueClass(____MapWritable.class);
conf.setOutputKeyClass(Text.____class);
conf.setOutputValueClass(____MapWritable.class);
conf.setOutputFormat(____TextOutputFormat.class);
conf.setMapperClass(mapper1.____class);
final Path outDir = new Path(TMP_DIR, "out");
FileOutputFormat.____setOutputPath(conf, outDir);
JobClient.runJob(conf);
}
}
The code for my mapper class is as follows:-
public class mapper1 extends MapReduceBase implements
Mapper{
@Override
public void map(Object key, Object value, OutputCollector
output,
Reporter reporter) throws IOException {
Text docId = (Text) key;
MapWritable doc = (MapWritable) value;
output.collect(docId,doc);
}
}
Kindly guide me on this issue.
Thanks
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to
elasticsearch+unsubscribe@__go__oglegroups.com
<http://googlegroups.com>
<mailto:elasticsearch%[email protected]
<mailto:elasticsearch%[email protected]>__>
<mailto:[email protected]
<mailto:elasticsearch%[email protected]>
<mailto:elasticsearch%[email protected]
<mailto:elasticsearch%[email protected]>__>>.
To view this discussion on the web visit
https://groups.google.com/d/____msgid/elasticsearch/3b110c43-____faa8-4a08-acfa-0fac5fd89585%____40googlegroups.com
<https://groups.google.com/d/__msgid/elasticsearch/3b110c43-__faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com>
<https://groups.google.com/d/__msgid/elasticsearch/3b110c43-__faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/3b110c43-faa8-4a08-acfa-0fac5fd89585%40googlegroups.com>>
<https://groups.google.com/d/____msgid/elasticsearch/3b110c43-____faa8-4a08-acfa-0fac5fd89585%____40googlegroups.com?utm___medium=__email&utm_source=__footer
<https://groups.google.com/d/__msgid/elasticsearch/3b110c43-__faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com?utm_medium=__email&utm_source=footer>
<https://groups.google.com/d/__msgid/elasticsearch/3b110c43-__faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com?utm_medium=__email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/3b110c43-faa8-4a08-acfa-0fac5fd89585%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
For more options, visit https://groups.google.com/d/____optout
<https://groups.google.com/d/__optout>
<https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>>.
--
Costin
--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/____topic/elasticsearch/HS1A_____psM7fw/unsubscribe
<https://groups.google.com/d/__topic/elasticsearch/HS1A___psM7fw/unsubscribe>
<https://groups.google.com/d/__topic/elasticsearch/HS1A___psM7fw/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/HS1A_psM7fw/unsubscribe>>.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@__go__oglegroups.com <http://googlegroups.com>
<mailto:elasticsearch%[email protected]
<mailto:elasticsearch%[email protected]>__>.
To view this discussion on the web visit
https://groups.google.com/d/____msgid/elasticsearch/540ED720.____5040501%40gmail.com
<https://groups.google.com/d/__msgid/elasticsearch/540ED720.__5040501%40gmail.com>
<https://groups.google.com/d/__msgid/elasticsearch/540ED720.__5040501%40gmail.com
<https://groups.google.com/d/msgid/elasticsearch/540ED720.5040501%40gmail.com>>.
For more options, visit https://groups.google.com/d/____optout
<https://groups.google.com/d/__optout>
<https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>>.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearch+unsubscribe@__googlegroups.com
<mailto:elasticsearch%[email protected]>
<mailto:[email protected]
<mailto:elasticsearch%[email protected]>>.
To view this discussion on the web visit
https://groups.google.com/d/__msgid/elasticsearch/CAJ1vTBJ%__2BgBi6oUwbfxJSrvjXWpu___FzhJ4T3BbSQ%2BNA-iE_NtHA%__40mail.gmail.com
<https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBJ%2BgBi6oUwbfxJSrvjXWpu_FzhJ4T3BbSQ%2BNA-iE_NtHA%40mail.gmail.com>
<https://groups.google.com/d/__msgid/elasticsearch/CAJ1vTBJ%__2BgBi6oUwbfxJSrvjXWpu___FzhJ4T3BbSQ%2BNA-iE_NtHA%__40mail.gmail.com?utm_medium=__email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBJ%2BgBi6oUwbfxJSrvjXWpu_FzhJ4T3BbSQ%2BNA-iE_NtHA%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__topic/elasticsearch/HS1A___psM7fw/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/HS1A_psM7fw/unsubscribe>.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@__googlegroups.com
<mailto:elasticsearch%[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/__msgid/elasticsearch/540EFF05.__6040508%40gmail.com
<https://groups.google.com/d/msgid/elasticsearch/540EFF05.6040508%40gmail.com>.
For more options, visit https://groups.google.com/d/__optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBKTUr11ck6rSvbzZuf%2Bt1C_tFMiZP407kvrDK4Yfd5P%3DQ%40mail.gmail.com
<https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBKTUr11ck6rSvbzZuf%2Bt1C_tFMiZP407kvrDK4Yfd5P%3DQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/540F61E9.2030303%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
