Hi Costin, Thanks for your inputs. I was able to get it running after I copied the elasticsearch-hadoop-2.0.0.jar to the "lib" directory of my hadoop installation. The reason why I was stuck with this issue is because i had already packaged this es-hadoop jar into my application and had built a jar. So when I was running the example as follows :-
hadoop jar es2.jar Es2 where Es2 is the name of the runner class which the main() function, I was expecting the program to find the required classes since I had already bundled the es-hadoop jar within the project jar. Also in the instructions on the elasticsearch-hadoop documentation at http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html#CO14-2 it was mentioned to add the jar to the HADOOP_CLASSPATH. I first added the path to the es-hadoop.jar to the HADOOP_CLASSPATH, but it gave the same error. Later I added the jar to one of paths mentioned within HADOOP_CLASSPATH, and the program executed. Can you guide as to why is it working in the second case and not the first case ? Or am I doing something wrong ? Anyway thanks for your guidance. Regards, Gaurav On Wed, Sep 10, 2014 at 1:54 AM, Costin Leau <[email protected]> wrote: > If by error you mean the ClassNotFoundException, you need to check again > your classpath. Also be sure to add es-hadoop to your job classpath > (typically pack it with the jar) - the documentation > describes some of the options available [1] > > [1] http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/ > 2.1.Beta/mapreduce.html#_installation > > > On 9/9/14 10:26 PM, gaurav redkar wrote: > >> Hi Costin, >> >> I had downloaded the elasticsearch-hadoop-2.1.0.Beta1.zip file and used >> all the jars from that for the program. Later I >> even tried replacing all the jars in my program with jars from with >> elasticsearch-hadoop-2.0.0.zip file, but still >> facing the same error. >> >> On Tue, Sep 9, 2014 at 6:52 PM, Costin Leau <[email protected] >> <mailto:[email protected]>> wrote: >> >> Most likely you have a classpath conflict caused by multiple versions >> of es-hadoop. Can you double check you only >> have one version (2.1.0.Beta1) available? >> Based on the error, I'm guessing you have some 1.3 Mx or the RC >> somewhere in there... >> >> On 9/9/14 4:06 PM, gaurav redkar wrote: >> >> Hi Costin, >> >> Thanks for the heads up regarding gist. I will try to follow the >> guidelines in the future. As for my program, I >> am using >> Elasticsearch Hadoop v2.1.0.Beta1 . I tried your suggestion and >> changed the output value class to >> LinkedMapWritable. but >> now I am getting the following error. >> >> https://gist.github.com/__gauravub/7d55bc6b10cb63935eb8 < >> https://gist.github.com/gauravub/7d55bc6b10cb63935eb8> >> >> Any idea why is this happening ? I even tried using the v2.0.0 of >> es-hadoop but am still getting the same error. >> >> On Tue, Sep 9, 2014 at 4:02 PM, Costin Leau < >> [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>__> >> wrote: >> >> Hi, >> >> What version of es-hadoop are you using? The problem stems >> from the difference in the types mentioned on your >> Mapper, namely the output value class: >> >> conf.setMapOutputValueClass(__ >> __MapWritable.class); >> >> >> to MapWritable while LinkedMapWritable is returned. The >> latest versions automatically detect this and use >> the proper >> type so I recommend upgrading. >> If that's not an option, use LinkedMapWritable. >> >> Cheers, >> >> P.S. Please don't post code and stracktraces on the mailing >> list since it highly reduces the readability of >> your >> email. Instead use gist or any other service >> to post the code as indicated in the docs [1]. Thanks >> >> [1] >> http://www.elasticsearch.org/____guide/en/elasticsearch/__ >> hadoop/__2.1.Beta/__troubleshooting.html#___where_ >> __do_i_post_my___information >> <http://www.elasticsearch.org/__guide/en/elasticsearch/ >> hadoop/__2.1.Beta/troubleshooting.html#___where_ >> do_i_post_my___information> >> >> <http://www.elasticsearch.org/__guide/en/elasticsearch/ >> hadoop/__2.1.Beta/troubleshooting.html#___where_ >> do_i_post_my___information >> <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/ >> 2.1.Beta/troubleshooting.html#_where_do_i_post_my_information>> >> >> >> >> >> On 9/9/14 11:59 AM, gaurav redkar wrote: >> >> Hi, I was following the example given on official >> elasticsearch documentation to read data from >> elasticsearch using >> hadoop but i am getting the following error. >> >> java.lang.Exception: java.io.IOException: Type mismatch >> in value from map: expected >> org.apache.hadoop.io <http://org.apache.hadoop.io> >> <http://org.apache.hadoop.io>.____MapWritable, >> received org.elasticsearch.hadoop.mr < >> http://org.elasticsearch.hadoop.mr> >> <http://org.elasticsearch.__hadoop.mr <http://org.elasticsearch. >> hadoop.mr>>.__LinkedMapWritable >> >> at org.apache.hadoop.mapred.____ >> LocalJobRunner$Job.runTasks(____LocalJobRunner.java:462) >> >> at org.apache.hadoop.mapred.____ >> LocalJobRunner$Job.run(____LocalJobRunner.java:522) >> >> Caused by: java.io.IOException: Type mismatch in value >> from map: expected org.apache.hadoop.io >> <http://org.apache.hadoop.io> >> <http://org.apache.hadoop.io>.____MapWritable, received >> org.elasticsearch.hadoop.mr <http://org.elasticsearch.hadoop.mr> >> <http://org.elasticsearch.__hadoop.mr >> <http://org.elasticsearch.hadoop.mr>>.__LinkedMapWritable >> >> at org.apache.hadoop.mapred.____ >> MapTask$MapOutputBuffer.____collect(MapTask.java:1054) >> >> at org.apache.hadoop.mapred.____ >> MapTask$OldOutputCollector.____collect(MapTask.java:587) >> >> at es2.mapper1.map(mapper1.java:____29) >> >> at org.apache.hadoop.mapred.____ >> MapRunner.run(MapRunner.java:____54) >> >> at org.apache.hadoop.mapred.____ >> MapTask.runOldMapper(MapTask.____java:430) >> >> at org.apache.hadoop.mapred.____ >> MapTask.run(MapTask.java:342) >> >> at org.apache.hadoop.mapred.____LocalJobRunner$Job$____ >> MapTaskRunnable.run(____LocalJobRunner.java:243) >> >> at java.util.concurrent.____ >> Executors$RunnableAdapter.____call(Executors.java:471) >> >> at java.util.concurrent.____ >> FutureTask$Sync.innerRun(____FutureTask.java:334) >> >> at java.util.concurrent.____ >> FutureTask.run(FutureTask.____java:166) >> >> at java.util.concurrent.____ >> ThreadPoolExecutor.runWorker(____ThreadPoolExecutor.java:1145) >> >> at java.util.concurrent.____ >> ThreadPoolExecutor$Worker.run(____ThreadPoolExecutor.java:615) >> >> at java.lang.Thread.run(Thread.____java:724) >> >> 14/09/09 13:25:51 INFO mapreduce.Job: Job >> job_local994189535_0001 failed with state FAILED due to: NA >> >> 14/09/09 13:25:51 INFO mapreduce.Job: Counters: 0 >> >> Exception in thread "main" java.io.IOException: Job >> failed! >> >> at org.apache.hadoop.mapred.____ >> JobClient.runJob(JobClient.____java:836) >> >> at es2.Es2.main(Es2.java:48) >> >> at >> sun.reflect.____NativeMethodAccessorImpl.____invoke0(Native >> Method) >> >> at sun.reflect.____NativeMethodAccessorImpl.____ >> invoke(____NativeMethodAccessorImpl.java:____57) >> >> at sun.reflect.____DelegatingMethodAccessorImpl._ >> ___invoke(____DelegatingMethodAccessorImpl.____java:43) >> >> at java.lang.reflect.Method.____invoke(Method.java:606) >> >> at org.apache.hadoop.util.RunJar. >> ____main(RunJar.java:212) >> >> 14/09/09 13:25:51 INFO mapred.LocalJobRunner: >> ShardInputSplit >> [node=[QMYbfSRMTsyXVVgfFweMkA/ >> ____Conceptnet|192.168.1.202:__9200 <http://192.168.1.202:9200> >> <http://192.168.1.202:9200>]____,shard=0] > map >> >> >> The code for my main runner class is as follows:- >> >> >> public class Es2 { >> >> static private final Path TMP_DIR = new >> Path(Es2.class.getSimpleName() >> >> + "_TMP_1"); >> >> >> /** >> >> * @param args the command line arguments >> >> */ >> >> public static void main(String[] args) throws >> IOException{ >> >> //#String node="192.168.1.202"; >> >> JobConf conf = new JobConf(); >> >> conf.set("es.resource", "data/content"); >> >> conf.set("es.nodes", "192.168.1.202:9200 < >> http://192.168.1.202:9200> <http://192.168.1.202:9200>"); >> >> conf.set("es.query", "?q=s_start:java"); >> >> conf.setInputFormat(____EsInputFormat.class); >> >> conf.setMapOutputKeyClass(____Text.class); >> >> conf.setMapOutputValueClass(__ >> __MapWritable.class); >> >> conf.setOutputKeyClass(Text.____class); >> >> conf.setOutputValueClass(____MapWritable.class); >> >> conf.setOutputFormat(____TextOutputFormat.class); >> >> conf.setMapperClass(mapper1.____class); >> >> final Path outDir = new Path(TMP_DIR, "out"); >> >> FileOutputFormat.____setOutputPath(conf, outDir); >> >> JobClient.runJob(conf); >> >> } >> >> } >> >> The code for my mapper class is as follows:- >> >> public class mapper1 extends MapReduceBase implements >> >> Mapper{ >> >> @Override >> >> public void map(Object key, Object value, >> OutputCollector output, >> >> Reporter reporter) throws >> IOException { >> >> Text docId = (Text) key; >> >> MapWritable doc = (MapWritable) value; >> >> output.collect(docId,doc); >> >> } >> >> } >> >> Kindly guide me on this issue. >> >> Thanks >> >> -- >> You received this message because you are subscribed to >> the Google Groups "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails >> from it, send an email to >> elasticsearch+unsubscribe@__go__oglegroups.com < >> http://googlegroups.com> >> <mailto:elasticsearch%[email protected] <mailto: >> elasticsearch%[email protected]>__> >> <mailto:[email protected] >> <mailto:elasticsearch%[email protected]> <mailto: >> elasticsearch%[email protected] >> <mailto:elasticsearch%[email protected]>__>>. >> To view this discussion on the web visit >> https://groups.google.com/d/____msgid/elasticsearch/ >> 3b110c43-____faa8-4a08-acfa-0fac5fd89585%____40googlegroups.com >> <https://groups.google.com/d/__msgid/elasticsearch/3b110c43- >> __faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com> >> >> <https://groups.google.com/d/__msgid/elasticsearch/3b110c43- >> __faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com >> <https://groups.google.com/d/msgid/elasticsearch/3b110c43- >> faa8-4a08-acfa-0fac5fd89585%40googlegroups.com>> >> >> <https://groups.google.com/d/____msgid/elasticsearch/ >> 3b110c43-____faa8-4a08-acfa-0fac5fd89585%____40googlegroups.com?utm___ >> medium=__email&utm_source=__footer >> <https://groups.google.com/d/__msgid/elasticsearch/3b110c43- >> __faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com?utm_ >> medium=__email&utm_source=footer> >> >> <https://groups.google.com/d/__msgid/elasticsearch/3b110c43- >> __faa8-4a08-acfa-0fac5fd89585%__40googlegroups.com?utm_ >> medium=__email&utm_source=footer >> <https://groups.google.com/d/msgid/elasticsearch/3b110c43- >> faa8-4a08-acfa-0fac5fd89585%40googlegroups.com?utm_medium= >> email&utm_source=footer>>>. >> For more options, visit https://groups.google.com/d/__ >> __optout <https://groups.google.com/d/__optout> >> <https://groups.google.com/d/__optout < >> https://groups.google.com/d/optout>>. >> >> >> -- >> Costin >> >> -- >> You received this message because you are subscribed to a >> topic in the Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/____topic/elasticsearch/HS1A____ >> _psM7fw/unsubscribe >> <https://groups.google.com/d/__topic/elasticsearch/HS1A___ >> psM7fw/unsubscribe> >> <https://groups.google.com/d/__topic/elasticsearch/HS1A___ >> psM7fw/unsubscribe >> <https://groups.google.com/d/topic/elasticsearch/HS1A_ >> psM7fw/unsubscribe>>. >> To unsubscribe from this group and all its topics, send an >> email to >> elasticsearch+unsubscribe@__go__oglegroups.com < >> http://googlegroups.com> >> <mailto:elasticsearch%[email protected] >> <mailto:elasticsearch%[email protected]>__>. >> To view this discussion on the web visit >> https://groups.google.com/d/____msgid/elasticsearch/ >> 540ED720.____5040501%40gmail.com >> <https://groups.google.com/d/__msgid/elasticsearch/540ED720. >> __5040501%40gmail.com> >> <https://groups.google.com/d/__msgid/elasticsearch/540ED720. >> __5040501%40gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/540ED720. >> 5040501%40gmail.com>>. >> >> For more options, visit https://groups.google.com/d/__ >> __optout <https://groups.google.com/d/__optout> >> <https://groups.google.com/d/__optout < >> https://groups.google.com/d/optout>>. >> >> >> -- >> You received this message because you are subscribed to the >> Google Groups "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, >> send an email to >> elasticsearch+unsubscribe@__googlegroups.com <mailto: >> elasticsearch%[email protected]> >> <mailto:[email protected] <mailto: >> elasticsearch%[email protected]>>. >> To view this discussion on the web visit >> https://groups.google.com/d/__msgid/elasticsearch/CAJ1vTBJ%_ >> _2BgBi6oUwbfxJSrvjXWpu___FzhJ4T3BbSQ%2BNA-iE_NtHA%__40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBJ% >> 2BgBi6oUwbfxJSrvjXWpu_FzhJ4T3BbSQ%2BNA-iE_NtHA%40mail.gmail.com> >> <https://groups.google.com/d/__msgid/elasticsearch/CAJ1vTBJ% >> __2BgBi6oUwbfxJSrvjXWpu___FzhJ4T3BbSQ%2BNA-iE_NtHA%__ >> 40mail.gmail.com?utm_medium=__email&utm_source=footer >> <https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBJ% >> 2BgBi6oUwbfxJSrvjXWpu_FzhJ4T3BbSQ%2BNA-iE_NtHA% >> 40mail.gmail.com?utm_medium=email&utm_source=footer>>. >> For more options, visit https://groups.google.com/d/__optout < >> https://groups.google.com/d/optout>. >> >> -- >> Costin >> >> -- >> You received this message because you are subscribed to a topic in >> the Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/__ >> topic/elasticsearch/HS1A___psM7fw/unsubscribe >> <https://groups.google.com/d/topic/elasticsearch/HS1A_ >> psM7fw/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> elasticsearch+unsubscribe@__googlegroups.com >> <mailto:elasticsearch%[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/__msgid/elasticsearch/540EFF05._ >> _6040508%40gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/540EFF05. >> 6040508%40gmail.com>. >> >> For more options, visit https://groups.google.com/d/__optout < >> https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to >> [email protected] <mailto:elasticsearch+ >> [email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBKTUr11ck6rSvbzZuf% >> 2Bt1C_tFMiZP407kvrDK4Yfd5P%3DQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/ >> CAJ1vTBKTUr11ck6rSvbzZuf%2Bt1C_tFMiZP407kvrDK4Yfd5P% >> 3DQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > Costin > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/elasticsearch/HS1A_psM7fw/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elasticsearch/540F61E9.2030303%40gmail.com. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJ1vTBJze%2BfmPRz52i03pmAT1F29CPA_mUzLtfZKjN2mkaSf1Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
