Ok, I had some problems with configuration and host resolution and I fixed them. I was able to run successfully the simple wordcount example, but I failed running wordcount-nopipe.cc. This is the stack trace:
Error: java.io.IOException: pipe child exception at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:226) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:102) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:125) Any suggestions? 2013/12/17 Silvina Caíno Lores <[email protected]> > I'm having similar problems with pipes, mostly because of issues with the > native shared libraries that leave the job stuck either at 0%-0% or before > launch (because the resource manager gets stuck as well and crashes). > > I found that out by looking at the stderr logs by the way. > > Let us know if you progress, good luck! > > Best, > Silvina > > > On 16 December 2013 19:32, Vinod Kumar Vavilapalli < > [email protected]> wrote: > >> You should navigate to the ResourceManager UI following the link and see >> what is happening on the ResourceManager as well as the application-master. >> Check if any nodes are active first. Then look at ResourceManager and >> NodeManager logs. >> >> +Vinod >> >> On Dec 16, 2013, at 10:29 AM, Mauro Del Rio <[email protected]> wrote: >> >> I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a >> simple wordcount in c++ with pipes, this time there was no exception, but >> the job didn't finish. This is the output on the shell where I launched >> pipes: >> >> mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc >> -input test.sh -output out >> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / >> 0.0.0.0:9052 >> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / >> 0.0.0.0:9052 >> 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set. User >> classes may not be found. See Job or Job#setJar(String). >> 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to >> process : 1 >> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2 >> 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is >> deprecated. Instead, use mapreduce.job.user.name >> 13/12/16 18:51:41 INFO Configuration.deprecation: >> mapred.cache.files.filesizes is deprecated. Instead, use >> mapreduce.job.cache.files.filesizes >> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is >> deprecated. Instead, use mapreduce.job.cache.files >> 13/12/16 18:51:41 INFO Configuration.deprecation: >> mapred.pipes.user.inputformat is deprecated. Instead, use >> mapreduce.pipes.inputformat >> 13/12/16 18:51:41 INFO Configuration.deprecation: >> mapred.output.value.class is deprecated. Instead, use >> mapreduce.job.output.value.class >> 13/12/16 18:51:41 INFO Configuration.deprecation: >> mapred.mapoutput.value.class is deprecated. Instead, use >> mapreduce.map.output.value.class >> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is >> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir >> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is >> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir >> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is >> deprecated. Instead, use mapreduce.job.maps >> 13/12/16 18:51:41 INFO Configuration.deprecation: >> hadoop.pipes.partitioner is deprecated. Instead, use >> mapreduce.pipes.partitioner >> 13/12/16 18:51:41 INFO Configuration.deprecation: hadoop.pipes.executable >> is deprecated. Instead, use mapreduce.pipes.executable >> 13/12/16 18:51:41 INFO Configuration.deprecation: >> mapred.cache.files.timestamps is deprecated. Instead, use >> mapreduce.job.cache.files.timestamps >> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.key.class >> is deprecated. Instead, use mapreduce.job.output.key.class >> 13/12/16 18:51:41 INFO Configuration.deprecation: >> mapred.mapoutput.key.class is deprecated. Instead, use >> mapreduce.map.output.key.class >> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.working.dir is >> deprecated. Instead, use mapreduce.job.working.dir >> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: >> job_1387213974967_0003 >> 13/12/16 18:51:42 INFO mapred.YARNRunner: Job jar is not present. Not >> adding any jar to the list of resources. >> 13/12/16 18:51:42 INFO impl.YarnClientImpl: Submitted application >> application_1387213974967_0003 to ResourceManager at /0.0.0.0:9052 >> 13/12/16 18:51:42 INFO mapreduce.Job: The url to track the job: >> http://mauro-VirtualBox:8088/proxy/application_1387213974967_0003/<http://mauro-virtualbox:8088/proxy/application_1387213974967_0003/> >> 13/12/16 18:51:42 INFO mapreduce.Job: Running job: job_1387213974967_0003 >> >> >> The job status from bin/mapred job -list is PREP. I didn't find any >> interesting information in logs file. >> >> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > > -- Mauro
