Do you put the hadoop conf on classpath ? It seems you are still using local file system but conncect Hadoop's JobTracker. Make sure you set the correct configuration in core-site.xml hdfs-site.xml, mapred-site.xml, and put them on classpath.
On Thu, Aug 26, 2010 at 5:32 PM, rahul <rmalv...@apple.com> wrote: > Hi , > > I am trying to integrate Pig with Hadoop for processing of jobs. > > I am able to run Pig in local mode and Hadoop with streaming api perfectly. > > But when I try to run Pig with Hadoop I get follwong Error: > > Pig Stack Trace > --------------- > ERROR 2116: Unexpected error. Could not validate the output specification > for: file:///Users/rahulmalviya/Documents/Pig/dev/main_merged_hdp_out > > org.apache.pig.impl.plan.PlanValidationException: ERROR 0: An unexpected > exception caused the validation to stop > at > org.apache.pig.impl.plan.PlanValidator.validate(PlanValidator.java:56) > at > org.apache.pig.impl.logicalLayer.validators.InputOutputFileValidator.validate(InputOutputFileValidator.java:49) > at > org.apache.pig.impl.logicalLayer.validators.InputOutputFileValidator.validate(InputOutputFileValidator.java:37) > at > org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:89) > at org.apache.pig.PigServer.validate(PigServer.java:930) > at org.apache.pig.PigServer.compileLp(PigServer.java:910) > at org.apache.pig.PigServer.compileLp(PigServer.java:871) > at org.apache.pig.PigServer.compileLp(PigServer.java:852) > at org.apache.pig.PigServer.execute(PigServer.java:816) > at org.apache.pig.PigServer.access$100(PigServer.java:105) > at org.apache.pig.PigServer$Graph.execute(PigServer.java:1080) > at org.apache.pig.PigServer.executeBatch(PigServer.java:288) > at > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:109) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) > at org.apache.pig.Main.main(Main.java:391) > Caused by: org.apache.pig.impl.plan.PlanValidationException: ERROR 2116: > Unexpected error. Could not validate the output specification for: > file:///Users/rahulmalviya/Documents/Pig/dev/main_merged_hdp_out > at > org.apache.pig.impl.logicalLayer.validators.InputOutputFileVisitor.visit(InputOutputFileVisitor.java:93) > at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:140) > at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:37) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) > at > org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50) > at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51) > at > org.apache.pig.impl.plan.PlanValidator.validate(PlanValidator.java:50) > ... 16 more > Caused by: java.io.IOException: Call to localhost/127.0.0.1:9001 failed on > local exception: java.io.EOFException > at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) > at org.apache.hadoop.ipc.Client.call(Client.java:743) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) > at > org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429) > at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423) > at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410) > at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50) > at > org.apache.pig.impl.logicalLayer.validators.InputOutputFileVisitor.visit(InputOutputFileVisitor.java:89) > ... 24 more > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > ================================================================================ > > Did anyone got the same error. I think it related to connection between pig > and hadoop. > > Can someone tell me how to connect Pig and hadoop. > > Thanks. > -- Best Regards Jeff Zhang
