Hi Santhosh I tried with absolute path as well but error remains the same. I think absolute path should not be a issue as both Pig and Hadoop are at the same location.
Please let me know if there is some gap in my understanding ? Thanks, Rahul On Aug 26, 2010, at 6:25 PM, Santhosh Srinivasan wrote: > Can you try replacing localhost with the fully qualified name of your host? > > Santhosh > > > -----Original Message----- > From: rahul [mailto:rmalv...@apple.com] > Sent: Thursday, August 26, 2010 6:22 PM > To: pig-user@hadoop.apache.org > Subject: Re: Pig and Hadoop Integration Error > > Hi Jeff, > > I have set the hadoop conf in class path by setting $HADOOP_CONF_DIR variable. > > But I have both Pig and hadoop running at the same machine, so localhost > should not make a difference. > > So I have used all the default config setting for the core-site.xml, > hdfs-site.xml, mapred-site.xml, as per the hadoop tutorial. > > Please let me know if my understanding is correct ? > > I am attaching the conf files as well : > hdfs-site.xml: > > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- Put site-specific property overrides in this file. --> > > <configuration> > <property> > <name>fs.default.name</name> > <value>hdfs://localhost:9000</value> > <description>The name of the default file system. A URI whose > scheme and authority determine the FileSystem implementation. The > uri's scheme determines the config property (fs.SCHEME.impl) naming > the FileSystem implementation class. The uri's authority is used to > determine the host, port, etc. for a filesystem.</description> </property> > > <property> > <name>dfs.replication</name> > <value>1</value> > <description>Default block replication. > The actual number of replications can be specified when the file is created. > The default is used if replication is not specified in create time. > </description> > </property> > > </configuration> > > core-site.xml > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- Put site-specific property overrides in this file. --> > > <configuration> > <property> > <name>hadoop.tmp.dir</name> > > <value>/Users/rahulmalviya/Documents/Hadoop/hadoop-0.21.0/hadoop-${user.name}</value> > <description>A base for other temporary directories.</description> > </property> </configuration> > > mapred-site.xml > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- Put site-specific property overrides in this file. --> > > <configuration> > <property> > <name>mapred.job.tracker</name> > <value>localhost:9001</value> > <description>The host and port that the MapReduce job tracker runs > at. If "local", then jobs are run in-process as a single map > and reduce task. > </description> > </property> > > <property> > <name>mapred.tasktracker.tasks.maximum</name> > <value>8</value> > <description>The maximum number of tasks that will be run simultaneously by a > a task tracker </description> </property> </configuration> > > Please let me know if there is a issue in my configurations ? Any input is > valuable for me. > > Thanks, > Rahul > > On Aug 26, 2010, at 6:10 PM, Jeff Zhang wrote: > >> Do you put the hadoop conf on classpath ? It seems you are still using >> local file system but conncect Hadoop's JobTracker. >> Make sure you set the correct configuration in core-site.xml >> hdfs-site.xml, mapred-site.xml, and put them on classpath. >> >> >> >> On Thu, Aug 26, 2010 at 5:32 PM, rahul <rmalv...@apple.com> wrote: >>> Hi , >>> >>> I am trying to integrate Pig with Hadoop for processing of jobs. >>> >>> I am able to run Pig in local mode and Hadoop with streaming api perfectly. >>> >>> But when I try to run Pig with Hadoop I get follwong Error: >>> >>> Pig Stack Trace >>> --------------- >>> ERROR 2116: Unexpected error. Could not validate the output >>> specification for: >>> file:///Users/rahulmalviya/Documents/Pig/dev/main_merged_hdp_out >>> >>> org.apache.pig.impl.plan.PlanValidationException: ERROR 0: An unexpected >>> exception caused the validation to stop >>> at >>> org.apache.pig.impl.plan.PlanValidator.validate(PlanValidator.java:56) >>> at >>> org.apache.pig.impl.logicalLayer.validators.InputOutputFileValidator.validate(InputOutputFileValidator.java:49) >>> at >>> org.apache.pig.impl.logicalLayer.validators.InputOutputFileValidator.validate(InputOutputFileValidator.java:37) >>> at >>> org.apache.pig.impl.logicalLayer.validators.LogicalPlanValidationExecutor.validate(LogicalPlanValidationExecutor.java:89) >>> at org.apache.pig.PigServer.validate(PigServer.java:930) >>> at org.apache.pig.PigServer.compileLp(PigServer.java:910) >>> at org.apache.pig.PigServer.compileLp(PigServer.java:871) >>> at org.apache.pig.PigServer.compileLp(PigServer.java:852) >>> at org.apache.pig.PigServer.execute(PigServer.java:816) >>> at org.apache.pig.PigServer.access$100(PigServer.java:105) >>> at org.apache.pig.PigServer$Graph.execute(PigServer.java:1080) >>> at org.apache.pig.PigServer.executeBatch(PigServer.java:288) >>> at >>> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:109) >>> at >>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166) >>> at >>> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:138) >>> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) >>> at org.apache.pig.Main.main(Main.java:391) >>> Caused by: org.apache.pig.impl.plan.PlanValidationException: ERROR 2116: >>> Unexpected error. Could not validate the output specification for: >>> file:///Users/rahulmalviya/Documents/Pig/dev/main_merged_hdp_out >>> at >>> org.apache.pig.impl.logicalLayer.validators.InputOutputFileVisitor.visit(InputOutputFileVisitor.java:93) >>> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:140) >>> at org.apache.pig.impl.logicalLayer.LOStore.visit(LOStore.java:37) >>> at >>> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:67) >>> at >>> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) >>> at >>> org.apache.pig.impl.plan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:69) >>> at >>> org.apache.pig.impl.plan.DepthFirstWalker.walk(DepthFirstWalker.java:50) >>> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51) >>> at >>> org.apache.pig.impl.plan.PlanValidator.validate(PlanValidator.java:50) >>> ... 16 more >>> Caused by: java.io.IOException: Call to localhost/127.0.0.1:9001 failed on >>> local exception: java.io.EOFException >>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) >>> at org.apache.hadoop.ipc.Client.call(Client.java:743) >>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) >>> at org.apache.hadoop.mapred.$Proxy0.getProtocolVersion(Unknown Source) >>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) >>> at >>> org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:429) >>> at org.apache.hadoop.mapred.JobClient.init(JobClient.java:423) >>> at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:410) >>> at org.apache.hadoop.mapreduce.Job.<init>(Job.java:50) >>> at >>> org.apache.pig.impl.logicalLayer.validators.InputOutputFileVisitor.visit(InputOutputFileVisitor.java:89) >>> ... 24 more >>> Caused by: java.io.EOFException >>> at java.io.DataInputStream.readInt(DataInputStream.java:375) >>> at >>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:501) >>> at >>> org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) >>> ===================================================================== >>> =========== >>> >>> Did anyone got the same error. I think it related to connection between pig >>> and hadoop. >>> >>> Can someone tell me how to connect Pig and hadoop. >>> >>> Thanks. >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >
