Hi Daniel I was using this command: "/bin/pig -x local -x tez tutorial/scripts/script1-local.pig"
On Tue, May 27, 2014 at 1:07 PM, Daniel Dai <da...@hortonworks.com> wrote: > Hi, Chen, > > The hook for for tez local mode is: > > bin/pig -x tez_local > > You might need to set PIG_OPTS="-Dopt.fetch=false" to prevent it > bypassing the tez engine. > > How do you run tutorial example? Using MR local mode? > > Thanks, > Daniel > > On Tue, May 27, 2014 at 9:32 AM, Chen He <airb...@gmail.com> wrote: > > We are running pig tutorial example and get following errors. I think we > > are pick up some hadoop config that specify HDFS instead of localFS. > > > > Here are the errors from pig.log > > > > Pig Stack Trace > > --------------- > > ERROR 6007: Unable to check name hdfs://localhost:9000/user/chenhe > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error > > during parsing. Unable to check name hdfs://localhost:9000/user/chenhe > > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1736) > > at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1431) > > at org.apache.pig.PigServer.parseAndBuild(PigServer.java:375) > > at org.apache.pig.PigServer.executeBatch(PigServer.java:400) > > at org.apache.pig.PigServer.executeBatch(PigServer.java:386) > > at > > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) > > at > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232) > > at > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203) > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > > at org.apache.pig.Main.run(Main.java:608) > > at org.apache.pig.Main.main(Main.java:156) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > > Caused by: Failed to parse: Pig script failed to parse: > > <file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6> > > pig script failed to validate: > > org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: > Unable > > to check name hdfs://localhost:9000/user/chenhe > > at > > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196) > > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1723) > > ... 15 more > > Caused by: > > <file ../../../../tutorial/scripts/script1-local.pig, line 28, column 6> > > pig script failed to validate: > > org.apache.pig.backend.datastorage.DataStorageException: ERROR 6007: > Unable > > to check name hdfs://localhost:9000/user/chenhe > > at > > > org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:881) > > at > > > org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) > > at > > > org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) > > at > > > org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) > > at > > > org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) > > at > > > org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) > > at > > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188) > > ... 16 more > > Caused by: org.apache.pig.backend.datastorage.DataStorageException: ERROR > > 6007: Unable to check name hdfs://localhost:9000/user/chenhe > > at > > > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:207) > > at > > > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:128) > > at > > > org.apache.pig.backend.hadoop.datastorage.HDataStorage.asElement(HDataStorage.java:138) > > at > > > org.apache.pig.parser.QueryParserUtils.getCurrentDir(QueryParserUtils.java:90) > > at > > > org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:873) > > ... 22 more > > Caused by: java.net.ConnectException: Call From > > guardwritten-dl.corp.yahoo.com/10.74.90.55 to localhost:9000 failed on > > connection exception: java.net.ConnectException: Connection refused; For > > more details see: http://wiki.apache.org/hadoop/ConnectionRefused > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:700) > > at org.apache.hadoop.ipc.Client.call(Client.java:1098) > > at > > > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:195) > > at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:102) > > at > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:67) > > at com.sun.proxy.$Proxy6.getFileInfo(Unknown Source) > > at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1305) > > at > > > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:734) > > at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1272) > > at > > > org.apache.pig.backend.hadoop.datastorage.HDataStorage.isContainer(HDataStorage.java:200) > > ... 26 more > > Caused by: java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) > > at > > > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) > > at > > org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:462) > > at > > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:559) > > at > org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:207) > > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1204) > > at org.apache.hadoop.ipc.Client.call(Client.java:1074) > > ... 39 more > > > ================================================================================ > > > > > > > > On Tue, May 27, 2014 at 11:04 AM, Chen He <airb...@gmail.com> wrote: > > > >> Hi Daniel > >> > >> Thank you for your reply. Any suggestion about how can I run pig local > >> mode using Tez without minicluster? I am sure the code on Tez side is > >> ready. What changes need to be made on PIG? I am not very familar with > PIG. > >> > >> Regards! > >> > >> Chen > >> > >> > >> On Mon, May 26, 2014 at 8:22 PM, Chen He <airb...@gmail.com> wrote: > >> > >>> Hi Jon, Daniel > >>> > >>> Daniel was right. I tried pig local mode using "pig -x local -x tez > …..". > >>> It tries to connect mini cluster and finally timeout. The error that I > >>> mentioned before : "ERROR 2998: Unhandled internal error. Found > >>> interface org.apache.hadoop.yarn.api.records.LocalResource, but class > >>> was expected" > >>> is because I have a hadoop 0.23 mini cluster running in background. > >>> > >>> Chen > >>> > >>> > >>> On Mon, May 26, 2014 at 9:39 AM, Jonathan Eagles <jeag...@gmail.com > >wrote: > >>> > >>>> Hi Daniel, > >>>> > >>>> You've been very helpful in your replies. I believe the missing > context > >>>> is > >>>> that Chen is working on adding Tez Local Mode (TEZ-684) and is trying > to > >>>> integrate with Pig. The question as I interpret it is whether Pig > needs > >>>> or > >>>> creates a MiniCluster in local mode. Chen can you confirm this? > >>>> > >>>> Jon > >>>> > >>>> > >>>> On Mon, May 26, 2014 at 2:55 AM, Daniel Dai <da...@hortonworks.com> > >>>> wrote: > >>>> > >>>> > Hi, Chen, > >>>> > > >>>> > Please note Pig local mode does not run tez. It uses MR engine. > >>>> > Tez-local mode is not implemented (depend on TEZ-235). You will > need a > >>>> > real cluster or use TezMiniCluster to test Pig on Tez. > >>>> > > >>>> > Thanks, > >>>> > Daniel > >>>> > > >>>> > On Sun, May 25, 2014 at 11:42 AM, Chen He <airb...@gmail.com> > wrote: > >>>> > > Hi Daniel > >>>> > > > >>>> > > I was testing the pig local mode. Does pig need mini cluster when > it > >>>> runs > >>>> > > in local mode? > >>>> > > > >>>> > > Regards! > >>>> > > > >>>> > > Chen > >>>> > > > >>>> > > > >>>> > > On Sun, May 25, 2014 at 1:40 PM, Daniel Dai < > da...@hortonworks.com> > >>>> > wrote: > >>>> > > > >>>> > >> Which Hadoop are you running against? Also 2.2? > >>>> > >> > >>>> > >> On Sun, May 25, 2014 at 6:52 AM, Chen He <airb...@gmail.com> > wrote: > >>>> > >> > Hi Daniel > >>>> > >> > > >>>> > >> > Thank you for the reply. I compiled Tez and Pig against hadoop > >>>> 2.2. > >>>> > >> > > >>>> > >> > Regards! > >>>> > >> > > >>>> > >> > Chen > >>>> > >> > > >>>> > >> > > >>>> > >> > On Thu, May 22, 2014 at 5:22 PM, Daniel Dai < > >>>> da...@hortonworks.com> > >>>> > >> wrote: > >>>> > >> > > >>>> > >> >> Which version of Hadoop are you using? Seem LocalResource > changed > >>>> > from > >>>> > >> >> interface to class in 2.1 (YARN-711). If you are using > pre-2.1, > >>>> this > >>>> > >> >> might be the case. > >>>> > >> >> > >>>> > >> >> Thanks, > >>>> > >> >> Daniel > >>>> > >> >> > >>>> > >> >> On Thu, May 22, 2014 at 3:06 PM, Chen He <airb...@gmail.com> > >>>> wrote: > >>>> > >> >> > Hi Daniel > >>>> > >> >> > > >>>> > >> >> > Thank you for the reply. I did as you suggested but get > >>>> following > >>>> > >> errors: > >>>> > >> >> > > >>>> > >> >> > ERROR 2998: Unhandled internal error. Found interface > >>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class > was > >>>> > >> expected > >>>> > >> >> > > >>>> > >> >> > java.lang.IncompatibleClassChangeError: Found interface > >>>> > >> >> > org.apache.hadoop.yarn.api.records.LocalResource, but class > was > >>>> > >> expected > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.getTezResources(TezResourceManager.java:124) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.backend.hadoop.executionengine.tez.TezResourceManager.addTezResources(TezResourceManager.java:96) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.backend.hadoop.executionengine.tez.TezPlanContainer.getLocalResources(TezPlanContainer.java:104) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.getJob(TezJobControlCompiler.java:106) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.backend.hadoop.executionengine.tez.TezJobControlCompiler.compile(TezJobControlCompiler.java:89) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher.launchPig(TezLauncher.java:92) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277) > >>>> > >> >> > at > org.apache.pig.PigServer.launchPlan(PigServer.java:1378) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> > >>>> > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363) > >>>> > >> >> > at org.apache.pig.PigServer.execute(PigServer.java:1352) > >>>> > >> >> > at > >>>> org.apache.pig.PigServer.executeBatch(PigServer.java:403) > >>>> > >> >> > at > >>>> org.apache.pig.PigServer.executeBatch(PigServer.java:386) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> > >>>> > > >>>> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203) > >>>> > >> >> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) > >>>> > >> >> > at org.apache.pig.Main.run(Main.java:608) > >>>> > >> >> > at org.apache.pig.Main.main(Main.java:156) > >>>> > >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>> Method) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >>>> > >> >> > at > >>>> > >> >> > > >>>> > >> >> > >>>> > >> > >>>> > > >>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>>> > >> >> > at java.lang.reflect.Method.invoke(Method.java:606) > >>>> > >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > >>>> > >> >> > > >>>> > >> >> > Is this a problem that caused by hadoop incompatiblilty? > Which > >>>> > >> version of > >>>> > >> >> > Hadoop is the best? > >>>> > >> >> > > >>>> > >> >> > Regards! > >>>> > >> >> > > >>>> > >> >> > Chen > >>>> > >> >> > > >>>> > >> >> > > >>>> > >> >> > On Thu, May 22, 2014 at 4:19 PM, Daniel Dai < > >>>> da...@hortonworks.com > >>>> > > > >>>> > >> >> wrote: > >>>> > >> >> > > >>>> > >> >> >> Need to set PIG_CLASSPATH: > >>>> > >> >> >> export > >>>> PIG_CLASSPATH=$TEZ_HOME/*:$TEZ_HOME/lib/*:$TEZ_HOME/conf > >>>> > >> >> >> > >>>> > >> >> >> Thanks, > >>>> > >> >> >> Daniel > >>>> > >> >> >> > >>>> > >> >> >> On Thu, May 22, 2014 at 2:11 PM, Chen He < > airb...@gmail.com> > >>>> > wrote: > >>>> > >> >> >> > Any one met this error before? > >>>> > >> >> >> > > >>>> > >> >> >> > [main] ERROR org.apache.pig.Main - ERROR 2998: Unhandled > >>>> > internal > >>>> > >> >> error. > >>>> > >> >> >> > org/apache/tez/common/counters/DAGCounter > >>>> > >> >> >> > > >>>> > >> >> >> > Any reply will be appreciated! > >>>> > >> >> >> > > >>>> > >> >> >> > Regards! > >>>> > >> >> >> > > >>>> > >> >> >> > Chen > >>>> > >> >> >> > >>>> > >> >> >> -- > >>>> > >> >> >> CONFIDENTIALITY NOTICE > >>>> > >> >> >> NOTICE: This message is intended for the use of the > >>>> individual or > >>>> > >> >> entity to > >>>> > >> >> >> which it is addressed and may contain information that is > >>>> > >> confidential, > >>>> > >> >> >> privileged and exempt from disclosure under applicable law. > >>>> If the > >>>> > >> >> reader > >>>> > >> >> >> of this message is not the intended recipient, you are > hereby > >>>> > >> notified > >>>> > >> >> that > >>>> > >> >> >> any printing, copying, dissemination, distribution, > >>>> disclosure or > >>>> > >> >> >> forwarding of this communication is strictly prohibited. If > >>>> you > >>>> > have > >>>> > >> >> >> received this communication in error, please contact the > >>>> sender > >>>> > >> >> immediately > >>>> > >> >> >> and delete it from your system. Thank You. > >>>> > >> >> >> > >>>> > >> >> > >>>> > >> >> -- > >>>> > >> >> CONFIDENTIALITY NOTICE > >>>> > >> >> NOTICE: This message is intended for the use of the > individual or > >>>> > >> entity to > >>>> > >> >> which it is addressed and may contain information that is > >>>> > confidential, > >>>> > >> >> privileged and exempt from disclosure under applicable law. If > >>>> the > >>>> > >> reader > >>>> > >> >> of this message is not the intended recipient, you are hereby > >>>> > notified > >>>> > >> that > >>>> > >> >> any printing, copying, dissemination, distribution, > disclosure or > >>>> > >> >> forwarding of this communication is strictly prohibited. If > you > >>>> have > >>>> > >> >> received this communication in error, please contact the > sender > >>>> > >> immediately > >>>> > >> >> and delete it from your system. Thank You. > >>>> > >> >> > >>>> > >> > >>>> > >> -- > >>>> > >> CONFIDENTIALITY NOTICE > >>>> > >> NOTICE: This message is intended for the use of the individual or > >>>> > entity to > >>>> > >> which it is addressed and may contain information that is > >>>> confidential, > >>>> > >> privileged and exempt from disclosure under applicable law. If > the > >>>> > reader > >>>> > >> of this message is not the intended recipient, you are hereby > >>>> notified > >>>> > that > >>>> > >> any printing, copying, dissemination, distribution, disclosure or > >>>> > >> forwarding of this communication is strictly prohibited. If you > have > >>>> > >> received this communication in error, please contact the sender > >>>> > immediately > >>>> > >> and delete it from your system. Thank You. > >>>> > >> > >>>> > > >>>> > -- > >>>> > CONFIDENTIALITY NOTICE > >>>> > NOTICE: This message is intended for the use of the individual or > >>>> entity to > >>>> > which it is addressed and may contain information that is > confidential, > >>>> > privileged and exempt from disclosure under applicable law. If the > >>>> reader > >>>> > of this message is not the intended recipient, you are hereby > notified > >>>> that > >>>> > any printing, copying, dissemination, distribution, disclosure or > >>>> > forwarding of this communication is strictly prohibited. If you have > >>>> > received this communication in error, please contact the sender > >>>> immediately > >>>> > and delete it from your system. Thank You. > >>>> > > >>>> > >>> > >>> > >> > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
