[
https://issues.apache.org/jira/browse/PIG-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411095#comment-13411095
]
Araceli Henley commented on PIG-2791:
-------------------------------------
Hi Daniel
I tried a couple of tests with the patch you provided. I'm getting a different
error now.
Assuming the client side mount table has the following:
<property><name>fs.viewfs.impl</name>
<value>org.apache.hadoop.fs.viewfs.ViewFileSystem</value>
<description>The File System for viewfs:uris</description>
</property>
<property><name>fs.default.name</name>
<value>viewfs:///</value>
<final>true</final>
</property>
<property>
<name>fs.viewfs.mounttable.default.link./data1</name>
<value>hdfs://mycluster.yahoo.com:8020/user/me/pig/tests/data</value>
</property>
I confirm the file is visible ( I also tried fs -cat and it was successful)
-bash-3.1$ hadoop fs -ls /data1/singlefile/studenttab10k
-rw-r--r-- 3 hadoopqa hdfs 219190 2012-07-10 23:02
/data1/singlefile/studenttab10k
Next I try to a simple load and dump or store as follows:
a = load '/data1/singlefile/studenttab10k' as (name, age, gpa);
dump a;
This results in a stack trace:
RROR 1066: Unable to open iterator for alias a. Backend error : Trying to get
information for an absent application application_1341957183614_0010
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open
iterator for alias a. Backend error : Trying to get information for an absent
application application_1341957183614_0010
at org.apache.pig.PigServer.openIterator(PigServer.java:852)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:490)
at org.apache.pig.Main.main(Main.java:111)
Caused by: org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Trying to get information for an absent application
application_1341957183614_0010
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:156)
at $Proxy9.getApplicationReport(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClientImpl.java:116)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:338)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:143)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:298)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:383)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:481)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:627)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:625)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:625)
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:679)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:673)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:148)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:383)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1275)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1260)
at org.apache.pig.PigServer.storeEx(PigServer.java:957)
at org.apache.pig.PigServer.store(PigServer.java:924)
at org.apache.pig.PigServer.openIterator(PigServer.java:837)
... 7 more
================================================================================
Pig Stack Trace
---------------
ERROR 2997: Encountered IOException. File or directory -l does not exist.
java.io.IOException: File or directory -l does not exist.
at
org.apache.pig.tools.grunt.GruntParser.processLS(GruntParser.java:766)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:366)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:490)
at org.apache.pig.Main.main(Main.java:111)
Additionally the following also fails:
cd /data1/singlefile
a = load 'studenttab10k' as (name, age, gpa);
dump a;
> Can not enter grunt shell when using viewFS filesystem with CSMT and
> Federation
> -------------------------------------------------------------------------------
>
> Key: PIG-2791
> URL: https://issues.apache.org/jira/browse/PIG-2791
> Project: Pig
> Issue Type: Bug
> Components: grunt
> Affects Versions: 0.10.0
> Environment: Pig QE
> Reporter: patrick white
> Priority: Blocker
> Attachments: PIG-2791-0.patch
>
>
> The Yahoo Pig QE team ran into a blocking issue when trying to test
> Client-Side Mount Tables, on a Federated cluster with two NNs, this blocks
> Pig Testing on Federation.
> Federation relies strongly on the use of CSMT with viewFS, QE found that in
> this configuration it is not possible to enter grunt shell because Pig makes
> a call to getDefaultReplication() on the fs, which is ambiguous over viewFS
> and causes core to throw a
> org.apache.hadoop.fs.viewfs.NotInMountpointException: "getDefaultReplication
> on empty path is invalid".
> This in turn cause Pig to exit with an internal error as follows:
> 2012-07-06 22:20:25,657 [main] INFO org.apache.pig.Main - Apache Pig version
> 0.10.1.0.1206081058 (r1348169) compiled Jun 08 2012, 17:58:42
> 2012-07-06 22:20:26,074 [main] WARN org.apache.hadoop.conf.Configuration -
> mapred.used.genericoptionsparser is deprecated. Instead, use
> mapreduce.client.genericoptionsparser.used
> 2012-07-06 22:20:26,076 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: viewfs:///
> 2012-07-06 22:20:26,080 [main] WARN org.apache.hadoop.conf.Configuration -
> fs.default.name is deprecated. Instead, use fs.defaultFS
> 2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main - ERROR 2999:
> Unexpected internal error. getDefaultReplication on empty path is invalid
> 2012-07-06 22:20:26,522 [main] WARN org.apache.pig.Main - There is no log
> file to write to.
> 2012-07-06 22:20:26,522 [main] ERROR org.apache.pig.Main -
> org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultReplication
> on empty path is invalid
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:482)
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.init(HDataStorage.java:77)
> at
> org.apache.pig.backend.hadoop.datastorage.HDataStorage.<init>(HDataStorage.java:58)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:205)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:118)
> at org.apache.pig.impl.PigContext.connect(PigContext.java:208)
> at org.apache.pig.PigServer.<init>(PigServer.java:246)
> at org.apache.pig.PigServer.<init>(PigServer.java:231)
> at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:47)
> at org.apache.pig.Main.run(Main.java:487)
> at org.apache.pig.Main.main(Main.java:111)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
