[ 
https://issues.apache.org/jira/browse/PIG-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Bhat updated PIG-974:
---------------------------

    Attachment: studenttab10k

Testdata

> Issues with mv command when used after store when using -param_file/-param 
> options
> ----------------------------------------------------------------------------------
>
>                 Key: PIG-974
>                 URL: https://issues.apache.org/jira/browse/PIG-974
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>         Environment: Hadoop 18 and 20
>            Reporter: Viraj Bhat
>             Fix For: 0.6.0
>
>         Attachments: studenttab10k
>
>
> I have a Pig script which moves the final output to another HDFS directory to 
> signal completion, so that another Pig script can start working on these 
> results.
> {code}
> studenttab = LOAD '/user/viraj/studenttab10k' AS (name:chararray, 
> age:int,gpa:float);
> X = GROUP studenttab by age;
> Y = FOREACH X GENERATE group, COUNT(studenttab);
> store Y into '$finalop' using PigStorage();
> mv '$finalop' '$finalmove';
> {code}
> where "finalop" and "finalmove" are parameters used storing intermediate and 
> final results.
> I run this script as this:
> {code}
> $shell> java -cp pig20.jar:/path/tohadoop/site.xml 
> -Dmapred.job.queue.name=default org.apache.pig.Main -M -param 
> finalop=/user/viraj/finaloutput -param finalmove=/user/viraj/finalmove 
> testmove.pig 
> {code}
> or using the param_file option
> {code}
> $shell>java -cp pig20.jar:/path/tohadoop/site.xml 
> -Dmapred.job.queue.name=default org.apache.pig.Main -M -param_file 
> moveparamfile  testmove.pig
> {code}
> ================================================================================
> The underlying Map Reduce jobs run well but the move command seems to be 
> failing:
> ================================================================================
> 2009-09-23 23:26:21,781 [main] INFO  org.apache.pig.Main - Logging error 
> messages to: /homes/viraj/pigscripts/pig_1253748381778.log
> 2009-09-23 23:26:21,963 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
> to hadoop file system at: hdfs://localhost:8020
> 2009-09-23 23:26:22,227 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting 
> to map-reduce job tracker at: localhost:50300
> 2009-09-23 23:26:27,187 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer
>  - Choosing to move algebraic foreach to combiner
> 2009-09-23 23:26:27,203 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>  - MR plan size before optimization: 1
> 2009-09-23 23:26:27,203 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>  - MR plan size after optimization: 1
> 2009-09-23 23:26:28,828 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>  - Setting up single store job
> 2009-09-23 23:26:29,423 [Thread-9] WARN  org.apache.hadoop.mapred.JobClient - 
> Use GenericOptionsParser for parsing the arguments. Applications should 
> implement Tool for the same.
> 2009-09-23 23:26:29,478 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - 0% complete
> 2009-09-23 23:27:29,828 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - 50% complete
> 2009-09-23 23:27:59,764 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - 50% complete
> 2009-09-23 23:28:57,249 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - 100% complete
> 2009-09-23 23:28:57,249 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - Successfully stored result in: "/user/viraj/finaloutput"
> 2009-09-23 23:28:57,267 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - Records written : 60
> 2009-09-23 23:28:57,267 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - Bytes written : 420
> 2009-09-23 23:28:57,267 [main] INFO  
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>  - Success!
> 2009-09-23 23:28:57,367 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
> 2998: Unhandled internal error. File or directory '/user/viraj/finaloutput' 
> does not exist.
> Details at logfile: /homes/viraj/pigscripts/pig_1253748381778.log
> ================================================================================
> {code}
> $shell> hadoop fs -ls /user/viraj/finaloutput 
> Found 1 items
> -rw-------   3 viraj users        420 2009-09-23 23:42 
> /user/viraj/finaloutput/part-00000
> {code}
> ================================================================================
> Opening the log file:
> ================================================================================
> Pig Stack Trace
> ---------------
> ERROR 2998: Unhandled internal error. File or directory 
> '/user/viraj/finaloutput' does not exist.
> java.io.IOException: File or directory '/user/viraj/finaloutput' does not 
> exist.
>         at 
> org.apache.pig.tools.grunt.GruntParser.processMove(GruntParser.java:641)
>         at 
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:264)
>         at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166)
>         at 
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:142)
>         at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
>         at org.apache.pig.Main.main(Main.java:397)
> ================================================================================
> Viraj

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to