[ https://issues.apache.org/jira/browse/PIG-974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich resolved PIG-974. -------------------------------- Resolution: Won't Fix > Issues with mv command when used after store when using -param_file/-param > options > ---------------------------------------------------------------------------------- > > Key: PIG-974 > URL: https://issues.apache.org/jira/browse/PIG-974 > Project: Pig > Issue Type: Bug > Affects Versions: 0.6.0 > Environment: Hadoop 18 and 20 > Reporter: Viraj Bhat > Fix For: 0.6.0 > > Attachments: studenttab10k > > > I have a Pig script which moves the final output to another HDFS directory to > signal completion, so that another Pig script can start working on these > results. > {code} > studenttab = LOAD '/user/viraj/studenttab10k' AS (name:chararray, > age:int,gpa:float); > X = GROUP studenttab by age; > Y = FOREACH X GENERATE group, COUNT(studenttab); > store Y into '$finalop' using PigStorage(); > mv '$finalop' '$finalmove'; > {code} > where "finalop" and "finalmove" are parameters used storing intermediate and > final results. > I run this script as this: > {code} > $shell> java -cp pig20.jar:/path/tohadoop/site.xml > -Dmapred.job.queue.name=default org.apache.pig.Main -M -param > finalop=/user/viraj/finaloutput -param finalmove=/user/viraj/finalmove > testmove.pig > {code} > or using the param_file option > {code} > $shell>java -cp pig20.jar:/path/tohadoop/site.xml > -Dmapred.job.queue.name=default org.apache.pig.Main -M -param_file > moveparamfile testmove.pig > {code} > ================================================================================ > The underlying Map Reduce jobs run well but the move command seems to be > failing: > ================================================================================ > 2009-09-23 23:26:21,781 [main] INFO org.apache.pig.Main - Logging error > messages to: /homes/viraj/pigscripts/pig_1253748381778.log > 2009-09-23 23:26:21,963 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting > to hadoop file system at: hdfs://localhost:8020 > 2009-09-23 23:26:22,227 [main] INFO > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting > to map-reduce job tracker at: localhost:50300 > 2009-09-23 23:26:27,187 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer > - Choosing to move algebraic foreach to combiner > 2009-09-23 23:26:27,203 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size before optimization: 1 > 2009-09-23 23:26:27,203 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > - MR plan size after optimization: 1 > 2009-09-23 23:26:28,828 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > - Setting up single store job > 2009-09-23 23:26:29,423 [Thread-9] WARN org.apache.hadoop.mapred.JobClient - > Use GenericOptionsParser for parsing the arguments. Applications should > implement Tool for the same. > 2009-09-23 23:26:29,478 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 0% complete > 2009-09-23 23:27:29,828 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 50% complete > 2009-09-23 23:27:59,764 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 50% complete > 2009-09-23 23:28:57,249 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - 100% complete > 2009-09-23 23:28:57,249 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Successfully stored result in: "/user/viraj/finaloutput" > 2009-09-23 23:28:57,267 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Records written : 60 > 2009-09-23 23:28:57,267 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Bytes written : 420 > 2009-09-23 23:28:57,267 [main] INFO > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher > - Success! > 2009-09-23 23:28:57,367 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR > 2998: Unhandled internal error. File or directory '/user/viraj/finaloutput' > does not exist. > Details at logfile: /homes/viraj/pigscripts/pig_1253748381778.log > ================================================================================ > {code} > $shell> hadoop fs -ls /user/viraj/finaloutput > Found 1 items > -rw------- 3 viraj users 420 2009-09-23 23:42 > /user/viraj/finaloutput/part-00000 > {code} > ================================================================================ > Opening the log file: > ================================================================================ > Pig Stack Trace > --------------- > ERROR 2998: Unhandled internal error. File or directory > '/user/viraj/finaloutput' does not exist. > java.io.IOException: File or directory '/user/viraj/finaloutput' does not > exist. > at > org.apache.pig.tools.grunt.GruntParser.processMove(GruntParser.java:641) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:264) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:142) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89) > at org.apache.pig.Main.main(Main.java:397) > ================================================================================ > Viraj -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.