[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12716246#action_12716246 ] Hudson commented on PIG-564: Integrated in Pig-trunk #463 (See [http://hudson.zones.apache.org/hudson/job/Pig-trunk/463/]) : problem with parameter substitution and special charachters (olgan) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Assignee: Olga Natkovich Attachments: PIG-564.patch Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12715977#action_12715977 ] Olga Natkovich commented on PIG-564: Thanks, Giri. I looked at warnings anf they are result of adding new test files for parameter substitution that don't have apache license because they don't support comments. So nothing to fix there. I will be committing the patch later today. Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Assignee: Olga Natkovich Attachments: PIG-564.patch Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12715742#action_12715742 ] Hadoop QA commented on PIG-564: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12409702/PIG-564.patch against trunk revision 780722. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 159 release audit warnings (more than the trunk's current 156 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/artifact/trunk/current/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/console This message is automatically generated. Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Assignee: Olga Natkovich Attachments: PIG-564.patch Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example:
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12715764#action_12715764 ] Alan Gates commented on PIG-564: Questions/comments on the patch. 1) Why did output1.pig change to look exactly like the new input5.pig? It seems like output1.pig shouldn't have changed. 2) A comment in the javacc files on how OTHER and IDENTIFIER interact in the pattern matching might be helpful, as it isn't immediately obvious (at least to me :) ). As long as 1 is ok, then +1. Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Assignee: Olga Natkovich Attachments: PIG-564.patch Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12715767#action_12715767 ] Olga Natkovich commented on PIG-564: Alan, thanks for review. (1) output1.pig is a generated file. I think it was checked in initially by mistake. Its content is irrelevant. (2) I might have to resubmit a patch anyway if I figure out the extra warnings (the link is broken at the moment). If I have to do that, I will also add comments. Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Assignee: Olga Natkovich Attachments: PIG-564.patch Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12715792#action_12715792 ] Giridharan Kesavan commented on PIG-564: Use this link for releaseaudit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.apache.org/68/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt I 've fixed the test-patch scripts for the broken link. Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Assignee: Olga Natkovich Attachments: PIG-564.patch Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12704455#action_12704455 ] Viraj Bhat commented on PIG-564: Another special character / is not handled correctly and is particularly useful when passing nested output directories. It is truncated when it is passed to Pig. Example case: {code} a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray); b = foreach a generate id1; store b into '/user/viraj/$outputfolder' using PigStorage(); {code} Run this script as: {code} shellhadoop fs -mkdir /user/viraj/paramtest shelljava -cp pig.jar:/home/viraj/hadoop-0.18.0-dev/conf/ -Dhod.server='' org.apache.pig.Main -param outputfolder=paramtest/moretest paramtest.pig 2009-04-30 03:00:21,234 [main] INFO org.apache.pig.Main - Dry run completed. Substituted pig script is at paramtest.pig.substituted {code} Now if we open the substituted Pig script (paramtest.pig.substituted). {code} a = load '/user/viraj/test1' using PigStorage(',') as (id1:int, char1:chararray); b = foreach a generate id1; store b into '/user/viraj/paramtest' using PigStorage(); {code} Viraj Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Assignee: Olga Natkovich Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,'
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12696266#action_12696266 ] David Ciemiewicz commented on PIG-564: -- Period (.) is also a special character that seems to cause problems. See related JIRA PIG-754 Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?,' --- Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: 0.2.0 Reporter: Viraj Bhat Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-564) Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,?
[ https://issues.apache.org/jira/browse/PIG-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12667988#action_12667988 ] Romain Rigaux commented on PIG-564: --- Another possible workaround seems to wrap the param between two ' or and escape its beginning with a \ -param url='http://www.yahoo.com' -- http -param url='\http://www.yahoo.com' -- http://www.yahoo.com Parameter Substitution using -param option does not seem to work when parameters contain special characters such as +,=,-,? Key: PIG-564 URL: https://issues.apache.org/jira/browse/PIG-564 Project: Pig Issue Type: Bug Components: impl Affects Versions: types_branch Reporter: Viraj Bhat Fix For: types_branch Consider the following Pig script which uses parameter substitution {code} %default qual '/user/viraj' %default mydir 'mydir_myextraqual' VISIT_LOGS = load '$qual/$mydir' as (a,b,c); dump VISIT_LOGS; {code} If you run the script as: == java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param mydir=mydir-myextraqual mypigparamsub.pig == You get the following error: == 2008-12-15 19:49:43,964 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: /user/viraj/mydir does not exist at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:109) at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) at org.apache.pig.impl.io.ValidatingInputFileSpec.init(ValidatingInputFileSpec.java:44) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Unable to open iterator for alias: VISIT_LOGS [Job terminated with anomalous status FAILED] at org.apache.pig.PigServer.openIterator(PigServer.java:389) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:269) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:178) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64) at org.apache.pig.Main.main(Main.java:306) Caused by: java.io.IOException: Job terminated with anomalous status FAILED ... 6 more == Also tried using: -param mydir='mydir\-myextraqual' This behavior occurs if the parameter value contains characters such as +,=, ?. A workaround for this behavior is using a param_file which contains param_name=param_value on each line, with the param_value enclosed by quotes. For example: mydir='mydir-myextraqual' and then running the pig script as: java -cp pig.jar:${HADOOP_HOME}/conf/ -Dhod.server='' org.apache.pig.Main -param_file myparamfile mypigparamsub.pig The following issues need to be fixed: 1) In -param option if parameter value contains special characters, it is truncated 2) In param_file, if param_value contains a special characters, it should be enclosed in quotes 3) If 2 is a known issue then it should be documented in http://wiki.apache.org/pig/ParameterSubstitution -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.