[
https://issues.apache.org/jira/browse/PIG-4880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257141#comment-15257141
]
Koji Noguchi commented on PIG-4880:
-----------------------------------
bq. error msg: <file myscript.pig, line 4> Macro inline failed for macro
'mygroupby'. Reason: null
By making it dump the stack trace in a testcase, it showed
{noformat}
java.lang.NullPointerException
at
org.apache.pig.tools.parameters.PreprocessorContext.processOrdLine(PreprocessorContext.java:179)
at
org.apache.pig.tools.parameters.PigFileParser.param_value(PigFileParser.java:117)
at
org.apache.pig.tools.parameters.PigFileParser.input(PigFileParser.java:57)
at
org.apache.pig.tools.parameters.PigFileParser.Parse(PigFileParser.java:43)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:95)
at
org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:76)
at org.apache.pig.parser.PigMacro.substituteParams(PigMacro.java:182)
at org.apache.pig.parser.PigMacro.inline(PigMacro.java:96)
at org.apache.pig.parser.PigMacro.macroInline(PigMacro.java:489)
at
org.apache.pig.parser.QueryParserDriver.inlineMacro(QueryParserDriver.java:301)
at
org.apache.pig.parser.QueryParserDriver.expandMacro(QueryParserDriver.java:290)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:183)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1729)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1702)
at org.apache.pig.PigServer.registerQuery(PigServer.java:645)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1075)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:505)
at org.apache.pig.tools.grunt.GruntParser.loadScript(GruntParser.java:557)
at
org.apache.pig.tools.grunt.GruntParser.processExplain(GruntParser.java:370)
at org.apache.pig.tools.grunt.Grunt.checkScript(Grunt.java:95)
at org.apache.pig.Main.run(Main.java:629)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at
org.apache.pig.test.TestMacroExpansion.verify(TestMacroExpansion.java:2410)
at
org.apache.pig.test.TestMacroExpansion.testParamOverLap4(TestMacroExpansion.java:2357)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:535)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1182)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1033)
{noformat}
Corresponding code at
{code:title=PreprocessorContext.java}
176 public void processOrdLine(String key, String val, Boolean overwrite)
throws ParameterSubstitutionException {
177
178 if (param_val.containsKey(key)) {
179 if (param_source.get(key).equals(val) || !overwrite) {
{code}
and this showed the case when {{param_val}} and {{param_source}} became
inconsistent in terms of list of keys. As a result, {{param_source.get(key)}}
returned null.
Tracing the code, inconsistency was introduced at
{code:title=PigMacro.java}
165 try {
166 PreprocessorContext pc = new PreprocessorContext(50);
167 pc.loadParamVal(Arrays.asList(args), null);
168
169 Map<String, String> paramVal = pc.getParamVal();
170 for (Map.Entry<String, String> e :
pigContext.getParamVal().entrySet()) {
171 if (paramVal.containsKey(e.getKey())) {
172 throw new ParserException(
173 "Macro contains argument or return value " +
e.getKey() + " which conflicts " +
174 "with a Pig parameter of the same name."
175 );
176 } else {
177 paramVal.put(e.getKey(), e.getValue());
178 }
179 }
{code}
where PigMacro updates the {{param_val}} but not {{param_source}} for global
param substitutions.
> Overlapping of parameter substitution names inside&outside a macro fails with
> NPE
> ---------------------------------------------------------------------------------
>
> Key: PIG-4880
> URL: https://issues.apache.org/jira/browse/PIG-4880
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.12.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
>
> With
> {code:title=macro.pig}
> DEFINE mygroupby(REL, key) RETURNS G {
> %declare number 333;
> $G = GROUP $REL by $key parallel $number;
> };
> {code}
> and
> {code:title=test.pig}
> -- equivalent of -param number=111
> %declare number 111;
> IMPORT 'macro.pig';
> data = LOAD '1234.txt' USING PigStorage() AS (i: int);
> result = mygroupby(data, i);
> STORE result INTO 'test.out' USING PigStorage();
> {code}
> Fails with
> {{error msg: <file myscript.pig, line 4> Macro inline failed for macro
> 'mygroupby'. Reason: null}}
> Similarly, when macro param and command-line param overlap, it fails with
> {{Macro inline failed for macro 'mygroupby'. Reason: Macro contains argument
> or return value number which conflicts with a Pig parameter of the same
> name.}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
