I am using pig 0.9.0 and running in a local mode.
I have a requirement to call a UDF from within a macro. The reason for doing
so, is that I would like to nest a FOREACH loop and hence cannot call the UDF
directly in the GENERATE Clause - I am trying to obtain results based on all
possible combination of datasets.
When I put the UDF call into a macro I receive an error - whereas if I call it
directly I don't get any error and the execution is as expected
============================================================================================
The code fragment is as follows
//From the main calling program
REGISTER /home/vinay/HadoopData/tmp/PatternAnalysis.jar;
DEFINE crossPattern(TimesSeriesList, TupleName,currgroup,currDataset) RETURNS
crossPtrStr{
$crossPtrStr = ForEach $TimeSeriesList generate
DataHandling.crossAnalysis(group,$TupleName,$currgroup,$currDAtaSet);
};
DEFINE genTimeSeries(TimeSeries,customer ,variates) RETURNS customerTimeSeries{
custTimeSeries = Filter $TimeSeries By group == '$customer';
custTimeSeriesDataset = ForEach custTimeSeries generate DataSet;
$customerTimeSeries = Group custTimeSeriesDataset By
customer,$variates;
};
DEFINE genPatterns(AllTimeSeries,customer,variates) RETURNS allCrossPtrStr{
TimeSeriesList =
genTimeSeries($AllTimeSeries,$customer,$variates); -- gets one time seriesList
based on customer and other variate combinations
$allCrossPtrStr = foreach TimeSeriesList generate
crossPattern(TimeSeriesList,group,DataSet); --group/DataSet of the
TimeSeriesList
};
DEFINE loaddata(filename,delimiter,recordFormat,variates) RETURNS VOID {
File = LOAD '$filename' USING PigStorage('$delimiter') as $recordFormat;
DataSet = FILTER File BY $0!='DateTime';
Customer = Group DataSet BY customer; -- get distinct customers
crossPattern = FOREACH Customer GENERATE
group,FLATTEN(genPatterns(DataSet,group,$variates));
};
I face an issue while executing this - the error generated is
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve
genPatterns using imports: [, org.apache.pig.builtin.,
org.apache.pig.impl.builtin.]
Details at logfile: /home/vinay/HadoopData/tmp/pig_1329911992303.log
Based on the log entries it seems pig is unable to resolve the class name - I
checked the class name definition and there is no issue with that as I could
run the same UDF by directly calling it in a FOR EACH loop .
With the same UDF definitions I am able to execute the following command .
A = load 'xnD.csv' as
(date:chararray,customer:chararray,Merchant:chararray,Merchantzip:chararray,amt:float,relnusr:chararray,reln:chararray);
B = Group A by customer;
D = Limit B 1;
E = crossPattern(B,'cust1','D.A','A');
Log file contains the following
Pig Stack Trace
---------------
ERROR 1070: Could not resolve genPatterns using imports: [,
org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse:
at expanding macro 'loaddata' (null:29)
<line 23, column 56> Failed to generate logical plan. Nested exception:
java.lang.RuntimeException: Cannot instantiate: genPatterns
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:178)
at
org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1622)
at
org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1595)
at org.apache.pig.PigServer.registerQuery(PigServer.java:583)
at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:67)
at org.apache.pig.Main.run(Main.java:487)
at org.apache.pig.Main.main(Main.java:108)
Caused by:
<line 23, column 56> Failed to generate logical plan. Nested exception:
java.lang.RuntimeException: Cannot instantiate: genPatterns
at
org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:888)
at
org.apache.pig.parser.LogicalPlanGenerator.func_eval(LogicalPlanGenerator.java:6435)
at
org.apache.pig.parser.LogicalPlanGenerator.projectable_expr(LogicalPlanGenerator.java:7766)
at
org.apache.pig.parser.LogicalPlanGenerator.var_expr(LogicalPlanGenerator.java:7549)
at
org.apache.pig.parser.LogicalPlanGenerator.expr(LogicalPlanGenerator.java:6959)
at
org.apache.pig.parser.LogicalPlanGenerator.flatten_clause(LogicalPlanGenerator.java:5344)
at
org.apache.pig.parser.LogicalPlanGenerator.flatten_generated_item(LogicalPlanGenerator.java:5180)
at
org.apache.pig.parser.LogicalPlanGenerator.generate_clause(LogicalPlanGenerator.java:11165)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_plan(LogicalPlanGenerator.java:10932)
at
org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:10812)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1272)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:638)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:459)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:357)
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:171)
... 10 more
Caused by: java.lang.RuntimeException: Cannot instantiate: genPatterns
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:460)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:509)
at
org.apache.pig.impl.PigContext.instantiateFuncFromAlias(PigContext.java:532)
at
org.apache.pig.parser.LogicalPlanBuilder.buildUDF(LogicalPlanBuilder.java:885)
... 24 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070:
Could not resolve genPatterns using imports: [, org.apache.pig.builtin.,
org.apache.pig.impl.builtin.]
at
org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:445)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:457)
... 27 more
=================================================================================================================
**************** CAUTION - Disclaimer *****************
This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
for the use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you are
not
to copy, disclose, or distribute this e-mail or its contents to any other
person and
any such actions are unlawful. This e-mail may contain viruses. Infosys has
taken
every reasonable precaution to minimize this risk, but is not liable for any
damage
you may sustain as a result of any virus in this e-mail. You should carry out
your
own virus checks before opening the e-mail or attachment. Infosys reserves the
right to monitor and review the content of all messages sent to or from this
e-mail
address. Messages sent to or from this e-mail address may be stored on the
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***
