Reducer plan generation fails when UDF contains integers as parameters
----------------------------------------------------------------------
Key: PIG-568
URL: https://issues.apache.org/jira/browse/PIG-568
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Viraj Bhat
Priority: Critical
Fix For: types_branch
Consider the following pig script which contains a UDF known as MYUDF. MYUDF is
a dummy UDF which takes in a Bag E and a set of integers as offsets.
{code}
register myudf.jar;
A = load 'visits.txt' using PigStorage() as ( name:chararray, url:chararray,
timestamp:chararray);
B = filter A by (
(name is not null) AND
(timestamp is not null)
);
C = group B by (
url
);
D = foreach C {
E = order B by timestamp;
generate E;
}
G = foreach D generate
param.MYUDF(E, -1, 0, 1);
--this works
--param.MYUDF(E,'-1','0','1');
explain G;
dump G;
{code}
If you execute the above script, it fails during the reducer phase where the
POUserFunc(MYUDF)[bag] is being called. The MYUDF code is infact not called but
somehow the parameters passed to the MYUDF cause the exception in the reduce
plan. If you replace the offsets -1,0,1 with '-1', '0', '1' (strings) the UDF
seems to get called and the script works fine.
=============================================================================================================================
java.io.IOException: Received Error while processing the reduce plan.
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:307)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:247)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:224)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:136)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:318)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)
=============================================================================================================================
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
