Aggregation not working in conjunction with REGEX_EXTRACT_ALL
-------------------------------------------------------------
Key: PIG-1965
URL: https://issues.apache.org/jira/browse/PIG-1965
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Sven Krasser
For the script below, the AVG function fails with an exception:
a = load 'bug.txt' using TextLoader() as (line:chararray);
c = foreach a generate FLATTEN (REGEX_EXTRACT_ALL (line, '.* f1:(\\S+) .*
f2:(\\d+).*')) as (f1:chararray, f2:int);
describe c; -- this is c: {f1: chararray,f2: int}
d = group c by f1;
e = foreach d generate group, AVG(c.f2);
dump e
Data for bug.txt:
a f1:abc d f2:12
f f1:def d f2:23
w f1:abc w f2:45
r f1:abc w f2:24
e f1:def q f2:34
If I store the data after parsing it, the schema stays the same, but now the
AVG function works. Workaround script:
a = load 'bug.txt' using TextLoader() as (line:chararray);
b = foreach a generate FLATTEN (REGEX_EXTRACT_ALL (line, '.* f1:(\\S+) .*
f2:(\\d+).*')) as (f1:chararray, f2:int);
describe b; -- this is b: {f1: chararray,f2: int}
store b into 'temp';
c = load 'temp' as (f1:chararray, f2:int);
describe c; -- this is c: {f1: chararray,f2: int}
d = group c by f1;
e = foreach d generate group, AVG(c.f2);
dump e
Exception in the error case:
org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while
computing average in Initial
at org.apache.pig.builtin.IntAvg$Initial.exec(IntAvg.java:100)
at org.apache.pig.builtin.IntAvg$Initial.exec(IntAvg.java:75)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:229)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:273)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:343)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:291)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:276)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:256)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:236)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:231)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Integer
at org.apache.pig.builtin.IntAvg$Initial.exec(IntAvg.java:87)
... 14 more
[...]
2011-04-05 18:18:33,865 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2011-04-05 18:18:33,866 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics -
Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already
initialized
2011-04-05 18:18:33,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
1066: Unable to open iterator for alias e
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
