[
https://issues.apache.org/jira/browse/PIG-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13958560#comment-13958560
]
Nadav Weissman commented on PIG-502:
------------------------------------
I ran with Apache Pig version 0.11.1 and an illustrate of limit in a nested
foreach statement gives a wrong presentation (doing dump to it - gave the
correct results)
example:
b = group a by aid;
c_min = foreach b{
o = order a by start_ts ASC;
l = limit o 1;
generate flatten(l);
};
----------------------------------------------------------------------------------------------------------------
| a | aid:chararray | sid:chararray |
start_ts:datetime |
----------------------------------------------------------------------------------------------------------------
| | 8fa | 1167c | 2008-02-27T00:00:00.000-08:00 |
| | 8fa | 38a05 | 2008-02-29T00:00:00.000-08:00 |
----------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| b | group:chararray |
a:bag{:tuple(aid:chararray,sid:chararray,start_ts:datetime)}
|
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| | 8fa | {(8fa, 1167c, 2008-02-27T00:00:00.000-08:00), (8fa, 38a05,
2008-02-29T00:00:00.000-08:00)} |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------
| c_min.o | aid:chararray | sid:chararray |
start_ts:datetime |
----------------------------------------------------------------------------------------------------------------------
| | 8fa | 1167c | 2008-02-27T00:00:00.000-08:00 |
| | 8fa | 38a05 | 2008-02-29T00:00:00.000-08:00 |
----------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------
| c_min.l | aid:chararray | sid:chararray |
start_ts:datetime |
----------------------------------------------------------------------------------------------------------------------
| | 8fa | 1167c | 2008-02-27T00:00:00.000-08:00 |
| | 8fa | 38a05 | 2008-02-29T00:00:00.000-08:00 |
----------------------------------------------------------------------------------------------------------------------
c_min | l::aid:chararray | l::sid:chararray |
l::start_ts:datetime |
-----------------------------------------------------------------------------------------------------------------------
| | 8fa | 1167c | 2008-02-27T00:00:00.000-08:00 |
| | 8fa | 38a05 | 2008-02-29T00:00:00.000-08:00 |
> Limit and Illustrate do not work together
> -----------------------------------------
>
> Key: PIG-502
> URL: https://issues.apache.org/jira/browse/PIG-502
> Project: Pig
> Issue Type: Sub-task
> Components: tools
> Affects Versions: 0.2.0
> Environment: Hadoop 18
> Reporter: Viraj Bhat
> Assignee: Yan Zhou
> Fix For: 0.9.0
>
>
> Suppose a user wants to do an illustrate command after limiting his data to a
> certain number of records, it does not seem to work..
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> {code}
> MYDATA = load 'testfilelarge.txt' as (f1, f2, f3, f4, f5);
> MYDATA = limit MYDATA 10;
> describe MYDATA;
> illustrate MYDATA;
> {code}
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> Running this script produces the following output and error
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> MYDATA: {f1: bytearray,f2: bytearray,f3: bytearray,f4: bytearray,f5:
> bytearray}
> 2008-10-18 02:14:26,900 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop fil
> e system at: hdfs://localhost:9000
> 2008-10-18 02:14:27,013 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to map-reduce
> job tracker at: localhost:9001
> java.lang.RuntimeException: Unrecognized logical operator.
> at
> org.apache.pig.pen.EquivalenceClasses.GetEquivalenceClasses(EquivalenceClasses.java:60)
> at
> org.apache.pig.pen.DerivedDataVisitor.evaluateOperator(DerivedDataVisitor.java:368)
> at
> org.apache.pig.pen.DerivedDataVisitor.visit(DerivedDataVisitor.java:273)
> at org.apache.pig.impl.logicalLayer.LOLimit.visit(LOLimit.java:71)
> at org.apache.pig.impl.logicalLayer.LOLimit.visit(LOLimit.java:10)
> at
> org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:68)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
> at
> org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:98)
> at
> org.apache.pig.pen.LineageTrimmingVisitor.<init>(LineageTrimmingVisitor.java:90)
> at
> org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:106)
> at org.apache.pig.PigServer.getExamples(PigServer.java:630)
> at
> org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:279)
> at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:183)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:84)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:64)
> at org.apache.pig.Main.main(Main.java:306)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> If I remove the illustrate and replace it with "dump MYDATA;" it works..
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> {code}
> MYDATA = load 'testfilelarge.txt' as (f1, f2, f3, f4, f5);
> MYDATA = limit MYDATA 10;
> describe MYDATA;
> -- illustrate MYDATA;
> dump MYDATA;
> {code}
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--
This message was sent by Atlassian JIRA
(v6.2#6252)
