[
https://issues.apache.org/jira/browse/PIG-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692379#comment-13692379
]
Daniel Dai commented on PIG-3347:
---------------------------------
That's the incorrect PushUpFilter. Can be solved by disable PushUpFilter rule:
pig -t PushUpFilter -x local xxx.pig
Look at the logical plan:
{code}
c: (Name: LOStore Schema:
group#28:bytearray,a_distinct#29:bag{#30:tuple(#31:bytearray)})
|
|---b: (Name: LOForEach Schema:
group#28:bytearray,a_distinct#29:bag{#30:tuple(#31:bytearray)})
| |
| (Name: LOGenerate[false,false] Schema:
group#28:bytearray,a_distinct#29:bag{#30:tuple(#31:bytearray)})ColumnPrune:InputUids=[29,
28]ColumnPrune:OutputUids=[29, 28]
| | |
| | group:(Name: Project Type: bytearray Uid: 28 Input: 0 Column: (*))
| | |
| | a_distinct:(Name: Project Type: bag Uid: 29 Input: 1 Column: (*))
| |
| |---(Name: LOInnerLoad[0] Schema: group#28:bytearray)
| |
| |---a_distinct: (Name: LODistinct Schema: #31:bytearray)
| |
| |---1-7: (Name: LOForEach Schema: #31:bytearray)
| | |
| | (Name: LOGenerate[false] Schema: #31:bytearray)
| | | |
| | | (Name: Project Type: bytearray Uid: 31 Input: 0 Column:
(*))
| | |
| | |---(Name: LOInnerLoad[0] Schema: #31:bytearray)
| |
| |---a: (Name: LOInnerLoad[1] Schema: null)
|
|---c: (Name: LOFilter Schema: group#28:bytearray,a#29:bag{#36:tuple()})
| |
| (Name: Equal Type: boolean Uid: 35)
| |
| |---(Name: UserFunc(org.apache.pig.builtin.BagSize) Type: long Uid:
32)
| | |
| | |---a:(Name: Project Type: bag Uid: 29 Input: 0 Column: 1)
| |
| |---(Name: Cast Type: long Uid: 33)
| |
| |---(Name: Constant Type: int Uid: 33)
|
|---a_group: (Name: LOCogroup Schema:
group#28:bytearray,a#29:bag{#36:tuple()})
| |
| (Name: Project Type: bytearray Uid: 28 Input: 0 Column: 0)
|
|---a: (Name: LOLoad Schema: null)RequiredFields:null
{code}
Filter is pushed in front of foreach, which is wrong.
> Store invocation in local mode brings sire effect
> -------------------------------------------------
>
> Key: PIG-3347
> URL: https://issues.apache.org/jira/browse/PIG-3347
> Project: Pig
> Issue Type: Bug
> Components: grunt
> Affects Versions: 0.11
> Environment: local mode
> Reporter: Sergey
>
> The problem is that intermediate 'store' invocation "changes" the final store
> output. Looks like it brings some kind of side effect. We did use 'local'
> mode to run script
> here is the input data:
> 1
> 1
> Here is the script:
> {code}
> a = load 'test';
> a_group = group a by $0;
> b = foreach a_group {
> a_distinct = distinct a.$0;
> generate group, a_distinct;
> }
> --store b into 'b';
> c = filter b by SIZE(a_distinct) == 1;
> store c into 'out';
> {code}
> We expect output to be:
> 1 1
> The output is empty file.
> Uncomment {code}--store b into 'b';{code} line and see the diffrence.
> Yuo would get expected output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
