[
https://issues.apache.org/jira/browse/PIG-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johnny Zhang updated PIG-3279:
------------------------------
Attachment: PIG-3279-2.patch.txt
incomplete patch to show progress, the c's schema and its logical plan,
physical plan and mr pan is
{noformat}
c: {d::rank_a: long,d::name: chararray,d::industry: chararray,d::income: int}
#-----------------------------------------------
# New Logical Plan:
#-----------------------------------------------
c: (Name: LOStore Schema:
d::rank_a#42:long,d::name#30:chararray,d::industry#31:chararray,d::income#32:int)
|
|---c: (Name: LOForEach Schema:
d::rank_a#42:long,d::name#30:chararray,d::industry#31:chararray,d::income#32:int)
| |
| (Name: LOGenerate[true] Schema:
d::rank_a#42:long,d::name#30:chararray,d::industry#31:chararray,d::income#32:int)ColumnPrune:InputUids=[40]ColumnPrune:OutputUids=[32,
42, 31, 30]
| | |
| | d:(Name: Project Type: bag Uid: 40 Input: 0 Column: (*))
| |
| |---d: (Name: LORank Schema:
rank_a#42:long,name#30:chararray,industry#31:chararray,income#32:int)
| | |
| | income:(Name: Project Type: int Uid: 32 Input: 0 Column: 2)
| |
| |---a: (Name: LOInnerLoad[1] Schema:
name#30:chararray,industry#31:chararray,income#32:int)
|
|---b: (Name: LOCogroup Schema:
group#31:chararray,a#40:bag{#46:tuple(name#30:chararray,industry#31:chararray,income#32:int)})
| |
| industry:(Name: Project Type: chararray Uid: 31 Input: 0 Column: 1)
|
|---a: (Name: LOForEach Schema:
name#30:chararray,industry#31:chararray,income#32:int)
| |
| (Name: LOGenerate[false,false,false] Schema:
name#30:chararray,industry#31:chararray,income#32:int)ColumnPrune:InputUids=[32,
31, 30]ColumnPrune:OutputUids=[32, 31, 30]
| | |
| | (Name: Cast Type: chararray Uid: 30)
| | |
| | |---name:(Name: Project Type: bytearray Uid: 30 Input: 0
Column: (*))
| | |
| | (Name: Cast Type: chararray Uid: 31)
| | |
| | |---industry:(Name: Project Type: bytearray Uid: 31 Input:
1 Column: (*))
| | |
| | (Name: Cast Type: int Uid: 32)
| | |
| | |---income:(Name: Project Type: bytearray Uid: 32 Input: 2
Column: (*))
| |
| |---(Name: LOInnerLoad[0] Schema: name#30:bytearray)
| |
| |---(Name: LOInnerLoad[1] Schema: industry#31:bytearray)
| |
| |---(Name: LOInnerLoad[2] Schema: income#32:bytearray)
|
|---a: (Name: LOLoad Schema:
name#30:bytearray,industry#31:bytearray,income#32:bytearray)RequiredFields:null
#-----------------------------------------------
# Physical Plan:
#-----------------------------------------------
c: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-32
|
|---c: New For Each(true)[bag] - scope-31
| |
| RelationToExpressionProject[bag][*] - scope-15
| |
| |---New For Each(false,true)[tuple] - scope-30
| | |
| | Project[long][0] - scope-28
| | |
| | Project[bag][2] - scope-29
| |
| |---d: PORank[tuple] - scope-27
| | |
| | Project[int][0] - scope-24
| |
| |---d: POCounter[tuple] - scope-26
| | |
| | Project[int][0] - scope-24
| |
| |---d: POSort[tuple]() - scope-25
| | |
| | Project[int][0] - scope-24
| |
| |---New For Each(true,false)[tuple] - scope-23
| | |
| | Project[int][0] - scope-21
| | |
| | Project[bag][1] - scope-22
| |
| |---d: Package[tuple]{int} - scope-18
| |
| |---d: Global Rearrange[tuple] - scope-17
| |
| |---d: Local Rearrange[tuple]{int}(false) -
scope-19
| | |
| | Project[int][2] - scope-20
| |
| |---Project[bag][1] - scope-16
|
|---b: Package[tuple]{chararray} - scope-12
|
|---b: Global Rearrange[tuple] - scope-11
|
|---b: Local Rearrange[tuple]{chararray}(false) - scope-13
| |
| Project[chararray][1] - scope-14
|
|---a: New For Each(false,false,false)[bag] - scope-10
| |
| Cast[chararray] - scope-2
| |
| |---Project[bytearray][0] - scope-1
| |
| Cast[chararray] - scope-5
| |
| |---Project[bytearray][1] - scope-4
| |
| Cast[int] - scope-8
| |
| |---Project[bytearray][2] - scope-7
|
|---a:
Load(file:///home/xiaoyuz/PIG-trunk/pig/input1:org.apache.pig.builtin.PigStorage)
- scope-0
487 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File
concatenation threshold: 100 optimistic? false
505 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
505 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
#--------------------------------------------------
# Map Reduce Plan
#--------------------------------------------------
MapReduce node scope-33
Map Plan
b: Local Rearrange[tuple]{chararray}(false) - scope-13
| |
| Project[chararray][1] - scope-14
|
|---a: New For Each(false,false,false)[bag] - scope-10
| |
| Cast[chararray] - scope-2
| |
| |---Project[bytearray][0] - scope-1
| |
| Cast[chararray] - scope-5
| |
| |---Project[bytearray][1] - scope-4
| |
| Cast[int] - scope-8
| |
| |---Project[bytearray][2] - scope-7
|
|---a:
Load(file:///home/xiaoyuz/PIG-trunk/pig/input1:org.apache.pig.builtin.PigStorage)
- scope-0--------
Reduce Plan
c: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-32
|
|---c: New For Each(true)[bag] - scope-31
| |
| RelationToExpressionProject[bag][*] - scope-15
| |
| |---New For Each(false,true)[tuple] - scope-30
| | |
| | Project[long][0] - scope-28
| | |
| | Project[bag][2] - scope-29
| |
| |---d: PORank[tuple] - scope-27
| | |
| | Project[int][0] - scope-24
| |
| |---d: POCounter[tuple] - scope-26
| | |
| | Project[int][0] - scope-24
| |
| |---d: POSort[tuple]() - scope-25
| | |
| | Project[int][0] - scope-24
| |
| |---New For Each(true,false)[tuple] - scope-23
| | |
| | Project[int][0] - scope-21
| | |
| | Project[bag][1] - scope-22
| |
| |---d: Package[tuple]{int} - scope-18
| |
| |---d: Global Rearrange[tuple] - scope-17
| |
| |---d: Local Rearrange[tuple]{int}(false) -
scope-19
| | |
| | Project[int][2] - scope-20
| |
| |---Project[bag][1] - scope-16
|
|---b: Package[tuple]{chararray} - scope-12--------
Global sort: false
{noformat}
> Support nested RANK
> -------------------
>
> Key: PIG-3279
> URL: https://issues.apache.org/jira/browse/PIG-3279
> Project: Pig
> Issue Type: Improvement
> Reporter: Gianmarco De Francisci Morales
> Assignee: Johnny Zhang
> Attachments: PIG-3279-1.patch.txt, PIG-3279-2.patch.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
