[
https://issues.apache.org/jira/browse/PIG-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated PIG-4874:
------------------------------------
Attachment: PIG-4874-2.patch
Thanks [~knoguchi] for catching that. Updated patch.
A production job had one vertex doing two replicate joins.
Replicate input 1 - 870424 records
Replicate input 2 - 847854 records
So there was definitely a big size increase with the failing job.
Retained Size of replicate tables in memory
Before PIG-4874 patch:
input 1 hashmap - 389,467,984
input 2 hashmap - 333,995,048
After PIG-4874 patch:
input 1 hashmap - 312,870,672
input 2 hashmap - 198,338,416 (More savings here because the join key is just
String whereas input 1 had tuple for join key)
> Remove schema tuple reference overhead for replicate join hashmap
> -----------------------------------------------------------------
>
> Key: PIG-4874
> URL: https://issues.apache.org/jira/browse/PIG-4874
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.16.0
>
> Attachments: PIG-4874-1.patch, PIG-4874-2.patch
>
>
> Currently even if pig.schematuple is set to false which is the default, the
> usage of TupleToMapKey and TuplesToSchemaTupleList instead of plain
> HashMap<Object, ArrayList<Tuple>> costs a lot of memory. Also key is
> currently converted to a tuple which is unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
