[
https://issues.apache.org/jira/browse/PIG-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated PIG-5385:
------------------------------
Fix Version/s: 0.18.0
> Skip calling extra gc() before spilling large bag when unnecessary
> ------------------------------------------------------------------
>
> Key: PIG-5385
> URL: https://issues.apache.org/jira/browse/PIG-5385
> Project: Pig
> Issue Type: Improvement
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Trivial
> Fix For: 0.18.0
>
> Attachments: pig-5385-v01.patch
>
>
> In order to determine when to start spilling bags, Pig uses
> MemoryNotification for both MEMORY_THRESHOLD_EXCEEDED and
> MEMORY_COLLECTION_THRESHOLD_EXCEEDED.
> https://docs.oracle.com/javase/8/docs/api/java/lang/management/MemoryNotificationInfo.html
> Since spilling a large bag is expensive, Pig explicitly call System.gc() when
> the expected size is huge. I think we can skip this step when notification
> is based on MEMORY_COLLECTION_THRESHOLD_EXCEEDED since this means jvm has
> called the gc already.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
