[
https://issues.apache.org/jira/browse/PIG-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174198#comment-14174198
]
Rohini Palaniswamy commented on PIG-4012:
-----------------------------------------
When you have lot of objects (which is usually the case), creating another list
and copying over the Spillable to another list adds both memory and time
penalty. And you especially do not want to create another big list when you are
handling a notification for high memory usage. The comment in the code already
calls out that the sort is not stable and is only a best effort.
{code}
/**
* We don't lock anything, so this sort may not be stable if a
WeakReference suddenly
* becomes null, but it will be close enough.
* Also between the time we sort and we use these spillables,
they
* may actually change in size - so this is just best effort
*/
{code}
Unless the job fails due to the error, I don't see a need to create another
copy of list. The next handleNotifications() should go through.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract! SpillableMemoryManager
> -----------------------------------------------------------------------------------------------------------
>
> Key: PIG-4012
> URL: https://issues.apache.org/jira/browse/PIG-4012
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.12.0
> Environment: java version "1.7.0_60-ea"
> Java(TM) SE Runtime Environment (build 1.7.0_60-ea-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 24.60-b04, mixed mode)
> Reporter: David Dreyfus
> Assignee: David Dreyfus
> Fix For: 0.13.0
>
>
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:868)
> at java.util.TimSort.mergeAt(TimSort.java:485)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
> at java.util.TimSort.sort(TimSort.java:223)
> at java.util.TimSort.sort(TimSort.java:173)
> at java.util.Arrays.sort(Arrays.java:659)
> at java.util.Collections.sort(Collections.java:217)
> at
> org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:199)
> at
> sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:156)
> at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
> at
> sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:301)
> at sun.management.Sensor.trigger(Sensor.java:137)
> From SpillableMemoryManager.java:
> /**
> * We don't lock anything, so this sort may not be stable if
> a WeakReference suddenly
> * becomes null, but it will be close enough.
> * Also between the time we sort and we use these spillables,
> they
> * may actually change in size - so this is just best effort
> */
> Issue may be due to Java 7 and reporting vs ignoring the exception.
> Trying
> -Djava.util.Arrays.useLegacyMergeSort=true
> http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6804124
> suggests the newer MergeSort is much faster.
> Someone may want to make the sorting stable in SpillableMemoryManager so that
> the new merge sort can be used without failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
