[
https://issues.apache.org/jira/browse/PIG-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173060#comment-14173060
]
Rohini Palaniswamy commented on PIG-4012:
-----------------------------------------
This should just be a temporary exception right? Looking at java source code it
just seems to log the message if there is an Exception from
handleNotifications(). Are you saying that it actually fails the job for you.
I would expect the next handleNotifications call will go through and spill. We
have been running with jdk 7 for 1.5+ years now and have never seen a job fail
with this exception.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract! SpillableMemoryManager
> -----------------------------------------------------------------------------------------------------------
>
> Key: PIG-4012
> URL: https://issues.apache.org/jira/browse/PIG-4012
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.12.0
> Environment: java version "1.7.0_60-ea"
> Java(TM) SE Runtime Environment (build 1.7.0_60-ea-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 24.60-b04, mixed mode)
> Reporter: David Dreyfus
> Assignee: David Dreyfus
> Fix For: 0.13.0
>
>
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:868)
> at java.util.TimSort.mergeAt(TimSort.java:485)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
> at java.util.TimSort.sort(TimSort.java:223)
> at java.util.TimSort.sort(TimSort.java:173)
> at java.util.Arrays.sort(Arrays.java:659)
> at java.util.Collections.sort(Collections.java:217)
> at
> org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:199)
> at
> sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:156)
> at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168)
> at
> sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:301)
> at sun.management.Sensor.trigger(Sensor.java:137)
> From SpillableMemoryManager.java:
> /**
> * We don't lock anything, so this sort may not be stable if
> a WeakReference suddenly
> * becomes null, but it will be close enough.
> * Also between the time we sort and we use these spillables,
> they
> * may actually change in size - so this is just best effort
> */
> Issue may be due to Java 7 and reporting vs ignoring the exception.
> Trying
> -Djava.util.Arrays.useLegacyMergeSort=true
> http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6804124
> suggests the newer MergeSort is much faster.
> Someone may want to make the sorting stable in SpillableMemoryManager so that
> the new merge sort can be used without failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
