[
https://issues.apache.org/jira/browse/YARN-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962833#comment-16962833
]
kailiu_dev edited comment on YARN-9940 at 10/30/19 9:15 AM:
------------------------------------------------------------
YARN-8436 May not be suitable for this issure.
because :
1: node maybe be deleted, and if use {color:#000000}TreeSet{color} will occure
this exception:
{color:#FF0000}java.util.ConcurrentModificationException{color}
{color:#FF0000} at
java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909){color}
so: the current core is needed:
List<NodeId> nodeIdList = new ArrayList<NodeId>(nodes.keySet());
synchronized (this)
{ Collections.sort(nodeIdList, nodeAvailableResourceComparator); }
2: the other reason {color:#172b4d}not be suitable{color} to use
{color:#000000}TreeSet{color} is
when: compare(NodeId n1, NodeId n2)
if n1==n2 the resultList of TreeSet only keep one, this will not
suitable for continuous schedule, beause some node will not be
{color:#172b4d}schedule{color}d
{color:#172b4d}3:{color} in YARN-9940, we hold the sechdule lock to avoid
exception while sorting when some node'Available resources change
4: Comparison method violates its general is Caused by JDK， from jdk7, Sorting
algorithm is replaced with "TimSsort", about the continuous scheduling of
Fair sechdule , there you can use two solutions:
{color:#FF0000} one is : you can see YARN-9940, patch{color}
{color:#FF0000} two is: add jvm option:
{color:#333333}-Djava.util.Arrays.useLegacyMergeSort={color}{color:#d73a49}true{color}{color}
was (Author: kailiu_dev):
YARN-8436 May not be suitable for this issure.
because :
1: node maybe be deleted, and if use {color:#000000}TreeSet{color} will occure
this exception:
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
so: the current core is needed:
List<NodeId> nodeIdList = new ArrayList<NodeId>(nodes.keySet());
synchronized (this) {
Collections.sort(nodeIdList, nodeAvailableResourceComparator);
}
2: the other reason {color:#172b4d}not be suitable{color} to use
{color:#000000}TreeSet{color} is
when: compare(NodeId n1, NodeId n2)
if n1==n2 {color:#172b4d}the result
{color}{color:#172b4d}){color}{color:#007600} {color}
if n1==n2 the result ) the resultList of TreeSet only keep one, this will
not suitable for continuous schedule, beause some node will not be
{color:#172b4d}schedule{color}d
{color:#172b4d}3:{color} in YARN-9940, we hold the sechdule lock to avoid
exception while sorting when some node'Available resources change
*
4: Comparison method violates its general is Caused by JDK， from jdk7, Sorting
algorithm is replaced with "TimSsort", about the continuous scheduling of
Fair sechdule , there you can use two solutions:
one is : you can see YARN-9940, patch
two is: add jvm option:
{color:#333333}-Djava.util.Arrays.useLegacyMergeSort={color}{color:#d73a49}true{color}
> avoid continuous scheduling thread crashes while sorting nodes get
> 'Comparison method violates its general contract'
> --------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-9940
> URL: https://issues.apache.org/jira/browse/YARN-9940
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
> Affects Versions: 2.7.2
> Reporter: kailiu_dev
> Priority: Major
> Fix For: 3.2.0
>
> Attachments: 0001.patch
>
>
> 2019-10-16 09:14:51,215 ERROR
> org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
> Thread[FairSchedulerContinuousScheduling,5,main] threw an Exception.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:868)
> at java.util.TimSort.mergeAt(TimSort.java:485)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
> at java.util.TimSort.sort(TimSort.java:223)
> at java.util.TimSort.sort(TimSort.java:173)
> at java.util.Arrays.sort(Arrays.java:659)
> at java.util.Collections.sort(Collections.java:217)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:1117)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:296)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
