[
https://issues.apache.org/jira/browse/MAPREDUCE-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anubhav Dhoot updated MAPREDUCE-5966:
-------------------------------------
Status: Patch Available (was: Open)
Submitting patch that does not allow changes in the weight when sorting starts.
All updates to weight are done at the same time and in the same way demands and
other changes are updated.
> MR1 FairScheduler use of custom weight adjuster is not thread safe for
> comparisons
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5966
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5966
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.2.1
> Reporter: Anubhav Dhoot
> Assignee: Anubhav Dhoot
>
> When comparing JobSchedulables one of the factors is the weight. If someone
> uses a custom weight adjuster, that may be called multiple times during a
> sort causing different values to return. That causes a failure in sorting
> because the weight may change during the sort.
> This reproes as
> {code}
> java.io.IOException: java.lang.IllegalArgumentException: Comparison method
> violates its general contract!
> at java.util.TimSort.mergeHi(TimSort.java:868)
> at java.util.TimSort.mergeAt(TimSort.java:485)
> at java.util.TimSort.mergeCollapse(TimSort.java:410)
> at java.util.TimSort.sort(TimSort.java:214)
> at java.util.TimSort.sort(TimSort.java:173)
> at java.util.Arrays.sort(Arrays.java:659)
> at java.util.Collections.sort(Collections.java:217)
> at
> org.apache.hadoop.mapred.PoolSchedulable.assignTask(PoolSchedulable.java:163)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:499)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2961)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
