[
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3451:
-------------------------------
Attachment: MAPREDUCE-3451.v5.patch
Per MAPREDUCE-3812, I've tweaked your patch a bit to make it use the new
configs for min/max RM allocation (memory for containers).
One thing I noticed when tweaking your patch, is that there are quite a bit of
trailing white-spaces and possibly a few occurrences of tab+space mix you may
wanna clean up.
The diff I've added to your patch overall is below:
{code}
diff --git
a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
index 35eafc5..e888be8 100644
---
a/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
+++
b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
@@ -4,6 +4,7 @@ import java.io.File;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.server.resourcemanager.resource.Resources;
public class FairSchedulerConfiguration extends Configuration {
@@ -13,15 +14,7 @@ public class FairSchedulerConfiguration extends
Configuration {
protected static final String ALLOCATION_FILE = CONF_PREFIX +
"allocation.file";
protected static final String EVENT_LOG_DIR = "eventlog.dir";
-
- /** The minimum size container the scheduler will ever allocate. */
- protected static final String MINIMUM_MEMORY_MB = CONF_PREFIX +
"minimum-allocation-mb";
- protected static final int DEFAULT_MINIMUM_MEMORY_MB = 512;
-
- /** The maximum size container the scheduler will ever allocate. */
- protected static final String MAXIMUM_ALLOCATION_MB = CONF_PREFIX +
"maximum-allocation-mb";
- protected static final int DEFAULT_MAXIMUM_ALLOCATION_MB = 10240;
-
+
/** Whether to use the user name as the queue name (instead of "default") if
* the request does not specify a queue. */
protected static final String USER_AS_DEFAULT_QUEUE = CONF_PREFIX +
"user-as-default-queue";
@@ -60,17 +53,21 @@ public class FairSchedulerConfiguration extends
Configuration {
super(conf);
addResource(FS_CONFIGURATION_FILE);
}
-
+
public Resource getMinimumMemoryAllocation() {
- int mem = getInt(MINIMUM_MEMORY_MB, DEFAULT_MINIMUM_MEMORY_MB);
+ int mem = getInt(
+ YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB,
+ YarnConfiguration.DEFAULT_RM_SCHEDULER_MINIMUM_ALLOCATION_MB);
return Resources.createResource(mem);
}
-
+
public Resource getMaximumMemoryAllocation() {
- int mem = getInt(MAXIMUM_ALLOCATION_MB, DEFAULT_MAXIMUM_ALLOCATION_MB);
+ int mem = getInt(
+ YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB,
+ YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_MB);
return Resources.createResource(mem);
}
-
+
public boolean getUserAsDefaultQueue() {
return getBoolean(USER_AS_DEFAULT_QUEUE, DEFAULT_USER_AS_DEFAULT_QUEUE);
}
{code}
bq. Okay looks like that patch fixed everything except for the findbugs, there
are some member variables which are written only while holding a lock but can
be read outside of the lock and findbugs doesn't like this. I consider these
false positives - any idea how to suppress?
I didn't take a look at which specific mem-var you're discussing here, but what
happens if one reads it when its being updated? Does your locking mech cover
this scenario too? I remember making the same mistake a while ago at
https://issues.apache.org/jira/browse/MAPREDUCE-1347?focusedCommentId=13003257&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13003257
but that may be irrelevant here.
In case you wish to add it to ignores, see file
{{hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml}} and
add appropriate entries for your discovered warnings to suppress them. I'd
recommend suppressing them after the review/over another ticket and just leave
as a comment for now that this may not be harmful as it looks.
> Port Fair Scheduler to MR2
> --------------------------
>
> Key: MAPREDUCE-3451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2, scheduler
> Reporter: Patrick Wendell
> Assignee: Patrick Wendell
> Attachments: MAPREDUCE-3451.v1.patch.txt,
> MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt,
> MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet
> ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and
> will be updated to include design considerations and progress.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
