[
https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621466#action_12621466
]
Hemanth Yamijala commented on HADOOP-3759:
------------------------------------------
The latest patch adds a JUnit test class that works as follows:
- The test class defines a custom TaskScheduler that extends
JobQueueTaskScheduler. This is only a dummy scheduler, that overrides the
assignTasks method in which it verifies the values of the memory related
variables that are reported by the tasktrackers. This is the core of the test.
- It uses the SleepJob example as it just needs a dummy job to run.
- Each test case sets up a MiniMRCluster to use the custom TaskScheduler, adds
a SleepJob to it, and configures various values for the memory related
configuration variables defined above.
- Then it lets the schedule's assignTasks API to verify that the free memory
etc are computed and reported correctly to the JobTracker.
In order for this test to work, I had to make a few changes to some core
classes. Please comment on whether these are reasonable. The changes are:
- In JobTracker, defined a package-private API {code}TaskScheduler
getTaskScheduler(){code}
- In MiniMRCluster, defined an API to get the JobTracker instance that is
created
- Again in MiniMRCluster, starting the TaskTracker with a configured JobConf
variable, rather than null. I needed this because the TaskTracker should get
the memory related configuration variables.
> Provide ability to run memory intensive jobs without affecting other running
> tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-3759
> URL: https://issues.apache.org/jira/browse/HADOOP-3759
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-3759.patch, HADOOP-3759.patch, HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from
> affecting Hadoop daemons and other tasks running on a node. A related
> requirement is that users be provided an ability to run jobs which are memory
> intensive. The system must provide enough knobs to allow such jobs to be run
> while still maintaining the requirements of HADOOP-3581.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
