[
https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated HADOOP-4035:
------------------------------
Attachment: HADOOP-4035-20081126.1.txt
Attaching a new patch.
- Fixed the testcase TestCapacityScheduler and added tests to verify memory
based scheduling.
- Free memory computation is moved to the scheduler from the TT, so removed
TestHighRAMJobs and included TestTTMemoryReporting that verifies that memory
values are reported correctly by the TT.
- While writing test-cases found and fixed a bug in
JobQueuesManager.jobCompleted (+146).
{code}
if (qi.runningJobs.remove(oldInfo) != null) {
qi.waitingJobs.remove(oldInfo);
}
{code}
Should be
{code}
if (qi.runningJobs.remove(oldInfo) == null) {
qi.waitingJobs.remove(oldInfo);
}
{code}
This definitely needs to be fixed in previous versions too.
- Added killJob(JobID) and getJob(JobID) methods to TaskTrackerManager.
Scheduler needs access to these methods. This is an interface change and so I
had to modify TestJobQueueTaskScheduler and TestFairScheduler too. Should we
just change TaskTrackerManager to be an abstract class, may be in another JIRA?
- Minor documentation fixes and code-refactoring over the previous patch.
> Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory
> requirements and task trackers free memory
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4035
> URL: https://issues.apache.org/jira/browse/HADOOP-4035
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 4035.1.patch, HADOOP-4035-20080918.1.txt,
> HADOOP-4035-20081006.1.txt, HADOOP-4035-20081006.txt,
> HADOOP-4035-20081008.txt, HADOOP-4035-20081121.txt, HADOOP-4035-20081126.1.txt
>
>
> HADOOP-3759 introduced configuration variables that can be used to specify
> memory requirements for jobs, and also modified the tasktrackers to report
> their free memory. The capacity scheduler in HADOOP-3445 should schedule
> tasks based on these parameters. A task that is scheduled on a TT that uses
> more than the default amount of memory per slot can be viewed as effectively
> using more than one slot, as it would decrease the amount of free memory on
> the TT by more than the default amount while it runs. The scheduler should
> make the used capacity account for this additional usage while enforcing
> limits, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
