[
https://issues.apache.org/jira/browse/HADOOP-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651974#action_12651974
]
Hemanth Yamijala commented on HADOOP-4035:
------------------------------------------
Looking good. Few comments:
- ProcfsBasedProcessTree.getCumulativeVmem is still dividing the values by
1024, and hence treating the values as kB.
- Job recovery which is calling jobAdded will fail if any job has an invalid
memory specification due to the new IOException being thrown. Though this is an
unlikely case, in general, I think job recovery shouldn't fail due to one
faulty job.
- InterTrackerProtocol's version should change
- Some log statements at info level would be too verbose. Recommend moving them
to debug level. Including:
-- initializing jobid statement in JobInProgress
-- info statement in matchesMemoryRequirements
- The memory related APIs in JobInProgress need not be public, no ?
- For the MemoryPlugin, I think we should use the ReflectionUtils mechanism. By
default, the config value for this could be null, and this would mean that we
would use the operating system specific class. If it is not null, then am
object of the appropriate class would be used. Then the class that you are
using for testing purposes can move to the test code and need not be in the
production code.
- Properties in the capacity scheduler conf should be defined in
capacity-scheduler.xml.template and should be read from the capacity scheduler
conf object and not from JT's configuration object.
- I would suggest a few changes to the names in TaskLookupStatus, such as
TASK_FOUND, NO_TASK_IN_JOB, NO_TASK_IN_QUEUE,
NO_TASK_MATCHING_MEMORY_REQUIREMENTS
> Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory
> requirements and task trackers free memory
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4035
> URL: https://issues.apache.org/jira/browse/HADOOP-4035
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.19.0
> Reporter: Hemanth Yamijala
> Assignee: Vinod K V
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: 4035.1.patch, HADOOP-4035-20080918.1.txt,
> HADOOP-4035-20081006.1.txt, HADOOP-4035-20081006.txt,
> HADOOP-4035-20081008.txt, HADOOP-4035-20081121.txt,
> HADOOP-4035-20081126.1.txt, HADOOP-4035-20081128-4.txt
>
>
> HADOOP-3759 introduced configuration variables that can be used to specify
> memory requirements for jobs, and also modified the tasktrackers to report
> their free memory. The capacity scheduler in HADOOP-3445 should schedule
> tasks based on these parameters. A task that is scheduled on a TT that uses
> more than the default amount of memory per slot can be viewed as effectively
> using more than one slot, as it would decrease the amount of free memory on
> the TT by more than the default amount while it runs. The scheduler should
> make the used capacity account for this additional usage while enforcing
> limits, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
