[
https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622780#action_12622780
]
Hemanth Yamijala commented on HADOOP-3759:
------------------------------------------
Incorporated review comments and synchronized with trunk.
bq. TaskTracker's private method findFreeVirtualMemory should be synchronized
on runningTasks.
Done. Synchronized it on the tasktracker object.
bq. Would be good if the resources in TaskTrackerStatus are of the form
<String, Writable> instead of <String, Long>.
I was not able to see how to make it work with Writables. The problem is with
reading back the objects. It seemed like we need to know what Writable the
object actually is, in order to read it back fully. That would have prevented
us from making it generic. There is an ObjectWritable, but that seems an
overkill.
What I've done instead is to create a small wrapper object that implements
Writable called ResourceStatus. This is used to encapsulate all fields that
need to be reported to the JT in the TaskTrackerStatus. Please comment if this
makes sense.
bq. Just seen that HADOOP-657 is committed. Do we want to put available disk
space also as part of the same resource map?
Modified the availableSpace as a field in ResourceStatus. Ari, can you please
check if this change looks OK ? I didn't see a test case in HADOOP-657 which I
could run.
bq. In an offline discussion with Devaraj, he suggested that I not modify the
job conf when localizing the task.
Done this. Rather than localizing, I calculate the default free memory when
required - for e.g in findFreeVirtualMemory.
> Provide ability to run memory intensive jobs without affecting other running
> tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-3759
> URL: https://issues.apache.org/jira/browse/HADOOP-3759
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-3759.patch, HADOOP-3759.patch, HADOOP-3759.patch,
> HADOOP-3759.patch, HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from
> affecting Hadoop daemons and other tasks running on a node. A related
> requirement is that users be provided an ability to run jobs which are memory
> intensive. The system must provide enough knobs to allow such jobs to be run
> while still maintaining the requirements of HADOOP-3581.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
