[
https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Yamijala updated HADOOP-3759:
-------------------------------------
Attachment: HADOOP-3759.patch
Initial patch for review and to encourage discussion
The patch incorporates the approach mentioned in the earlier comments.
Specifically:
- Includes the configuration variables described and defines accessors in
JobConf
- Includes a Map for storing resource information in TaskTrackerStatus. Though
there is only one entry currently (free memory), this map is an attempt to keep
the wire protocol the same even in future when more resources are to be added.
I've currently used the value of these resources as Long, but possibly this
should be changed to a Writable so we can pass anything
- Defines a method to compute minimum free space that would be available for a
new task in TaskTracker (using the configuration variables for currently
running tasks), and setting that value in TaskTrackerStatus.
- Accessors in JobInProgress to define memory requirements for the job.
With these changes in place, schedulers such as HADOOP-3445 can decide to
schedule tasks according to the job's memory requirements, and the free memory
available on a tasktracker.
Please provide feedback on this implementation.
> Provide ability to run memory intensive jobs without affecting other running
> tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-3759
> URL: https://issues.apache.org/jira/browse/HADOOP-3759
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from
> affecting Hadoop daemons and other tasks running on a node. A related
> requirement is that users be provided an ability to run jobs which are memory
> intensive. The system must provide enough knobs to allow such jobs to be run
> while still maintaining the requirements of HADOOP-3581.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
