[
https://issues.apache.org/jira/browse/MAPREDUCE-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780882#action_12780882
]
Arun C Murthy commented on MAPREDUCE-1044:
------------------------------------------
This proposal seems like we are about to re-invent Torque by adding yet another
daemon to Hadoop Map-Reduce. You can also use HoD, along with features in
Torque, to have a few large clusters and move tasktrackers around (via Torque).
I do not have all the details, but I believe Torque can have custom monitoring
and can be used to do smarter (i.e. Map-Reduce aware) scheduling.
If isolation really is the end-goal one can use full VMs right now. The
capacity-scheduler, in conjunction with TaskController infrastructure in the
TaskTracker, has some of the features you want: it does monitoring of memory
consumed by the task process tree and ensures they do not go over a limit. Yes,
it's harder to do cpu/io monitoring - but it is something everyone is looking
to do. Your efforts in this space will be very useful to the whole community
at-large... as indicated by our collaboration on MAPREDUCE-220 and other
related jiras.
-1 for the direction proposed in this jira.
> Ability to automatically move machines from one MR compute cluster to another
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-1044
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1044
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: DCD.pdf
>
>
> We have multiple map-reduce clusters that provide different service and
> support levels for their users. We have seen that utilization of hardware
> resources are not optimized if we have a static partition of existing
> hardware resources into these separate MR clusters. It would be nice to have
> a automatic way to move nodes from one MR cluster to another based on load
> characteristics and configured policies. This JIRA will discuss some of the
> ideas and possible implementations of those ideas.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
