This is great, thanks Tom. On Sat, Mar 28, 2015 at 6:52 AM, Tom Arnfeld <[email protected]> wrote:
> Hey everyone, > > > I thought it best to send an email to the list before merging and tagging > a 0.1.0 release for the Hadoop on Mesos framework. This release is for a > new feature we've been working on for quite some time, which allows Hadoop > TaskTrackers to be semi-terminated when they are idle, without destroying > any map output they may need to retain for running reduce tasks. > > > Essentially this means that over the lifetime of a job (one with more > map/reduce tasks than the size of the cluster) the ratio of map and reduce > slots can change, resulting in significantly better resource utilization, > because the map slots can be freed up after they have finished doing work. > > > If anyone is running Hadoop on Mesos or would be kind enough to contribute > to reviewing the code in the diff, or giving the branch a go on their > cluster, that would be very much appreciated! We've been running the patch > in production for several months and have seen some quite significant > performance gains with our type of workload. > > > The pull request is here https://github.com/mesos/hadoop/pull/33. > > > Feel free to get in touch if you have any questions! Thanks! > > > -- > > Tom Arnfeld > Developer // DueDil
