That's precisely it.
The framework has actually been around for quite some time, probably one of the very early Mesos PoC framework implementations. It's certainly very stable and get's the job done for us, and allows us to focus on progressing with more appropriate technologies like Spark (or YARN for Hadoop fans)! We'll probably be pushing a few more improvements in the short/medium term, again focussed around ensuring Hadoop MRv1 doesn't create shared cluster woes. FairScheduler job pools are next on the list. -- Tom Arnfeld Developer // DueDil On Saturday, Mar 28, 2015 at 2:51 pm, Jeff Schroeder <[email protected]>, wrote: Gotcha so this is just a better mesos framework for Hadoop and has nothing to do with the Myriad / Yarn stuff, which similarly, prevents you from having to statically setup a Hadoop cluster. https://mesosphere.com/2015/02/11/yarn-on-mesos-big-data/ Nice stuff On Saturday, March 28, 2015, Tom Arnfeld <[email protected]> wrote: To follow up, this is also a decent solution to a nasty problem in the current framework detailed here, https://github.com/mesos/hadoop/issues/32. -- Tom Arnfeld Developer // DueDil On Sat, Mar 28, 2015 at 2:40 PM, Jeff Schroeder <[email protected]> wrote: Does this have any pros / cons over Myriad, which runs Yarn on Mesos? Other than not requiring Yarn :) On Saturday, March 28, 2015, Tom Arnfeld <[email protected]> wrote: Hey everyone, I thought it best to send an email to the list before merging and tagging a 0.1.0 release for the Hadoop on Mesos framework. This release is for a new feature we've been working on for quite some time, which allows Hadoop TaskTrackers to be semi-terminated when they are idle, without destroying any map output they may need to retain for running reduce tasks. Essentially this means that over the lifetime of a job (one with more map/reduce tasks than the size of the cluster) the ratio of map and reduce slots can change, resulting in significantly better resource utilization, because the map slots can be freed up after they have finished doing work. If anyone is running Hadoop on Mesos or would be kind enough to contribute to reviewing the code in the diff, or giving the branch a go on their cluster, that would be very much appreciated! We've been running the patch in production for several months and have seen some quite significant performance gains with our type of workload. The pull request is here https://github.com/mesos/hadoop/pull/33. Feel free to get in touch if you have any questions! Thanks! -- Tom Arnfeld Developer // DueDil -- Text by Jeff, typos by iPhone -- Text by Jeff, typos by iPhone
