On Saturday 02 August 2014 16:53:26 James wrote: > Alan McKinnon <alan.mckinnon <at> gmail.com> writes: > > Well, we've found 2 projects that at least in part seek to achieve our > > general goals - chronos and Martin's new project. > > Why don't we both fool around with them for a bit and get a sense of > > what it will take to add features etc? Then we can meet back here and > > discuss. Always better to build on an existing foundation > > Mesos looks promising for a variety of (Apache) reasons. Some key > technologies folks may want google about that are related: > > Quincy (fair schedular) > Chronos (scheduler) > Hadoop (scheduler)
Hadoop not a scheduler. It's a framework for a Big Data clustered database. > HDFS (clusterd file system) Unless it's changed recently, not suitable for anything else then Hadoop and contains a single point of failure. > http://gpo.zugaina.org/sys-cluster/apache-hadoop-common > > Zookeeper (Fault tolerance) > SPARK ( optimized for interative jobs where a datase is resued in many > parallel operations (advanced math/science and many other apps.) > https://spark.apache.org/ > > Dryad Torque Mpiche2 MPI > Globus tookit > > mesos_tech_report.pdf > > It looks as though Amazon, google, facebook and many others > large in the Cluster/Cloud arena are using Mesos......? > > So let's all post what we find, particularly in overlays. Unless you are dealing with Big Data projects, like Google, Facebook, Amazon, big banks,... you don't have much use for those projects. Mesos looks like a nice project, just like Hadoop and related are also nice. But for most people, they are as usefull as using Exalytics. A scheduler should not have a large set of dependencies that you wouldn't use otherwise. That makes Chronos a non-option to me. Martin's project looks promising, but doesn't store the schedules internally. For repeating schedules, like what Alan was describing, you need to put those into scripts and start those from an existing cron. Of the 2, I think improving Martin's project is the most likely option for me as it doesn't have additional dependencies and seems to be easily implemented. -- Joost
