Joost Roeleveld <joost <at> antarean.org> writes:
> > Mesos looks promising for a variety of (Apache) reasons. Some key > > technologies folks may want google about that are related: > > > > Quincy (fair schedular) > > Chronos (scheduler) > > Hadoop (scheduler) > > Hadoop not a scheduler. It's a framework for a Big Data clustered > database. > > HDFS (clusterd file system) > Unless it's changed recently, not suitable for anything else then Hadoop > and contains a single point of failure. I'm curious as to more information about this 'single point of failure. Can you be more specific or provides links? On this resource: http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithQJM.html JournalNode machines talks about surviving faults: "increase the number of failures the system can tolerate, you should run an odd number of JNs, (i.e. 3, 5, 7, etc.). Note that when running with N JournalNodes, the system can tolerate at most (N - 1) / 2 failures and continue to function normally. " > > > http://gpo.zugaina.org/sys-cluster/apache-hadoop-common > > > > Zookeeper (Fault tolerance) > > SPARK ( optimized for interative jobs where a datase is resued in many > > parallel operations (advanced math/science and many other apps.) > > https://spark.apache.org/ > > > > Dryad Torque Mpiche2 MPI > > Globus tookit > > > > mesos_tech_report.pdf > > > > It looks as though Amazon, google, facebook and many others > > large in the Cluster/Cloud arena are using Mesos......? > > > > So let's all post what we find, particularly in overlays. > > Unless you are dealing with Big Data projects, like Google, Facebook, Amazon, big banks,... you don't have much use for those projects. Many scientific applications are using the cluster (cloud) or big data approach to all sorts of problems. Furthermore, as GPU and the new Arm systems with dozens and dozens of cpu cores inside one computer become readily available, the cluster-cloud (big data) approach will become much more pervasive in the next few years, imho. http://blog.rescale.com/reservoir-simulation-moves-to-the-cloud/ There are thousands of small companies needing reservoir simulation, not to mention the millions of folks working on carbon sequestration..... Anything to do with Biological or Chemical Science is using or moving to the Cloud-Clustered world. For me, a Cluster is just a cloud internally managee, rather than outsourcing it to others; ymmv. > Mesos looks like a nice project, just like Hadoop and related are also > nice. But for most people, they are as usefull as using Exalytics. I'm not excited about an Oracle solution to anything. Many of the folks I know consult on moving technologies away from Oracle's spear of influence, not limited to mysql; ymmv. I know of one very large communications company that went broke and had to merge because of those ridiculous Oracle fees. Caveat Emptor; long live Postresql. > A scheduler should not have a large set of dependencies that you wouldn't > use otherwise. That makes Chronos a non-option to me. Those other technologies are often useful to folks who would be attracted to something like chronos. > Martin's project looks promising, but doesn't store the schedules > internally. For repeating schedules, like what Alan was describing, you > need to put those into scripts and start those from an existing cron. > Of the 2, I think improving Martin's project is the most likely option > for me as it doesn't have additional dependencies and seems to be > easily implemented. > Joost Understood. Like others, I'll be curious to follow what develops out of Martin's work. For me Chronos, Mesos and the other aforementioned technologies look to be more viable; particularly if one is preparing for a clustered world with CPUs, GPUs, SoCs and Arm machines distributed about the ethernet as resources to be scheduled and utilized in a variety of schema. It's the quest for one-infrastructure to solve many problems where scenarios compete. Big data is not the only reason for cloud-clusters. Theoretically, (Clustered) systems can have a far greater resource utilization of networked resources than traditional (distributed) approaches. I grant you that this is a work in progress, but I personally know of dozens of mathematically complex distributed systems that are migrating to the clustered approach rather than something custom or traditionally distributed. Granted, Cloud <--> Clustered <--> Distributed are all overlaping approaches to big problems. I do appreciate the candor of this thread. James
