Maybe you need to read a bit :) Hadoop/Spark are batch processing frameworks, both can run on top of Mesos. If you want to do online processing the you have the Apache Storm child. On the other hand super computer != distributed computing. You referred to croons and I thought you were asking for a scheduler. You need to read maybe a bit to understand the technology stack, cause the answers to your question are rather obvious for a guy with ds background, even basic, following the market. Jobs can be either executed with Hadoop executors or delegate jobs to processes configured in docker containers that mess can bootstrap.
kind regards \n\m On 07 Oct 2015, at 10:37, Brian Candler <b.cand...@pobox.com<mailto:b.cand...@pobox.com>> wrote: On 07/10/2015 09:01, Nikolaos Ballas neXus wrote: Check for Marathon I don't see how Marathon does what I want. Maybe I wasn't clear enough in explaining my requirements. What I need is basically a supercomputer cluster where I can take a large computation job, break it into lots of sub-tasks, and run as many of those sub-tasks in parallel as possible given the CPU resources available, until all the sub-tasks are done. The core of any sort of system like that is a "job queue" where all the sub-tasks are entered. The executor picks out another task whenever there is some free resource available, and when it finishes, it is removed from the queue. I don't see how Marathon has such a job queue. As far as I can tell, Marathon is for starting long-lived applications; you define what things you want running, it starts them, and restarts them if they die for any reason. Or have I misunderstood what Marathon is capable of? If so, can you point me at the relevant documentation? The advantage of running such a supercomputer cluster under Mesos would be that I could run *other* applications (including those started by Marathon or Chronos) on the same hardware. Thanks, Brian. Nikolaos Ballas | Software Development Manager Technology Nexus S.a.r.l. 2-4 Rue Eugene Rupert 2453 Luxembourg Delivery address: 2-3 Rue Eugene Rupert,Vertigo Polaris Building Tel: + 3522619113580 cont...@nexusgroup.com<mailto:contact...@nexusgroup.com> | nexusgroup.com<http://www.nexusgroup.com/> LinkedIn.com<http://www.linkedin.com/company/nexus-technology> | Twitter<http://www.twitter.com/technologynexus> | Facebook.com<https://www.facebook.com/pages/Technology-Nexus/133756470003189> [cid:87987ACD-6CF7-41BE-9517-E612DBF86ABA@pwcacc.com] \
