What you are looking for is probably a workflow manager. It is more or
less independent from a cluster management system, such as Mesos.
Here is a suggestion for a tool shopping list:
https://github.com/spotify/luigi
https://azkaban.github.io/
https://github.com/airbnb/airflow
https://github.com/
The OP might also be interested in Stolos:
https://github.com/sailthru/stolos
combined with Relay: https://github.com/sailthru/relay
On Wed, Oct 7, 2015 at 8:15 AM, Clarke, Trevor wrote:
> I'm currently working on this sort of framework. Unfortunately, source is
> not currently available but t
I'm currently working on this sort of framework. Unfortunately, source is not
currently available but there is a plan to open source in the next couple of
months. I'm not sure if your need is immediate or if it can wait for a bit. The
framework handles jobs in docker containers with pre and post
I answer below...
On Wed, Oct 7, 2015 at 8:17 AM, Brian Candler wrote:
> On 07/10/2015 11:08, Pablo Cingolani wrote:
>
> It looks like you are looking for something like BDS
>
> http://pcingola.github.io/BigDataScript/
>
> It has the additional advantage that you can port your scripts seamles
On 07/10/2015 11:08, Pablo Cingolani wrote:
It looks like you are looking for something like BDS
http://pcingola.github.io/BigDataScript/
It has the additional advantage that you can port your scripts seamlessly
between Mesos and other cluster systems (SGE, PBS, Torque, etc.).
Yes, that looks
I think any pub/sub system(name it typical jms / rabbitmq/ kafka) etc would do
what you describe. All of them can be run as containers inside apache mess
cluster. Kafka has really good integration with MEsos and YARN and also is more
lightweight than a typical jus implementation.
regards
\n\m
O
Another great option is Cook: https://github.com/twosigma/Cook
Cook combines a simple REST API for batch jobs with sophisticated
fair-sharing and preemption features on Mesos. Tomorrow, at MesosCon
Europe, I'll be speaking about it in more detail. When we want to use
dependencies with Cook, we use
It looks like you are looking for something like BDS
http://pcingola.github.io/BigDataScript/
It has the additional advantage that you can port your scripts seamlessly
between Mesos and other cluster systems (SGE, PBS, Torque, etc.).
On Wed, Oct 7, 2015 at 7:05 AM, F21 wrote:
> I am also
I am also interested in something like this, although my requirements
are much more simpler.
I am interested in a work queue like beanstalkd that will allow me to
push to a queue from a web app and have workers to do things like send
emails, generate pdfs and resize images.
I have thought ab
On 07/10/2015 09:44, Nikolaos Ballas neXus wrote:
Maybe you need to read a bit :)
I have read plenty, including those you list, and I didn't find anything
which met my requirements. Again I apologise if I was not clear in my
question.
Spark has a very specific data model (RDDs) and applicati
Maybe you need to read a bit :) Hadoop/Spark are batch processing frameworks,
both can run on top of Mesos. If you want to do online processing the you have
the Apache Storm child. On the other hand super computer != distributed
computing. You referred to croons and I thought you were asking fo
On 07/10/2015 09:01, Nikolaos Ballas neXus wrote:
Check for Marathon
I don't see how Marathon does what I want. Maybe I wasn't clear enough
in explaining my requirements.
What I need is basically a supercomputer cluster where I can take a
large computation job, break it into lots of sub-tas
Check for Marathon
On 07 Oct 2015, at 09:56, Brian Candler
mailto:b.cand...@pobox.com>> wrote:
Are there any open-source job queue/batch systems which run under Mesos? I am
thinking of things like HTCondor, Torque etc.
The requirement is to be able to:
- define an overall job as a set of sub-ta
13 matches
Mail list logo
