Re: Scheduler for distributed builds
Thanks everyone for the responses. I'll look into the different options (Cook, Aurora and Chronos). Thx, -Paulo On Wed, Apr 27, 2016 at 3:51 PM, David Palaitis <david.j.palai...@gmail.com> wrote: > Cookies a fit in the case where you have more jobs than resources > available to run them. It manages large job queues, prioritizes jobs and > balances resources fairly across users, or roles, etc. > > Cook is dfntly interesting for a build farm e.g distributed Basel, > although the scheduling overhead for the job sizes you mention doesn't seem > worth it. If you were to do parallelism at the code base level then I > could see a possible fit with cook. > > Alternatively. schedule a set of long running build workers with marathon > and proxy work requests through a Kafka. > > I'd be happy to discuss in more detail if you think Cook might be a fit. > > Sent from my iPhone > > On Apr 27, 2016, at 6:06 PM, Erb, Stephan <stephan@blue-yonder.com> > wrote: > > FWIW, Apache Aurora is also supporting ad-hoc jobs. In contrast to to > chronos however, only without job dependencies. > -- > *From:* Guangya Liu <gyliu...@gmail.com> > *Sent:* Wednesday, April 27, 2016 09:59 > *To:* user@mesos.apache.org > *Subject:* Re: Scheduler for distributed builds > > The Chronos may also help for your case http://mesos.github.io/chronos/ > > On Wed, Apr 27, 2016 at 6:40 AM, David Greenberg <dsg123456...@gmail.com> > wrote: > >> http://github.com/twosigma/cook could be a good fit for this. It >> supports scheduling arbitrary jobs within seconds of submission, and it has >> advanced QoS features. >> >> On Tue, Apr 26, 2016 at 3:13 PM Paulo Gallo <pga...@arista.com> wrote: >> >>> Hi, >>> >>> I'm looking for a scheduler to do distributed builds, i.e. most of the >>> tasks would be short lived, like a few hundred ms long. >>> >>> Is there's any Mesos based scheduler that would be a good fit for this? >>> >>> Any help is appreciated. >>> >>> Thanks, >>> -Paulo >>> >>> PS: I know that Jenkins supports distributed builds and integrates with >>> Mesos, but we're looking for alternatives. >>> >>> >
Re: Scheduler for distributed builds
Cookies a fit in the case where you have more jobs than resources available to run them. It manages large job queues, prioritizes jobs and balances resources fairly across users, or roles, etc. Cook is dfntly interesting for a build farm e.g distributed Basel, although the scheduling overhead for the job sizes you mention doesn't seem worth it. If you were to do parallelism at the code base level then I could see a possible fit with cook. Alternatively. schedule a set of long running build workers with marathon and proxy work requests through a Kafka. I'd be happy to discuss in more detail if you think Cook might be a fit. Sent from my iPhone > On Apr 27, 2016, at 6:06 PM, Erb, Stephan <stephan@blue-yonder.com> wrote: > > FWIW, Apache Aurora is also supporting ad-hoc jobs. In contrast to to > chronos however, only without job dependencies. > From: Guangya Liu <gyliu...@gmail.com> > Sent: Wednesday, April 27, 2016 09:59 > To: user@mesos.apache.org > Subject: Re: Scheduler for distributed builds > > The Chronos may also help for your case http://mesos.github.io/chronos/ > >> On Wed, Apr 27, 2016 at 6:40 AM, David Greenberg <dsg123456...@gmail.com> >> wrote: >> http://github.com/twosigma/cook could be a good fit for this. It supports >> scheduling arbitrary jobs within seconds of submission, and it has advanced >> QoS features. >> >>> On Tue, Apr 26, 2016 at 3:13 PM Paulo Gallo <pga...@arista.com> wrote: >>> Hi, >>> >>> I'm looking for a scheduler to do distributed builds, i.e. most of the >>> tasks would be short lived, like a few hundred ms long. >>> >>> Is there's any Mesos based scheduler that would be a good fit for this? >>> >>> Any help is appreciated. >>> >>> Thanks, >>> -Paulo >>> >>> PS: I know that Jenkins supports distributed builds and integrates with >>> Mesos, but we're looking for alternatives. >
Re: Scheduler for distributed builds
?FWIW, Apache Aurora is also supporting ad-hoc jobs. In contrast to to chronos however, only without job dependencies. From: Guangya Liu <gyliu...@gmail.com> Sent: Wednesday, April 27, 2016 09:59 To: user@mesos.apache.org Subject: Re: Scheduler for distributed builds The Chronos may also help for your case http://mesos.github.io/chronos/ On Wed, Apr 27, 2016 at 6:40 AM, David Greenberg <dsg123456...@gmail.com<mailto:dsg123456...@gmail.com>> wrote: http://github.com/twosigma/cook could be a good fit for this. It supports scheduling arbitrary jobs within seconds of submission, and it has advanced QoS features. On Tue, Apr 26, 2016 at 3:13 PM Paulo Gallo <pga...@arista.com<mailto:pga...@arista.com>> wrote: Hi, I'm looking for a scheduler to do distributed builds, i.e. most of the tasks would be short lived, like a few hundred ms long. Is there's any Mesos based scheduler that would be a good fit for this? Any help is appreciated. Thanks, -Paulo PS: I know that Jenkins supports distributed builds and integrates with Mesos, but we're looking for alternatives.
Re: Scheduler for distributed builds
The Chronos may also help for your case http://mesos.github.io/chronos/ On Wed, Apr 27, 2016 at 6:40 AM, David Greenberg <dsg123456...@gmail.com> wrote: > http://github.com/twosigma/cook could be a good fit for this. It supports > scheduling arbitrary jobs within seconds of submission, and it has advanced > QoS features. > > On Tue, Apr 26, 2016 at 3:13 PM Paulo Gallo <pga...@arista.com> wrote: > >> Hi, >> >> I'm looking for a scheduler to do distributed builds, i.e. most of the >> tasks would be short lived, like a few hundred ms long. >> >> Is there's any Mesos based scheduler that would be a good fit for this? >> >> Any help is appreciated. >> >> Thanks, >> -Paulo >> >> PS: I know that Jenkins supports distributed builds and integrates with >> Mesos, but we're looking for alternatives. >> >>
Re: Scheduler for distributed builds
http://github.com/twosigma/cook could be a good fit for this. It supports scheduling arbitrary jobs within seconds of submission, and it has advanced QoS features. On Tue, Apr 26, 2016 at 3:13 PM Paulo Gallo <pga...@arista.com> wrote: > Hi, > > I'm looking for a scheduler to do distributed builds, i.e. most of the > tasks would be short lived, like a few hundred ms long. > > Is there's any Mesos based scheduler that would be a good fit for this? > > Any help is appreciated. > > Thanks, > -Paulo > > PS: I know that Jenkins supports distributed builds and integrates with > Mesos, but we're looking for alternatives. > >
Scheduler for distributed builds
Hi, I'm looking for a scheduler to do distributed builds, i.e. most of the tasks would be short lived, like a few hundred ms long. Is there's any Mesos based scheduler that would be a good fit for this? Any help is appreciated. Thanks, -Paulo PS: I know that Jenkins supports distributed builds and integrates with Mesos, but we're looking for alternatives.
