Hi Oliver, GoDocker looks very promising. I have been looking for a a framework to substitute AWS Lambda for batch jobs scheduling based on docker for flexibility. That said, I would like to know your thought on this to better understand if GoDocker could do the job. A couple of things I would need to know better:
- How fast can GoDocker launch a job? Can GoDocker pre populate all nodes with
the container and files needed for the job to achieve <500ms launches? I
realize this depends a lot on the Job size, just looking for a generic view
before getting into full testing.
- Is it possible to extend GoDocker to enable usage metrics reports? This could
be achieved by simply logging a few metrics relating to the user and job.
{ name: “João Ribeiro”,
email: “[email protected] <mailto:[email protected]>”,
phone: “+351 916 002 675 <tel:+351916002675>”,
skype: “jmn_ribeiro <skype:jmn_ribeiro?add>”,
linkedin: “https://pt.linkedin.com/in/jmnribeiro
<https://pt.linkedin.com/in/jmnribeiro>”
github: “https://github.com/JonnyBGod <https://github.com/JonnyBGod>” }
> On 09 Dec 2015, at 10:22, Elouan Keryell-Even <[email protected]>
> wrote:
>
> Thanks for the insight, very interesting indeed.
>
> Elouan Keryell-Even
> Software Engineer @ Atos Integration
>
> Toulouse, France
>
>
> 2015-12-09 11:08 GMT+01:00 Olivier Sallou <[email protected]
> <mailto:[email protected]>>:
>
>
> De: "Elouan Keryell-Even" <[email protected]
> <mailto:[email protected]>>
> À: [email protected] <mailto:[email protected]>
> Envoyé: Mercredi 9 Décembre 2015 10:48:10
> Objet: Re: GenOuest makes use of Mesos
>
> Hi Olivier,
>
> I've just read the presentation on your project's webpage, and it seems cool!
> I'm curious about the following mentioned feature: "Optional mount of user
> home or other shared directories in container". Does your framework take care
> of remotely copying the home directory onto the node where the container is
> going to run, or does the directory have to be already available (as a shared
> directory for example).
> We focus to "replace" classical computing cluster frameworks like Sun Grid
> Engine. So home directories etc... are shared among nodes (nfs, etc...), like
> in a usual cluster environment.
> The mount of "user home dir" is based on an Auth/ACL plugin/configuration.
> Admin specifies the available mounts for all users, or on a project basis.
> Then the Auth/ACL plugin maps those volumes to real available directories (if
> using the LDAP auth plugin, then get homeDirectory from LDAP and add as a
> volume to container).
> Otherwise, we are currently using framework Chronos for our Batch oriented
> containers, and your framework seems to fit the same spot. If you have some
> experience with Chronos, I'd be interested in a brief comparison of both
> frameworks. From what I understand, GODocker offers scheduling policy's
> customization, which I don't think Chronos does.
> There is indeed scheduling algorithms plugin mechanism. An interesting one is
> the fair-share policy imlpementation, where tasks are scheduler (ordered)
> based on previous user/project consumption.
>
> Difference with Chronos, is Chronos is made for cron like jobs. GODocker is a
> batch submission tool. User specifies the computing script he wants to
> execute and it is executed "immediatly". The scheduling is a matter of
> putting priorities when executing jobs (if users submits 1000 jobs but I have
> only 100 slots available). We focus on users submitting "many" jobs or many
> users submitting jobs, each job could be a computing task of a few seconds or
> several days.
>
>
>
> Olivier
>
>
> Thanks for your attention,
>
> Elouan Keryell-Even
> Software Engineer @ Atos Integration
>
> Toulouse, France
>
>
> 2015-12-08 14:49 GMT+01:00 Olivier Sallou <[email protected]
> <mailto:[email protected]>>:
> Hi,
> the GenOuest (http://www.genouest.org <http://www.genouest.org/>) academic
> lab is now using Mesos
> in production in its core facility to manage scientists computing tasks
> (for bioinformatics)
> To do so, we have developed a new mesos framework, GoDocker
> (http://www.genouest.org/godocker <http://www.genouest.org/godocker>) to
> submit batch computing scripts on
> premises. It mounts users home directory or other shared resources to
> execute jobs in Docker containers, using Mesos as main scheduler.
> GoDocker schedules the jobs according to user/groups priorities and
> quotas and provides a CLI, a REST web interface and a partial DRMAA
> library support. Framewok is open source.
>
> Thanks for adding us to the Mesos fellows ;-)
>
> Regards
>
> Olivier (GenOuest core developer member)
>
> --
> Olivier Sallou
> IRISA / University of Rennes 1
> Campus de Beaulieu, 35000 RENNES - FRANCE
> Tel: 02.99.84.71.95 <tel:02.99.84.71.95>
>
> gpg key id: 4096R/326D8438 (keyring.debian.org <http://keyring.debian.org/>)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
>
>
>
>
>
signature.asc
Description: Message signed with OpenPGP using GPGMail
