Apache Aurora [1] uses a custom Python executor and supports Docker via the containerizer. There's just one problem - the container has to have a Python2.7 runtime inside that can run the executor PEX file [2]. So if you're okay with that restriction you're in business (and you can use the Aurora configuration DSL to describe setup/teardown steps).
[1] https://aurora.apache.org [2] https://pex.readthedocs.org/en/latest/ On Tue, Aug 11, 2015 at 4:42 PM, Tim Chen <[email protected]> wrote: > So currently there is a review out for pre-hooks ( > https://reviews.apache.org/r/36185/) before a docker container launches. > > We can also add a post hook, but like to see if the specified hook > satifies what you guys are looking for. > > Tim > > On Tue, Aug 11, 2015 at 4:28 PM, Tom Fordon <[email protected]> wrote: > >> We ended up implementing a solution where we did the pre/post steps as >> separate mesos tasks and adding logic to our scheduler to ensure they were >> run on the same machine. If anybody knows of a standard / openly available >> DockerExecutor like what is described below, my team would be greatly >> interested. >> >> >> On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <[email protected]> wrote: >> >>> Hi, >>> >>> >>> >>> We have a similar usecase while running multi-user workloads on mesos. >>> Users provide docker images encapsulating application logic, which we (we = >>> say some “Central API”) schedule on Chronos / Marathon. However, we need to >>> run some standard pre / post steps for every docker submitted by users. We >>> have following options – >>> >>> >>> >>> 1. Ask every user to embed their logic inside a pre-defined >>> docker template which will perform pre/post steps. >>> >>> è This is error prone, makes us dependent on whether the users followed >>> template, and not very popular with users either. >>> >>> >>> >>> 2. Extend every user docker (FROM <>) and find a way to add >>> pre-post steps in our docker. Refer this docker when scheduling on chronos >>> / marathon. >>> >>> è Building new dockers does not scale as users and applications grow >>> >>> >>> >>> 3. Write a custom executor which will perform the pre-post steps >>> and manage the user docker lifetime. >>> >>> è Deals with user docker lifetime and is obviously complex. >>> >>> >>> >>> Is there a standard / openly available DockerExecutor which manages the >>> docker lifetime and which I can extend to build my custom executor? This >>> way I will be concerned only with my custom logic (pre/post steps) and >>> still get benefits of a standard way to manage docker containers. >>> >>> >>> >>> Btw, thanks for the meaningful discussion below, it is very helpful. >>> >>> >>> >>> Thanks and regards, >>> >>> >>> >>> Kapil Malik | [email protected] | 33430 / 8800836581 >>> >>> >>> >>> *From:* James DeFelice [mailto:[email protected]] >>> *Sent:* 09 April 2015 18:12 >>> *To:* [email protected] >>> *Subject:* Re: Custom python executor with Docker >>> >>> >>> >>> If you can run the pre/post steps in a container then I'd recommend >>> building a Docker image that includes your pre/post step scripting + your >>> algorithm and launching it using the built-in mesos Docker containerizer. >>> It's much simpler than managing the lifetime of the Docker container >>> yourself. >>> >>> >>> >>> On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <[email protected]> wrote: >>> >>> Thanks for all the responses, I really appreciate the help. Let me try >>> to state my problem more clearly >>> >>> >>> >>> Our project is performing file-based data processing. I would like to >>> keep the actual algorithm as contained as possible since we are in an R&D >>> setting and will be getting untested code. We have some pre/post steps >>> that need to be run on the same box as the actual algorithm: >>> downloading/uploading files and database calls. >>> >>> >>> >>> We can run the pre/post steps and algorithm within the same container. >>> The algorithm will be a little less contained, but it will work. >>> >>> >>> >>> Docker letting you specify a cgroup parent is really exciting. If I >>> invoke a docker container with the executor as the cgroup-parent are there >>> any other steps I need to perform? Would I need to do anything special to >>> make mesos aware of the resource usage, or is that handled since the docker >>> process would be in the executors cgroup? >>> >>> >>> >>> Thanks again, >>> >>> Tom >>> >>> >>> >>> On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <[email protected]> wrote: >>> >>> Hi Tom(s), >>> >>> Tom Arnfeld is right, if you want to launch your own docker container >>> in your custom executor you will have to handle all the issues >>> yourself and not able to use the Docker containerizer at all. >>> >>> Alternatively, you can actually launch your custom executor in a >>> Docker container by Mesos, by specifying the ContainerInfo in the >>> ExecutorInfo. >>> What this means is that your custom executor is already running in a >>> docker container, and you can do your custom logic afterwards. This >>> does means you can simply just launch multiple containers in the >>> executor anymore. >>> >>> If there is something you want to do and doesnt' fit these let us know >>> what you're trying to achieve and we can see what we can do. >>> >>> Tim >>> >>> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <[email protected]> wrote: >>> >>> > It's not possible to invoke the docker containerizer from outside of >>> Mesos, >>> > as far as I know. >>> > >>> > If you persue this route, you can run into issues with orphaned >>> containers >>> > as your executor may die for some unknown reason, and the container is >>> still >>> > running. Recovering from this can be tricky business, so it's better >>> if you >>> > can adapt your framework design to fit within the Mesos Task/Executor >>> > pattern. >>> > >>> > -- >>> > >>> > Tom Arnfeld >>> > Developer // DueDil >>> > >>> > (+44) 7525940046 >>> > 25 Christopher Street, London, EC2A 2BS >>> > >>> > >>> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <[email protected]> >>> wrote: >>> >> >>> >> Tim, do you want answer this? >>> >> >>> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <[email protected]> >>> wrote: >>> >>> >>> >>> Hi. I'm trying to understand using docker within a custom executor. >>> For >>> >>> each of my tasks, I would like to perform some steps on the node >>> before >>> >>> launching a docker container. I was planning on writing a custom >>> python >>> >>> executor for this, but I wasn't sure how to launch docker from >>> within this >>> >>> executor. >>> >>> >>> >>> Can I just call docker in a subprocess using the ContainerInfo from >>> the >>> >>> Task? If I do this, how does the Containerizer fit in? >>> >>> >>> >>> Thank you, >>> >>> Tom Fordon >>> >> >>> >> >>> > >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> James DeFelice >>> 585.241.9488 (voice) >>> 650.649.6071 (fax) >>> >> >> >
