If you can run the pre/post steps in a container then I'd recommend building a Docker image that includes your pre/post step scripting + your algorithm and launching it using the built-in mesos Docker containerizer. It's much simpler than managing the lifetime of the Docker container yourself.
On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <[email protected]> wrote: > Thanks for all the responses, I really appreciate the help. Let me try to > state my problem more clearly > > Our project is performing file-based data processing. I would like to > keep the actual algorithm as contained as possible since we are in an R&D > setting and will be getting untested code. We have some pre/post steps > that need to be run on the same box as the actual algorithm: > downloading/uploading files and database calls. > > We can run the pre/post steps and algorithm within the same container. > The algorithm will be a little less contained, but it will work. > > Docker letting you specify a cgroup parent is really exciting. If I > invoke a docker container with the executor as the cgroup-parent are there > any other steps I need to perform? Would I need to do anything special to > make mesos aware of the resource usage, or is that handled since the docker > process would be in the executors cgroup? > > Thanks again, > Tom > > On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <[email protected]> wrote: > >> Hi Tom(s), >> >> Tom Arnfeld is right, if you want to launch your own docker container >> in your custom executor you will have to handle all the issues >> yourself and not able to use the Docker containerizer at all. >> >> Alternatively, you can actually launch your custom executor in a >> Docker container by Mesos, by specifying the ContainerInfo in the >> ExecutorInfo. >> What this means is that your custom executor is already running in a >> docker container, and you can do your custom logic afterwards. This >> does means you can simply just launch multiple containers in the >> executor anymore. >> >> If there is something you want to do and doesnt' fit these let us know >> what you're trying to achieve and we can see what we can do. >> >> Tim >> >> On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <[email protected]> wrote: >> > It's not possible to invoke the docker containerizer from outside of >> Mesos, >> > as far as I know. >> > >> > If you persue this route, you can run into issues with orphaned >> containers >> > as your executor may die for some unknown reason, and the container is >> still >> > running. Recovering from this can be tricky business, so it's better if >> you >> > can adapt your framework design to fit within the Mesos Task/Executor >> > pattern. >> > >> > -- >> > >> > Tom Arnfeld >> > Developer // DueDil >> > >> > (+44) 7525940046 >> > 25 Christopher Street, London, EC2A 2BS >> > >> > >> > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <[email protected]> >> wrote: >> >> >> >> Tim, do you want answer this? >> >> >> >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <[email protected]> >> wrote: >> >>> >> >>> Hi. I'm trying to understand using docker within a custom executor. >> For >> >>> each of my tasks, I would like to perform some steps on the node >> before >> >>> launching a docker container. I was planning on writing a custom >> python >> >>> executor for this, but I wasn't sure how to launch docker from within >> this >> >>> executor. >> >>> >> >>> Can I just call docker in a subprocess using the ContainerInfo from >> the >> >>> Task? If I do this, how does the Containerizer fit in? >> >>> >> >>> Thank you, >> >>> Tom Fordon >> >> >> >> >> > >> > > -- James DeFelice 585.241.9488 (voice) 650.649.6071 (fax)
