Thanks for all the responses, I really appreciate the help. Let me try to state my problem more clearly
Our project is performing file-based data processing. I would like to keep the actual algorithm as contained as possible since we are in an R&D setting and will be getting untested code. We have some pre/post steps that need to be run on the same box as the actual algorithm: downloading/uploading files and database calls. We can run the pre/post steps and algorithm within the same container. The algorithm will be a little less contained, but it will work. Docker letting you specify a cgroup parent is really exciting. If I invoke a docker container with the executor as the cgroup-parent are there any other steps I need to perform? Would I need to do anything special to make mesos aware of the resource usage, or is that handled since the docker process would be in the executors cgroup? Thanks again, Tom On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <[email protected]> wrote: > Hi Tom(s), > > Tom Arnfeld is right, if you want to launch your own docker container > in your custom executor you will have to handle all the issues > yourself and not able to use the Docker containerizer at all. > > Alternatively, you can actually launch your custom executor in a > Docker container by Mesos, by specifying the ContainerInfo in the > ExecutorInfo. > What this means is that your custom executor is already running in a > docker container, and you can do your custom logic afterwards. This > does means you can simply just launch multiple containers in the > executor anymore. > > If there is something you want to do and doesnt' fit these let us know > what you're trying to achieve and we can see what we can do. > > Tim > > On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <[email protected]> wrote: > > It's not possible to invoke the docker containerizer from outside of > Mesos, > > as far as I know. > > > > If you persue this route, you can run into issues with orphaned > containers > > as your executor may die for some unknown reason, and the container is > still > > running. Recovering from this can be tricky business, so it's better if > you > > can adapt your framework design to fit within the Mesos Task/Executor > > pattern. > > > > -- > > > > Tom Arnfeld > > Developer // DueDil > > > > (+44) 7525940046 > > 25 Christopher Street, London, EC2A 2BS > > > > > > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <[email protected]> wrote: > >> > >> Tim, do you want answer this? > >> > >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <[email protected]> > wrote: > >>> > >>> Hi. I'm trying to understand using docker within a custom executor. > For > >>> each of my tasks, I would like to perform some steps on the node before > >>> launching a docker container. I was planning on writing a custom python > >>> executor for this, but I wasn't sure how to launch docker from within > this > >>> executor. > >>> > >>> Can I just call docker in a subprocess using the ContainerInfo from the > >>> Task? If I do this, how does the Containerizer fit in? > >>> > >>> Thank you, > >>> Tom Fordon > >> > >> > > >
