Sorry this is a bit of a tangent to the thread: > For ex, if the executor crashes Mesos would get a TASK_LOST while the container might still be running.
Mesos should destroy the container when the executor exits, are you seeing otherwise? > So we are doing something similar to Aurora's GC Executor to clean up containers behind us. Just to make sure you're aware, you don't need to do this with 0.21.0+, now that we provide reconciliation support. :) On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <[email protected]> wrote: > There are some downsides to this as well - For ex, if the executor crashes > Mesos would get a TASK_LOST while the container might still be running. So > we are doing something similar to Aurora's GC Executor to clean up > containers behind us. > > On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa < > [email protected]> wrote: > >> Hi Diptanu, >> Thanks for your feedback. This sounds like the approach we'll be taking, >> too. >> >> >> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <[email protected]> >> wrote: >> >>> Hi Janet, >>> >>> We implemented the same in our Titan Mesos Executor. We have an executor >>> manage multiple containers. We didn't want the overhead of running multiple >>> executors to manage multiple containers. We are using RxJava heavily in our >>> codebase, so whenever a launchTask callback is invoked in the MesosExecutor >>> we publish an event and return immediately from the callback so that our >>> executor can handle more callbacks from Mesos Slave. >>> >>> But the event which is published is picked up by the subscriber and that >>> starts the workflow of downloading a container, mount volumes, sets up >>> networking, sets up logging, and finally starts a monitoring process for >>> the container. The monitoring process has the reference to the executor >>> driver and we send back Mesos status updates whenever the state of the >>> process changes. >>> >>> >>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa < >>> [email protected]> wrote: >>> >>>> Hi David, >>>> Thanks for your suggestion. This approach sounds promising for what I >>>> need to do. I'm going to have to try this out. >>>> >>>> Thanks! >>>> Janet >>>> >>>> -- >>>> >>>> Janet Borschowa >>>> CodeFutures Corporation >>>> >>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg < >>>> [email protected]> wrote: >>>> >>>>> One cool feature of the docker containerizer is that you can actually >>>>> launch your executor inside the docker image, so that you can just layer >>>>> the executor's custom logic on top of whatever container you desire. This >>>>> way, you can more easily control what's happening in the docker image, and >>>>> still leverage the containerizer. >>>>> >>>>> On Thursday, November 20, 2014, Tim Chen <[email protected]> wrote: >>>>> >>>>>> Hi Janet, >>>>>> >>>>>> Can you elaborate more what you like to get back from the docker >>>>>> container that you launched? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Tim >>>>>> >>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <[email protected]> wrote: >>>>>> >>>>>>> Hi Janet, >>>>>>> >>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought >>>>>>> you were using the containerizer. What you're doing here is actually >>>>>>> going >>>>>>> to be quite difficult to do, the mesos docker containerizer has some >>>>>>> quite >>>>>>> complex logic implemented to ensure the slave stays in sync with the >>>>>>> containers that are running, and kills anything that goes rogue. >>>>>>> >>>>>>> It's going to be non-trivial for you to do that from the executor, >>>>>>> though I guess you could make use of the docker events API or poll other >>>>>>> endpoints in the API to check the status of your containers, and off the >>>>>>> back of that send status updates to the cluster. Doing this however >>>>>>> brings >>>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) >>>>>>> the >>>>>>> containers spawned will die... they'll keep running in the background >>>>>>> and >>>>>>> it'll be hard for you to know the state of your containers on the >>>>>>> cluster. >>>>>>> >>>>>>> You probably want to be aware (if you don't know already) that the >>>>>>> resource limits assigned to your tasks aren't going to be enforced by >>>>>>> mesos >>>>>>> because docker is running outside of its control. You'll need to pass >>>>>>> the >>>>>>> correct CPU/Memory limit parameters to your docker containers to ensure >>>>>>> this happens correctly. >>>>>>> >>>>>>> Here are the docker API docs; >>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/ >>>>>>> >>>>>>> Something you might want to consider, if all you're trying to do is >>>>>>> allow your container access to details about itself (e.g `docker >>>>>>> inspect`) >>>>>>> is to open up the docker remote API to be queried by your containers on >>>>>>> the >>>>>>> slave, and switch to using the mesos docker containerizer. >>>>>>> >>>>>>> I hope that helps somewhat! >>>>>>> >>>>>>> Tom. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Tom Arnfeld >>>>>>> Developer // DueDil >>>>>>> >>>>>>> (+44) 7525940046 >>>>>>> 25 Christopher Street, London, EC2A 2BS >>>>>>> >>>>>>> >>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> I'm implementing an executor which is used by the mesos slave to >>>>>>>> launch tasks. The tasks are to launch a docker container - this is >>>>>>>> because >>>>>>>> I need more info about the launched container than what the docker >>>>>>>> containerizer returns. >>>>>>>> >>>>>>>> Is it OK to block in the executor's launchTask method until the >>>>>>>> task completes? If not, how does the framework discover when that task >>>>>>>> completes? I could spawn a process which notifies my executor when the >>>>>>>> task >>>>>>>> completes and then have my executor send a status update. Or is there >>>>>>>> some >>>>>>>> other recommended way to deal with this when the task could run for an >>>>>>>> indefinite period of time before completing its work? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Janet >>>>>>>> >>>>>>>> -- >>>>>>>> Janet Borschowa >>>>>>>> CodeFutures Corporation >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Diptanu Choudhury >>> Web - www.linkedin.com/in/diptanu >>> Twitter - @diptanu <http://twitter.com/diptanu> >>> >> >> > > > -- > Thanks, > Diptanu Choudhury > Web - www.linkedin.com/in/diptanu > Twitter - @diptanu <http://twitter.com/diptanu> >
