Hey Ben, We are not using the Docker containerizer from Mesos Core yet, we just have a plain executor. And by "container" I meant the docker container here. The docker containers in this case are started by the executor [which uses Docker's remote API]. Mesos does kill the container that it creates to run the executor but however the docker containers are not controlled by Mesos Slave.
Also because Mesos doesn't know the existence of those containers because the executor was monitoring them, task reconciliation won't help either. This is another downside of this approach. So we use an out-of-band communication here to kill containers that the Framework and Mesos Master thinks should not be running. Our main problem is - we want the docker containers to be setup in a certain way, for example attach elastic network interfaces to containers, mount ZFS volumes etc, and that's why we had to write a custom executor in the first place. On Mon, Dec 1, 2014 at 1:01 PM, Benjamin Mahler <[email protected]> wrote: > Sorry this is a bit of a tangent to the thread: > > > For ex, if the executor crashes Mesos would get a TASK_LOST while the > container might still be running. > > Mesos should destroy the container when the executor exits, are you seeing > otherwise? > > > So we are doing something similar to Aurora's GC Executor to clean up > containers behind us. > > Just to make sure you're aware, you don't need to do this with 0.21.0+, > now that we provide reconciliation support. :) > > On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <[email protected]> > wrote: > >> There are some downsides to this as well - For ex, if the executor >> crashes Mesos would get a TASK_LOST while the container might still be >> running. So we are doing something similar to Aurora's GC Executor to clean >> up containers behind us. >> >> On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa < >> [email protected]> wrote: >> >>> Hi Diptanu, >>> Thanks for your feedback. This sounds like the approach we'll be taking, >>> too. >>> >>> >>> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <[email protected]> >>> wrote: >>> >>>> Hi Janet, >>>> >>>> We implemented the same in our Titan Mesos Executor. We have an >>>> executor manage multiple containers. We didn't want the overhead of running >>>> multiple executors to manage multiple containers. We are using RxJava >>>> heavily in our codebase, so whenever a launchTask callback is invoked in >>>> the MesosExecutor we publish an event and return immediately from the >>>> callback so that our executor can handle more callbacks from Mesos Slave. >>>> >>>> But the event which is published is picked up by the subscriber and >>>> that starts the workflow of downloading a container, mount volumes, sets up >>>> networking, sets up logging, and finally starts a monitoring process for >>>> the container. The monitoring process has the reference to the executor >>>> driver and we send back Mesos status updates whenever the state of the >>>> process changes. >>>> >>>> >>>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa < >>>> [email protected]> wrote: >>>> >>>>> Hi David, >>>>> Thanks for your suggestion. This approach sounds promising for what I >>>>> need to do. I'm going to have to try this out. >>>>> >>>>> Thanks! >>>>> Janet >>>>> >>>>> -- >>>>> >>>>> Janet Borschowa >>>>> CodeFutures Corporation >>>>> >>>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg < >>>>> [email protected]> wrote: >>>>> >>>>>> One cool feature of the docker containerizer is that you can actually >>>>>> launch your executor inside the docker image, so that you can just layer >>>>>> the executor's custom logic on top of whatever container you desire. This >>>>>> way, you can more easily control what's happening in the docker image, >>>>>> and >>>>>> still leverage the containerizer. >>>>>> >>>>>> On Thursday, November 20, 2014, Tim Chen <[email protected]> wrote: >>>>>> >>>>>>> Hi Janet, >>>>>>> >>>>>>> Can you elaborate more what you like to get back from the docker >>>>>>> container that you launched? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Janet, >>>>>>>> >>>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought >>>>>>>> you were using the containerizer. What you're doing here is actually >>>>>>>> going >>>>>>>> to be quite difficult to do, the mesos docker containerizer has some >>>>>>>> quite >>>>>>>> complex logic implemented to ensure the slave stays in sync with the >>>>>>>> containers that are running, and kills anything that goes rogue. >>>>>>>> >>>>>>>> It's going to be non-trivial for you to do that from the executor, >>>>>>>> though I guess you could make use of the docker events API or poll >>>>>>>> other >>>>>>>> endpoints in the API to check the status of your containers, and off >>>>>>>> the >>>>>>>> back of that send status updates to the cluster. Doing this however >>>>>>>> brings >>>>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) >>>>>>>> the >>>>>>>> containers spawned will die... they'll keep running in the background >>>>>>>> and >>>>>>>> it'll be hard for you to know the state of your containers on the >>>>>>>> cluster. >>>>>>>> >>>>>>>> You probably want to be aware (if you don't know already) that the >>>>>>>> resource limits assigned to your tasks aren't going to be enforced by >>>>>>>> mesos >>>>>>>> because docker is running outside of its control. You'll need to pass >>>>>>>> the >>>>>>>> correct CPU/Memory limit parameters to your docker containers to ensure >>>>>>>> this happens correctly. >>>>>>>> >>>>>>>> Here are the docker API docs; >>>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/ >>>>>>>> >>>>>>>> Something you might want to consider, if all you're trying to do is >>>>>>>> allow your container access to details about itself (e.g `docker >>>>>>>> inspect`) >>>>>>>> is to open up the docker remote API to be queried by your containers >>>>>>>> on the >>>>>>>> slave, and switch to using the mesos docker containerizer. >>>>>>>> >>>>>>>> I hope that helps somewhat! >>>>>>>> >>>>>>>> Tom. >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Tom Arnfeld >>>>>>>> Developer // DueDil >>>>>>>> >>>>>>>> (+44) 7525940046 >>>>>>>> 25 Christopher Street, London, EC2A 2BS >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> I'm implementing an executor which is used by the mesos slave to >>>>>>>>> launch tasks. The tasks are to launch a docker container - this is >>>>>>>>> because >>>>>>>>> I need more info about the launched container than what the docker >>>>>>>>> containerizer returns. >>>>>>>>> >>>>>>>>> Is it OK to block in the executor's launchTask method until the >>>>>>>>> task completes? If not, how does the framework discover when that task >>>>>>>>> completes? I could spawn a process which notifies my executor when >>>>>>>>> the task >>>>>>>>> completes and then have my executor send a status update. Or is there >>>>>>>>> some >>>>>>>>> other recommended way to deal with this when the task could run for an >>>>>>>>> indefinite period of time before completing its work? >>>>>>>>> >>>>>>>>> Thanks! >>>>>>>>> >>>>>>>>> Janet >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Janet Borschowa >>>>>>>>> CodeFutures Corporation >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Diptanu Choudhury >>>> Web - www.linkedin.com/in/diptanu >>>> Twitter - @diptanu <http://twitter.com/diptanu> >>>> >>> >>> >> >> >> -- >> Thanks, >> Diptanu Choudhury >> Web - www.linkedin.com/in/diptanu >> Twitter - @diptanu <http://twitter.com/diptanu> >> > > -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu <http://twitter.com/diptanu>
