Got it, thanks for clarifying. :) On Mon, Dec 1, 2014 at 1:40 PM, Diptanu Choudhury <[email protected]> wrote:
> Hey Ben, > > We are not using the Docker containerizer from Mesos Core yet, we just > have a plain executor. And by "container" I meant the docker container > here. The docker containers in this case are started by the executor [which > uses Docker's remote API]. Mesos does kill the container that it creates to > run the executor but however the docker containers are not controlled by > Mesos Slave. > > Also because Mesos doesn't know the existence of those containers because > the executor was monitoring them, task reconciliation won't help either. > This is another downside of this approach. So we use an out-of-band > communication here to kill containers that the Framework and Mesos Master > thinks should not be running. > > Our main problem is - we want the docker containers to be setup in a > certain way, for example attach elastic network interfaces to containers, > mount ZFS volumes etc, and that's why we had to write a custom executor in > the first place. > > > On Mon, Dec 1, 2014 at 1:01 PM, Benjamin Mahler <[email protected] > > wrote: > >> Sorry this is a bit of a tangent to the thread: >> >> > For ex, if the executor crashes Mesos would get a TASK_LOST while the >> container might still be running. >> >> Mesos should destroy the container when the executor exits, are you >> seeing otherwise? >> >> > So we are doing something similar to Aurora's GC Executor to clean up >> containers behind us. >> >> Just to make sure you're aware, you don't need to do this with 0.21.0+, >> now that we provide reconciliation support. :) >> >> On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury <[email protected]> >> wrote: >> >>> There are some downsides to this as well - For ex, if the executor >>> crashes Mesos would get a TASK_LOST while the container might still be >>> running. So we are doing something similar to Aurora's GC Executor to clean >>> up containers behind us. >>> >>> On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa < >>> [email protected]> wrote: >>> >>>> Hi Diptanu, >>>> Thanks for your feedback. This sounds like the approach we'll be >>>> taking, too. >>>> >>>> >>>> On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury <[email protected] >>>> > wrote: >>>> >>>>> Hi Janet, >>>>> >>>>> We implemented the same in our Titan Mesos Executor. We have an >>>>> executor manage multiple containers. We didn't want the overhead of >>>>> running >>>>> multiple executors to manage multiple containers. We are using RxJava >>>>> heavily in our codebase, so whenever a launchTask callback is invoked in >>>>> the MesosExecutor we publish an event and return immediately from the >>>>> callback so that our executor can handle more callbacks from Mesos Slave. >>>>> >>>>> But the event which is published is picked up by the subscriber and >>>>> that starts the workflow of downloading a container, mount volumes, sets >>>>> up >>>>> networking, sets up logging, and finally starts a monitoring process for >>>>> the container. The monitoring process has the reference to the executor >>>>> driver and we send back Mesos status updates whenever the state of the >>>>> process changes. >>>>> >>>>> >>>>> On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi David, >>>>>> Thanks for your suggestion. This approach sounds promising for what I >>>>>> need to do. I'm going to have to try this out. >>>>>> >>>>>> Thanks! >>>>>> Janet >>>>>> >>>>>> -- >>>>>> >>>>>> Janet Borschowa >>>>>> CodeFutures Corporation >>>>>> >>>>>> On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> One cool feature of the docker containerizer is that you can >>>>>>> actually launch your executor inside the docker image, so that you can >>>>>>> just >>>>>>> layer the executor's custom logic on top of whatever container you >>>>>>> desire. >>>>>>> This way, you can more easily control what's happening in the docker >>>>>>> image, >>>>>>> and still leverage the containerizer. >>>>>>> >>>>>>> On Thursday, November 20, 2014, Tim Chen <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Janet, >>>>>>>> >>>>>>>> Can you elaborate more what you like to get back from the docker >>>>>>>> container that you launched? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Janet, >>>>>>>>> >>>>>>>>> Oh sorry my mistake, I didn't read your email correctly, I thought >>>>>>>>> you were using the containerizer. What you're doing here is actually >>>>>>>>> going >>>>>>>>> to be quite difficult to do, the mesos docker containerizer has some >>>>>>>>> quite >>>>>>>>> complex logic implemented to ensure the slave stays in sync with the >>>>>>>>> containers that are running, and kills anything that goes rogue. >>>>>>>>> >>>>>>>>> It's going to be non-trivial for you to do that from the executor, >>>>>>>>> though I guess you could make use of the docker events API or poll >>>>>>>>> other >>>>>>>>> endpoints in the API to check the status of your containers, and off >>>>>>>>> the >>>>>>>>> back of that send status updates to the cluster. Doing this however >>>>>>>>> brings >>>>>>>>> no guarantees that if your executor dies exceptionally (perhaps OOMd) >>>>>>>>> the >>>>>>>>> containers spawned will die... they'll keep running in the background >>>>>>>>> and >>>>>>>>> it'll be hard for you to know the state of your containers on the >>>>>>>>> cluster. >>>>>>>>> >>>>>>>>> You probably want to be aware (if you don't know already) that the >>>>>>>>> resource limits assigned to your tasks aren't going to be enforced by >>>>>>>>> mesos >>>>>>>>> because docker is running outside of its control. You'll need to pass >>>>>>>>> the >>>>>>>>> correct CPU/Memory limit parameters to your docker containers to >>>>>>>>> ensure >>>>>>>>> this happens correctly. >>>>>>>>> >>>>>>>>> Here are the docker API docs; >>>>>>>>> https://docs.docker.com/reference/api/docker_remote_api_v1.15/ >>>>>>>>> >>>>>>>>> Something you might want to consider, if all you're trying to do >>>>>>>>> is allow your container access to details about itself (e.g `docker >>>>>>>>> inspect`) is to open up the docker remote API to be queried by your >>>>>>>>> containers on the slave, and switch to using the mesos docker >>>>>>>>> containerizer. >>>>>>>>> >>>>>>>>> I hope that helps somewhat! >>>>>>>>> >>>>>>>>> Tom. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Tom Arnfeld >>>>>>>>> Developer // DueDil >>>>>>>>> >>>>>>>>> (+44) 7525940046 >>>>>>>>> 25 Christopher Street, London, EC2A 2BS >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> I'm implementing an executor which is used by the mesos slave to >>>>>>>>>> launch tasks. The tasks are to launch a docker container - this is >>>>>>>>>> because >>>>>>>>>> I need more info about the launched container than what the docker >>>>>>>>>> containerizer returns. >>>>>>>>>> >>>>>>>>>> Is it OK to block in the executor's launchTask method until the >>>>>>>>>> task completes? If not, how does the framework discover when that >>>>>>>>>> task >>>>>>>>>> completes? I could spawn a process which notifies my executor when >>>>>>>>>> the task >>>>>>>>>> completes and then have my executor send a status update. Or is >>>>>>>>>> there some >>>>>>>>>> other recommended way to deal with this when the task could run for >>>>>>>>>> an >>>>>>>>>> indefinite period of time before completing its work? >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> Janet >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Janet Borschowa >>>>>>>>>> CodeFutures Corporation >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks, >>>>> Diptanu Choudhury >>>>> Web - www.linkedin.com/in/diptanu >>>>> Twitter - @diptanu <http://twitter.com/diptanu> >>>>> >>>> >>>> >>> >>> >>> -- >>> Thanks, >>> Diptanu Choudhury >>> Web - www.linkedin.com/in/diptanu >>> Twitter - @diptanu <http://twitter.com/diptanu> >>> >> >> > > > -- > Thanks, > Diptanu Choudhury > Web - www.linkedin.com/in/diptanu > Twitter - @diptanu <http://twitter.com/diptanu> >
