Forgot to mention, unless you have a custom executor that you launch as a docker container (by putting DockerInfo in the ExecutorInfo in your TaskInfo), you can then re-use that executor for multiple tasks.
Tim On Wed, Dec 3, 2014 at 11:47 AM, Tim Chen <[email protected]> wrote: > Hi Sharma, > > Yes currently docker doesn't really support (out-of-box) launching > multiple processes in the same container. They just recently added docker > exec but not quite clear how it's best fit in mesos integration yet. > > So each task run in the Docker containerizer has to be a seperate > container for now. > > Tim > > On Wed, Dec 3, 2014 at 11:09 AM, Sharma Podila <[email protected]> > wrote: > >> Yes, although, there's a nuance to this specific situation. Here, the >> same executor is being used for multiple tasks, but, the executor is >> launching a different Docker container for each task. I was extending the >> coarse grain allocation concept to within the executor (which is in a fine >> grained allocation model). >> What you mention, we do use already for a different framework, not the >> one Diptanu is talking about. >> >> On Wed, Dec 3, 2014 at 11:04 AM, Connor Doyle <[email protected]> >> wrote: >> >>> You're right Sharma, it's dependent upon the framework. If your >>> scheduler sets a unique ExecutorID for each TaskInfo, then the executor >>> will not be re-used and you won't have to worry about resizing the >>> executor's container to accomodate subsequent tasks. This might be a >>> reasonable simplification to start with, especially if your executor adds >>> relatively low resource overhead. >>> -- >>> Connor >>> >>> >>> > On Dec 3, 2014, at 10:20, Sharma Podila <[email protected]> wrote: >>> > >>> > This may have to do with fine-grain Vs coarse-grain resource >>> allocation. Things may be easier for you, Diptanu, if you are using one >>> Docker container per task (sort of coarse grain). In that case, I believe >>> there's no need to alter a running Docker container's resources. Instead, >>> the resource update of your executor translates into the right Docker >>> containers running. There's some details to be worked out there, I am sure. >>> > It sounds like Tom's strategy uses the same Docker container for >>> multiple tasks. Tom, do correct me otherwise. >>> > >>> > On Wed, Dec 3, 2014 at 3:38 AM, Tom Arnfeld <[email protected]> wrote: >>> > When Mesos is asked to a launch a task (with either a custom Executor >>> or the built in CommandExecutor) it will first spawn the executor which >>> _has_ to be a system process, launched via command. This process will be >>> launched inside of a Docker container when using the previously mentioned >>> containerizers. >>> > >>> > Once the Executor registers with the slave, the slave will send it a >>> number of launchTask calls based on the number of tasks queued up for that >>> executor. The Executor can then do as it pleases with those tasks, whether >>> it's just a sleep(1) or to spawn a subprocess and do some other work. Given >>> it is possible for the framework to specify resources for both tasks and >>> executors, and the only thing which _has_ to be a system process is the >>> executor, the mesos slave will limit the resources of the executor process >>> to the sum of (TaskInfo.Executor.Resources + TaskInfo.Resources). >>> > >>> > Mesos also has the ability to launch new tasks on an already running >>> executor, so it's important that mesos is able to dynamically scale the >>> resource limits up and down over time. Designing a framework around this >>> idea can lead to some complex and powerful workflows which would be a lot >>> more complex to build without Mesos. >>> > >>> > Just for an example... Spark. >>> > >>> > 1) User launches a job on spark to map over some data >>> > 2) Spark launches a first wave of tasks based on the offers it >>> received (let's say T1 and T2) >>> > 3) Mesos launches executors for those tasks (let's say E1 and E2) on >>> different slaves >>> > 4) Spark launches another wave of tasks based on offers, and tells >>> mesos to use the same executor (E1 and E2) >>> > 5) Mesos will simply call launchTasks(T{3,4}) on the two already >>> running executors >>> > >>> > At point (3) mesos is going to launch a Docker container and execute >>> your executor. However at (5) the executor is already running so the tasks >>> will be handed to the already running executor. >>> > >>> > Mesos will guarantee you (i'm 99% sure) that the resources for your >>> container have been updated to reflect the limits set on the tasks before >>> handing the tasks to you. >>> > >>> > I hope that makes some sense! >>> > >>> > -- >>> > >>> > Tom Arnfeld >>> > Developer // DueDil >>> > >>> > >>> > On Wed, Dec 3, 2014 at 10:54 AM, Diptanu Choudhury <[email protected]> >>> wrote: >>> > >>> > Thanks for the explanation Tom, yeah I just figured that out by >>> reading your code! You're touching the memory.soft_limit_in_bytes and >>> memory.limit_in_bytes directly. >>> > >>> > Still curios to understand in which situations Mesos Slave would call >>> the external containerizer to update the resource limits of a container? My >>> understanding was that once resource allocation happens for a task, >>> resources are not taken away until the task exits[fails, crashes or >>> finishes] or Mesos asks the slave to kill the task. >>> > >>> > On Wed, Dec 3, 2014 at 2:47 AM, Tom Arnfeld <[email protected]> wrote: >>> > Hi Diptanu, >>> > >>> > That's correct, the ECP has the responsibility of updating the >>> resource for a container, and it will do as new tasks are launched and >>> killed for an executor. Since docker doesn't support this, our >>> containerizer (Deimos does the same) goes behind docker to the cgroup for >>> the container and updates the resources in a very similar way to the >>> mesos-slave. I believe this is also what the built in Docker containerizer >>> will do. >>> > >>> > >>> https://github.com/duedil-ltd/mesos-docker-containerizer/blob/master/containerizer/commands/update.py#L35 >>> > >>> > Tom. >>> > >>> > -- >>> > >>> > Tom Arnfeld >>> > Developer // DueDil >>> > >>> > >>> > On Wed, Dec 3, 2014 at 10:45 AM, Diptanu Choudhury <[email protected]> >>> wrote: >>> > >>> > Hi, >>> > >>> > I had a quick question about the external containerizer. I see that >>> once the Task is launched, the ECP can receive the update calls, and the >>> protobuf message passed to ECP with the update call is >>> containerizer::Update. >>> > >>> > This protobuf has a Resources [list] field so does that mean Mesos >>> might ask a running task to re-adjust the enforced resource limits? >>> > >>> > How would this work if the ECP was launching docker containers because >>> Docker doesn't allow changing the resource limits once the container has >>> been started? >>> > >>> > I am wondering how does Deimos and mesos-docker-containerizer handle >>> this. >>> > >>> > -- >>> > Thanks, >>> > Diptanu Choudhury >>> > Web - www.linkedin.com/in/diptanu >>> > Twitter - @diptanu >>> > >>> > >>> > >>> > >>> > -- >>> > Thanks, >>> > Diptanu Choudhury >>> > Web - www.linkedin.com/in/diptanu >>> > Twitter - @diptanu >>> > >>> > >>> >>> >> >
