You're right Sharma, it's dependent upon the framework. If your scheduler sets a unique ExecutorID for each TaskInfo, then the executor will not be re-used and you won't have to worry about resizing the executor's container to accomodate subsequent tasks. This might be a reasonable simplification to start with, especially if your executor adds relatively low resource overhead. -- Connor
> On Dec 3, 2014, at 10:20, Sharma Podila <[email protected]> wrote: > > This may have to do with fine-grain Vs coarse-grain resource allocation. > Things may be easier for you, Diptanu, if you are using one Docker container > per task (sort of coarse grain). In that case, I believe there's no need to > alter a running Docker container's resources. Instead, the resource update of > your executor translates into the right Docker containers running. There's > some details to be worked out there, I am sure. > It sounds like Tom's strategy uses the same Docker container for multiple > tasks. Tom, do correct me otherwise. > > On Wed, Dec 3, 2014 at 3:38 AM, Tom Arnfeld <[email protected]> wrote: > When Mesos is asked to a launch a task (with either a custom Executor or the > built in CommandExecutor) it will first spawn the executor which _has_ to be > a system process, launched via command. This process will be launched inside > of a Docker container when using the previously mentioned containerizers. > > Once the Executor registers with the slave, the slave will send it a number > of launchTask calls based on the number of tasks queued up for that executor. > The Executor can then do as it pleases with those tasks, whether it's just a > sleep(1) or to spawn a subprocess and do some other work. Given it is > possible for the framework to specify resources for both tasks and executors, > and the only thing which _has_ to be a system process is the executor, the > mesos slave will limit the resources of the executor process to the sum of > (TaskInfo.Executor.Resources + TaskInfo.Resources). > > Mesos also has the ability to launch new tasks on an already running > executor, so it's important that mesos is able to dynamically scale the > resource limits up and down over time. Designing a framework around this idea > can lead to some complex and powerful workflows which would be a lot more > complex to build without Mesos. > > Just for an example... Spark. > > 1) User launches a job on spark to map over some data > 2) Spark launches a first wave of tasks based on the offers it received > (let's say T1 and T2) > 3) Mesos launches executors for those tasks (let's say E1 and E2) on > different slaves > 4) Spark launches another wave of tasks based on offers, and tells mesos to > use the same executor (E1 and E2) > 5) Mesos will simply call launchTasks(T{3,4}) on the two already running > executors > > At point (3) mesos is going to launch a Docker container and execute your > executor. However at (5) the executor is already running so the tasks will be > handed to the already running executor. > > Mesos will guarantee you (i'm 99% sure) that the resources for your container > have been updated to reflect the limits set on the tasks before handing the > tasks to you. > > I hope that makes some sense! > > -- > > Tom Arnfeld > Developer // DueDil > > > On Wed, Dec 3, 2014 at 10:54 AM, Diptanu Choudhury <[email protected]> wrote: > > Thanks for the explanation Tom, yeah I just figured that out by reading your > code! You're touching the memory.soft_limit_in_bytes and > memory.limit_in_bytes directly. > > Still curios to understand in which situations Mesos Slave would call the > external containerizer to update the resource limits of a container? My > understanding was that once resource allocation happens for a task, resources > are not taken away until the task exits[fails, crashes or finishes] or Mesos > asks the slave to kill the task. > > On Wed, Dec 3, 2014 at 2:47 AM, Tom Arnfeld <[email protected]> wrote: > Hi Diptanu, > > That's correct, the ECP has the responsibility of updating the resource for a > container, and it will do as new tasks are launched and killed for an > executor. Since docker doesn't support this, our containerizer (Deimos does > the same) goes behind docker to the cgroup for the container and updates the > resources in a very similar way to the mesos-slave. I believe this is also > what the built in Docker containerizer will do. > > https://github.com/duedil-ltd/mesos-docker-containerizer/blob/master/containerizer/commands/update.py#L35 > > Tom. > > -- > > Tom Arnfeld > Developer // DueDil > > > On Wed, Dec 3, 2014 at 10:45 AM, Diptanu Choudhury <[email protected]> wrote: > > Hi, > > I had a quick question about the external containerizer. I see that once the > Task is launched, the ECP can receive the update calls, and the protobuf > message passed to ECP with the update call is containerizer::Update. > > This protobuf has a Resources [list] field so does that mean Mesos might ask > a running task to re-adjust the enforced resource limits? > > How would this work if the ECP was launching docker containers because Docker > doesn't allow changing the resource limits once the container has been > started? > > I am wondering how does Deimos and mesos-docker-containerizer handle this. > > -- > Thanks, > Diptanu Choudhury > Web - www.linkedin.com/in/diptanu > Twitter - @diptanu > > > > > -- > Thanks, > Diptanu Choudhury > Web - www.linkedin.com/in/diptanu > Twitter - @diptanu > >
