Yes, although, there's a nuance to this specific situation. Here, the same
executor is being used for multiple tasks, but, the executor is launching a
different Docker container for each task. I was extending the coarse grain
allocation concept to within the executor (which is in a fine grained
allocation model).
What you mention, we do use already for a different framework, not the one
Diptanu is talking about.
On Wed, Dec 3, 2014 at 11:04 AM, Connor Doyle <con...@mesosphere.io> wrote:
> You're right Sharma, it's dependent upon the framework. If your scheduler
> sets a unique ExecutorID for each TaskInfo, then the executor will not be
> re-used and you won't have to worry about resizing the executor's container
> to accomodate subsequent tasks. This might be a reasonable simplification
> to start with, especially if your executor adds relatively low resource
> overhead.
> --
> Connor
>
>
> > On Dec 3, 2014, at 10:20, Sharma Podila <spod...@netflix.com> wrote:
> >
> > This may have to do with fine-grain Vs coarse-grain resource allocation.
> Things may be easier for you, Diptanu, if you are using one Docker
> container per task (sort of coarse grain). In that case, I believe there's
> no need to alter a running Docker container's resources. Instead, the
> resource update of your executor translates into the right Docker
> containers running. There's some details to be worked out there, I am sure.
> > It sounds like Tom's strategy uses the same Docker container for
> multiple tasks. Tom, do correct me otherwise.
> >
> > On Wed, Dec 3, 2014 at 3:38 AM, Tom Arnfeld <t...@duedil.com> wrote:
> > When Mesos is asked to a launch a task (with either a custom Executor or
> the built in CommandExecutor) it will first spawn the executor which _has_
> to be a system process, launched via command. This process will be launched
> inside of a Docker container when using the previously mentioned
> containerizers.
> >
> > Once the Executor registers with the slave, the slave will send it a
> number of launchTask calls based on the number of tasks queued up for that
> executor. The Executor can then do as it pleases with those tasks, whether
> it's just a sleep(1) or to spawn a subprocess and do some other work. Given
> it is possible for the framework to specify resources for both tasks and
> executors, and the only thing which _has_ to be a system process is the
> executor, the mesos slave will limit the resources of the executor process
> to the sum of (TaskInfo.Executor.Resources + TaskInfo.Resources).
> >
> > Mesos also has the ability to launch new tasks on an already running
> executor, so it's important that mesos is able to dynamically scale the
> resource limits up and down over time. Designing a framework around this
> idea can lead to some complex and powerful workflows which would be a lot
> more complex to build without Mesos.
> >
> > Just for an example... Spark.
> >
> > 1) User launches a job on spark to map over some data
> > 2) Spark launches a first wave of tasks based on the offers it received
> (let's say T1 and T2)
> > 3) Mesos launches executors for those tasks (let's say E1 and E2) on
> different slaves
> > 4) Spark launches another wave of tasks based on offers, and tells mesos
> to use the same executor (E1 and E2)
> > 5) Mesos will simply call launchTasks(T{3,4}) on the two already running
> executors
> >
> > At point (3) mesos is going to launch a Docker container and execute
> your executor. However at (5) the executor is already running so the tasks
> will be handed to the already running executor.
> >
> > Mesos will guarantee you (i'm 99% sure) that the resources for your
> container have been updated to reflect the limits set on the tasks before
> handing the tasks to you.
> >
> > I hope that makes some sense!
> >
> > --
> >
> > Tom Arnfeld
> > Developer // DueDil
> >
> >
> > On Wed, Dec 3, 2014 at 10:54 AM, Diptanu Choudhury <dipta...@gmail.com>
> wrote:
> >
> > Thanks for the explanation Tom, yeah I just figured that out by reading
> your code! You're touching the memory.soft_limit_in_bytes and
> memory.limit_in_bytes directly.
> >
> > Still curios to understand in which situations Mesos Slave would call
> the external containerizer to update the resource limits of a container? My
> understanding was that once resource allocation happens for a task,
> resources are not taken away until the task exits[fails, crashes or
> finishes] or Mesos asks the slave to kill the task.
> >
> > On Wed, Dec 3, 2014 at 2:47 AM, Tom Arnfeld <t...@duedil.com> wrote:
> > Hi Diptanu,
> >
> > That's correct, the ECP has the responsibility of updating the resource
> for a container, and it will do as new tasks are launched and killed for an
> executor. Since docker doesn't support this, our containerizer (Deimos does
> the same) goes behind docker to the cgroup for the container and updates
> the resources in a very similar way to the mesos-slave. I believe this is
> also what the built in Docker containerizer will do.
> >
> >
> https://github.com/duedil-ltd/mesos-docker-containerizer/blob/master/containerizer/commands/update.py#L35
> >
> > Tom.
> >
> > --
> >
> > Tom Arnfeld
> > Developer // DueDil
> >
> >
> > On Wed, Dec 3, 2014 at 10:45 AM, Diptanu Choudhury <dipta...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I had a quick question about the external containerizer. I see that once
> the Task is launched, the ECP can receive the update calls, and the
> protobuf message passed to ECP with the update call is
> containerizer::Update.
> >
> > This protobuf has a Resources [list] field so does that mean Mesos might
> ask a running task to re-adjust the enforced resource limits?
> >
> > How would this work if the ECP was launching docker containers because
> Docker doesn't allow changing the resource limits once the container has
> been started?
> >
> > I am wondering how does Deimos and mesos-docker-containerizer handle
> this.
> >
> > --
> > Thanks,
> > Diptanu Choudhury
> > Web - www.linkedin.com/in/diptanu
> > Twitter - @diptanu
> >
> >
> >
> >
> > --
> > Thanks,
> > Diptanu Choudhury
> > Web - www.linkedin.com/in/diptanu
> > Twitter - @diptanu
> >
> >
>
>
