Re: Implementing an Executor
Hi Diptanu, Thanks for your feedback. This sounds like the approach we'll be taking, too. On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury dipta...@gmail.com wrote: Hi Janet, We implemented the same in our Titan Mesos Executor. We have an executor manage multiple containers. We didn't want the overhead of running multiple executors to manage multiple containers. We are using RxJava heavily in our codebase, so whenever a launchTask callback is invoked in the MesosExecutor we publish an event and return immediately from the callback so that our executor can handle more callbacks from Mesos Slave. But the event which is published is picked up by the subscriber and that starts the workflow of downloading a container, mount volumes, sets up networking, sets up logging, and finally starts a monitoring process for the container. The monitoring process has the reference to the executor driver and we send back Mesos status updates whenever the state of the process changes. On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi David, Thanks for your suggestion. This approach sounds promising for what I need to do. I'm going to have to try this out. Thanks! Janet -- Janet Borschowa CodeFutures Corporation On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg dsg123456...@gmail.com wrote: One cool feature of the docker containerizer is that you can actually launch your executor inside the docker image, so that you can just layer the executor's custom logic on top of whatever container you desire. This way, you can more easily control what's happening in the docker image, and still leverage the containerizer. On Thursday, November 20, 2014, Tim Chen t...@mesosphere.io wrote: Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu
Re: Implementing an Executor
There are some downsides to this as well - For ex, if the executor crashes Mesos would get a TASK_LOST while the container might still be running. So we are doing something similar to Aurora's GC Executor to clean up containers behind us. On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi Diptanu, Thanks for your feedback. This sounds like the approach we'll be taking, too. On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury dipta...@gmail.com wrote: Hi Janet, We implemented the same in our Titan Mesos Executor. We have an executor manage multiple containers. We didn't want the overhead of running multiple executors to manage multiple containers. We are using RxJava heavily in our codebase, so whenever a launchTask callback is invoked in the MesosExecutor we publish an event and return immediately from the callback so that our executor can handle more callbacks from Mesos Slave. But the event which is published is picked up by the subscriber and that starts the workflow of downloading a container, mount volumes, sets up networking, sets up logging, and finally starts a monitoring process for the container. The monitoring process has the reference to the executor driver and we send back Mesos status updates whenever the state of the process changes. On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi David, Thanks for your suggestion. This approach sounds promising for what I need to do. I'm going to have to try this out. Thanks! Janet -- Janet Borschowa CodeFutures Corporation On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg dsg123456...@gmail.com wrote: One cool feature of the docker containerizer is that you can actually launch your executor inside the docker image, so that you can just layer the executor's custom logic on top of whatever container you desire. This way, you can more easily control what's happening in the docker image, and still leverage the containerizer. On Thursday, November 20, 2014, Tim Chen t...@mesosphere.io wrote: Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu
Re: Implementing an Executor
Sorry this is a bit of a tangent to the thread: For ex, if the executor crashes Mesos would get a TASK_LOST while the container might still be running. Mesos should destroy the container when the executor exits, are you seeing otherwise? So we are doing something similar to Aurora's GC Executor to clean up containers behind us. Just to make sure you're aware, you don't need to do this with 0.21.0+, now that we provide reconciliation support. :) On Mon, Dec 1, 2014 at 12:56 PM, Diptanu Choudhury dipta...@gmail.com wrote: There are some downsides to this as well - For ex, if the executor crashes Mesos would get a TASK_LOST while the container might still be running. So we are doing something similar to Aurora's GC Executor to clean up containers behind us. On Mon, Dec 1, 2014 at 10:19 AM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi Diptanu, Thanks for your feedback. This sounds like the approach we'll be taking, too. On Sat, Nov 29, 2014 at 11:34 AM, Diptanu Choudhury dipta...@gmail.com wrote: Hi Janet, We implemented the same in our Titan Mesos Executor. We have an executor manage multiple containers. We didn't want the overhead of running multiple executors to manage multiple containers. We are using RxJava heavily in our codebase, so whenever a launchTask callback is invoked in the MesosExecutor we publish an event and return immediately from the callback so that our executor can handle more callbacks from Mesos Slave. But the event which is published is picked up by the subscriber and that starts the workflow of downloading a container, mount volumes, sets up networking, sets up logging, and finally starts a monitoring process for the container. The monitoring process has the reference to the executor driver and we send back Mesos status updates whenever the state of the process changes. On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi David, Thanks for your suggestion. This approach sounds promising for what I need to do. I'm going to have to try this out. Thanks! Janet -- Janet Borschowa CodeFutures Corporation On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg dsg123456...@gmail.com wrote: One cool feature of the docker containerizer is that you can actually launch your executor inside the docker image, so that you can just layer the executor's custom logic on top of whatever container you desire. This way, you can more easily control what's happening in the docker image, and still leverage the containerizer. On Thursday, November 20, 2014, Tim Chen t...@mesosphere.io wrote: Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task
Re: Implementing an Executor
to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu
Re: Implementing an Executor
) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu
Re: Implementing an Executor
Hi Janet, We implemented the same in our Titan Mesos Executor. We have an executor manage multiple containers. We didn't want the overhead of running multiple executors to manage multiple containers. We are using RxJava heavily in our codebase, so whenever a launchTask callback is invoked in the MesosExecutor we publish an event and return immediately from the callback so that our executor can handle more callbacks from Mesos Slave. But the event which is published is picked up by the subscriber and that starts the workflow of downloading a container, mount volumes, sets up networking, sets up logging, and finally starts a monitoring process for the container. The monitoring process has the reference to the executor driver and we send back Mesos status updates whenever the state of the process changes. On Thu, Nov 20, 2014 at 8:40 AM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi David, Thanks for your suggestion. This approach sounds promising for what I need to do. I'm going to have to try this out. Thanks! Janet -- Janet Borschowa CodeFutures Corporation On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg dsg123456...@gmail.com wrote: One cool feature of the docker containerizer is that you can actually launch your executor inside the docker image, so that you can just layer the executor's custom logic on top of whatever container you desire. This way, you can more easily control what's happening in the docker image, and still leverage the containerizer. On Thursday, November 20, 2014, Tim Chen t...@mesosphere.io wrote: Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation -- Thanks, Diptanu Choudhury Web - www.linkedin.com/in/diptanu Twitter - @diptanu http://twitter.com/diptanu
Re: Implementing an Executor
Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Re: Implementing an Executor
One cool feature of the docker containerizer is that you can actually launch your executor inside the docker image, so that you can just layer the executor's custom logic on top of whatever container you desire. This way, you can more easily control what's happening in the docker image, and still leverage the containerizer. On Thursday, November 20, 2014, Tim Chen t...@mesosphere.io wrote: Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com javascript:_e(%7B%7D,'cvml','t...@duedil.com'); wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com javascript:_e(%7B%7D,'cvml','janet.borsch...@codefutures.com'); wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Re: Implementing an Executor
Hi Tom, Thank you - this helps a lot! Janet On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Re: Implementing an Executor
Hi Tim, I'd like to get the NetworkSetting's IPAddress and Ports from the launched docker container right now. But there is other information that I'll probably need in the future such as the Volumes. What would be ideal is if one could get the information about the docker container that is available via docker's inspect command. Thanks, Janet On Thu, Nov 20, 2014 at 1:23 AM, Tim Chen t...@mesosphere.io wrote: Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Re: Implementing an Executor
Hi David, Thanks for your suggestion. This approach sounds promising for what I need to do. I'm going to have to try this out. Thanks! Janet -- Janet Borschowa CodeFutures Corporation On Thu, Nov 20, 2014 at 5:04 AM, David Greenberg dsg123456...@gmail.com wrote: One cool feature of the docker containerizer is that you can actually launch your executor inside the docker image, so that you can just layer the executor's custom logic on top of whatever container you desire. This way, you can more easily control what's happening in the docker image, and still leverage the containerizer. On Thursday, November 20, 2014, Tim Chen t...@mesosphere.io wrote: Hi Janet, Can you elaborate more what you like to get back from the docker container that you launched? Thanks, Tim On Wed, Nov 19, 2014 at 5:22 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Implementing an Executor
Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Re: Implementing an Executor
Hi Janet, Great to hear you're using Mesos! It's not the best idea to block in callbacks from the mesos drivers, either in the Executor or Framework, this is because you won't be notified correctly when other events happen (master failover, shutdown request, kill task request) as I believe the driver will guarantee you only get one callback at once and uses a blocking call into userland code. The slave and master become aware of task status by the executor correctly sending the TaskStatus message with TASK_STARTING, TASK_RUNNING, TASK_LOST, TASK_FAILED and TASK_FINISHED. Your executor is guaranteed to be alive for your task to be alive (unless there are any failures cases i'm not aware of) so you it's easier to monitor your task from the outside (if it's a subprocess, for example). Your use of the docker containerizer will also contribute to this behaviour as the mesos slave is going to kill the container ASAP after the executor disconnects. Sending task status updates should do the trick for you here. Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Re: Implementing an Executor
Hi Tom, Thanks for your reply. To be sure I understand correctly, my executor launches my task. (Which is actually launching a docker container without using the docker containerizer at all. I'm directly communicating with the docker daemon in my executor.) When the task has been launched, I send a status update TASK_RUNNING and the call to the executor returns so that it isn't blocking. Now, I need some way of watching the status of this task outside of mesos (in my case, the task is a docker container) and then, when I see it is done I notify the framework with another status update, e.g. TASK_FINISHED or TASK_FAILED. I was hoping there was missing something. Even if I was not launching a container but just some other task like a short running app, it seems there's not a way to have Mesos to keep an eye on the tasks' status to tell when/if it completes when using a custom Executor. Instead I need another process to do that. Thanks for your help! Janet -- On Wed, Nov 19, 2014 at 2:29 PM, Tom Arnfeld t...@duedil.com wrote: Hi Janet, Great to hear you're using Mesos! It's not the best idea to block in callbacks from the mesos drivers, either in the Executor or Framework, this is because you won't be notified correctly when other events happen (master failover, shutdown request, kill task request) as I believe the driver will guarantee you only get one callback at once and uses a blocking call into userland code. The slave and master become aware of task status by the executor correctly sending the TaskStatus message with TASK_STARTING, TASK_RUNNING, TASK_LOST, TASK_FAILED and TASK_FINISHED. Your executor is guaranteed to be alive for your task to be alive (unless there are any failures cases i'm not aware of) so you it's easier to monitor your task from the outside (if it's a subprocess, for example). Your use of the docker containerizer will also contribute to this behaviour as the mesos slave is going to kill the container ASAP after the executor disconnects. Sending task status updates should do the trick for you here. Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
Re: Implementing an Executor
Hi Janet, Oh sorry my mistake, I didn't read your email correctly, I thought you were using the containerizer. What you're doing here is actually going to be quite difficult to do, the mesos docker containerizer has some quite complex logic implemented to ensure the slave stays in sync with the containers that are running, and kills anything that goes rogue. It's going to be non-trivial for you to do that from the executor, though I guess you could make use of the docker events API or poll other endpoints in the API to check the status of your containers, and off the back of that send status updates to the cluster. Doing this however brings no guarantees that if your executor dies exceptionally (perhaps OOMd) the containers spawned will die... they'll keep running in the background and it'll be hard for you to know the state of your containers on the cluster. You probably want to be aware (if you don't know already) that the resource limits assigned to your tasks aren't going to be enforced by mesos because docker is running outside of its control. You'll need to pass the correct CPU/Memory limit parameters to your docker containers to ensure this happens correctly. Here are the docker API docs; https://docs.docker.com/reference/api/docker_remote_api_v1.15/ Something you might want to consider, if all you're trying to do is allow your container access to details about itself (e.g `docker inspect`) is to open up the docker remote API to be queried by your containers on the slave, and switch to using the mesos docker containerizer. I hope that helps somewhat! Tom. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Wed, Nov 19, 2014 at 10:16 PM, Janet Borschowa janet.borsch...@codefutures.com wrote: Hi, I'm implementing an executor which is used by the mesos slave to launch tasks. The tasks are to launch a docker container - this is because I need more info about the launched container than what the docker containerizer returns. Is it OK to block in the executor's launchTask method until the task completes? If not, how does the framework discover when that task completes? I could spawn a process which notifies my executor when the task completes and then have my executor send a status update. Or is there some other recommended way to deal with this when the task could run for an indefinite period of time before completing its work? Thanks! Janet -- Janet Borschowa CodeFutures Corporation
