Customized framework with Mesos persistent volumn can come to your desired feature, IMO. Which will recover the volumn after framework deregistered.
> 在 2015年11月10日，23:43，Aaron Carey <[email protected]> 写道： > > Yeah.. it'd be nice to do it natively though :) > > From: Plotka, Bartlomiej [[email protected]] > Sent: 10 November 2015 15:38 > To: [email protected] > Subject: RE: Custom python executor with Docker > > This is somehow possible using Kubernetes over Mesos: > https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/mesos.md > > Kind Regards, > Bartek Plotka > > From: Aaron Carey [mailto:[email protected]] > Sent: Tuesday, November 10, 2015 4:33 PM > To: [email protected] > Subject: RE: Custom python executor with Docker > > We would also be interested in some sort of standardised DockerExecutor which > would allow us to add pre and post launch steps. > > Also having the ability to run two containers together as one task would be > very useful (ie on the same host and linked together) > > From: Tom Fordon [[email protected]] > Sent: 12 August 2015 00:28 > To: [email protected] > Subject: Re: Custom python executor with Docker > > We ended up implementing a solution where we did the pre/post steps as > separate mesos tasks and adding logic to our scheduler to ensure they were > run on the same machine. If anybody knows of a standard / openly available > DockerExecutor like what is described below, my team would be greatly > interested. > > On Fri, Aug 7, 2015 at 4:01 AM, Kapil Malik <[email protected]> wrote: > Hi, > > We have a similar usecase while running multi-user workloads on mesos. Users > provide docker images encapsulating application logic, which we (we = say > some “Central API”) schedule on Chronos / Marathon. However, we need to run > some standard pre / post steps for every docker submitted by users. We have > following options – > > 1. Ask every user to embed their logic inside a pre-defined docker > template which will perform pre/post steps. > è This is error prone, makes us dependent on whether the users followed > template, and not very popular with users either. > > 2. Extend every user docker (FROM <>) and find a way to add pre-post > steps in our docker. Refer this docker when scheduling on chronos / marathon. > è Building new dockers does not scale as users and applications grow > > 3. Write a custom executor which will perform the pre-post steps and > manage the user docker lifetime. > è Deals with user docker lifetime and is obviously complex. > > Is there a standard / openly available DockerExecutor which manages the > docker lifetime and which I can extend to build my custom executor? This way > I will be concerned only with my custom logic (pre/post steps) and still get > benefits of a standard way to manage docker containers. > > Btw, thanks for the meaningful discussion below, it is very helpful. > > Thanks and regards, > > Kapil Malik | [email protected] | 33430 / 8800836581 > > From: James DeFelice [mailto:[email protected]] > Sent: 09 April 2015 18:12 > To: [email protected] > Subject: Re: Custom python executor with Docker > > If you can run the pre/post steps in a container then I'd recommend building > a Docker image that includes your pre/post step scripting + your algorithm > and launching it using the built-in mesos Docker containerizer. It's much > simpler than managing the lifetime of the Docker container yourself. > > On Thu, Apr 9, 2015 at 8:29 AM, Tom Fordon <[email protected]> wrote: > Thanks for all the responses, I really appreciate the help. Let me try to > state my problem more clearly > > Our project is performing file-based data processing. I would like to keep > the actual algorithm as contained as possible since we are in an R&D setting > and will be getting untested code. We have some pre/post steps that need to > be run on the same box as the actual algorithm: downloading/uploading files > and database calls. > > We can run the pre/post steps and algorithm within the same container. The > algorithm will be a little less contained, but it will work. > > Docker letting you specify a cgroup parent is really exciting. If I invoke a > docker container with the executor as the cgroup-parent are there any other > steps I need to perform? Would I need to do anything special to make mesos > aware of the resource usage, or is that handled since the docker process > would be in the executors cgroup? > > Thanks again, > Tom > > On Tue, Apr 7, 2015 at 8:10 PM, Timothy Chen <[email protected]> wrote: > Hi Tom(s), > > Tom Arnfeld is right, if you want to launch your own docker container > in your custom executor you will have to handle all the issues > yourself and not able to use the Docker containerizer at all. > > Alternatively, you can actually launch your custom executor in a > Docker container by Mesos, by specifying the ContainerInfo in the > ExecutorInfo. > What this means is that your custom executor is already running in a > docker container, and you can do your custom logic afterwards. This > does means you can simply just launch multiple containers in the > executor anymore. > > If there is something you want to do and doesnt' fit these let us know > what you're trying to achieve and we can see what we can do. > > Tim > > On Tue, Apr 7, 2015 at 4:15 PM, Tom Arnfeld <[email protected]> wrote: > > It's not possible to invoke the docker containerizer from outside of Mesos, > > as far as I know. > > > > If you persue this route, you can run into issues with orphaned containers > > as your executor may die for some unknown reason, and the container is still > > running. Recovering from this can be tricky business, so it's better if you > > can adapt your framework design to fit within the Mesos Task/Executor > > pattern. > > > > -- > > > > Tom Arnfeld > > Developer // DueDil > > > > (+44) 7525940046 > > 25 Christopher Street, London, EC2A 2BS > > > > > > On Mon, Apr 6, 2015 at 7:00 PM, Vinod Kone <[email protected]> wrote: > >> > >> Tim, do you want answer this? > >> > >> On Wed, Apr 1, 2015 at 7:27 AM, Tom Fordon <[email protected]> wrote: > >>> > >>> Hi. I'm trying to understand using docker within a custom executor. For > >>> each of my tasks, I would like to perform some steps on the node before > >>> launching a docker container. I was planning on writing a custom python > >>> executor for this, but I wasn't sure how to launch docker from within this > >>> executor. > >>> > >>> Can I just call docker in a subprocess using the ContainerInfo from the > >>> Task? If I do this, how does the Containerizer fit in? > >>> > >>> Thank you, > >>> Tom Fordon > >> > >> > > > > > > > -- > James DeFelice > 585.241.9488 (voice) > 650.649.6071 (fax) > > --------------------------------------------------------------------- > Intel Technology Poland sp. z o.o. > ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII > Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP > 957-07-52-316 | Kapitał zakładowy 200.000 PLN. > Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i > może zawierać informacje poufne. W razie przypadkowego otrzymania tej > wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; > jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione. > This e-mail and any attachments may contain confidential material for the > sole use of the intended recipient(s). If you are not the intended recipient, > please contact the sender and delete all copies; any review or distribution > by others is strictly prohibited.
