Re: Using mesos' cfs limits on a docker container?

[Zhitao Li]

Hi Mark,

About the curl issue on SSL, can you please check
https://issues.apache.org/jira/browse/MESOS-6005 is similar to what you see?

On Sun, Aug 14, 2016 at 12:23 PM, Artem Harutyunyan 
wrote:

> Hi Mark,
>
> Good to hear you figured it out. Can you please post curl errors that you
> were observing and describe your image repository setup? I'd like to make
> sure that we have instructions on how to mitigate those.
>
> Artem.
>
> On Sunday, August 14, 2016, Mark Hammons 
> wrote:
>
>> In specific, I wanted the process control capabilities of a mesos
>> framework with custom schedulers and executors, but wanted to run my tasks
>> in a framework definable environment (like running my tasks on a copy of
>> Ubuntu 14 with certain libs installed). Using mixed-mode containerization
>> worked with some fiddling, but it was painful in certain ways. The sandbox
>> mounted in a mixed-mode container wasn't accessible from within the
>> container thanks to selinux unless I ran the container in privileged mode
>> and the cou limits per executor were no longer enforced unlike a mesos task
>> with cfs isolation enabled. Further, setting up the default working
>> directory and user was a pain.
>>
>> Unified mode (also called mesos containerizer for some reason) solves a
>> lot of these issues, though using it with private image repositories was
>> not as straightforward as the docker containerizer. I eventually had to use
>> an image directory to get that working, cause curl kept throwing vague ssl
>> errors(I'm fairly certain this is due to my private image repository not
>> having https set up since it's a test environment).
>>
>> Once I get things set up and cleaned up I'll post a more involved guide
>> on how to get this particular use case setup and running, especially a part
>> on preparing your container image for use with mesos.
>>
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>>
>> On 14 Aug 2016, at 18:11, Erik Weathers  wrote:
>>
>> What was the problem and how did you overcome it?  (i.e. This would be a
>> sad resolution to this thread for someone faced with this same problem in
>> the future.)
>>
>> On Sunday, August 14, 2016, Mark Hammons 
>> wrote:
>>
>>> I finally got this working after fiddling with it all night. It works
>>> great so far!
>>>
>>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>>> 0603695656
>>>
>>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>>
>>> If you're not against running Docker containers without the Docker
>>> daemon, try using the Unified containerizer.
>>> See the latter half of this document: http://mesos.apache.org/docume
>>> ntation/latest/mesos-containerizer/
>>> 
>>>
>>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>>
 Hi All,



 I was having a lot of success having mesos force sandboxed programs to
 work within cpu and memory constraints, but when I added docker into the
 mix, the cpu limitations go out the window (not sure about the memory
 limitations. Is there any way to mix these two methods of isolation? I'd
 like my executor/algorithm to run inside a docker container, but have that
 container's memory and cpu usage controlled by systemd/mesos.



 Thanks,

 Mark
 --

 Mark Hammons - +33 06 03 69 56 56

 Research Engineer @ BioEmergences
 

 Lab Phone: 01 69 82 34 19

>>>
>>>


-- 
Cheers,

Zhitao Li

Re: Using mesos' cfs limits on a docker container?

[Artem Harutyunyan]

Hi Mark,

Good to hear you figured it out. Can you please post curl errors that you
were observing and describe your image repository setup? I'd like to make
sure that we have instructions on how to mitigate those.

Artem.

On Sunday, August 14, 2016, Mark Hammons 
wrote:

> In specific, I wanted the process control capabilities of a mesos
> framework with custom schedulers and executors, but wanted to run my tasks
> in a framework definable environment (like running my tasks on a copy of
> Ubuntu 14 with certain libs installed). Using mixed-mode containerization
> worked with some fiddling, but it was painful in certain ways. The sandbox
> mounted in a mixed-mode container wasn't accessible from within the
> container thanks to selinux unless I ran the container in privileged mode
> and the cou limits per executor were no longer enforced unlike a mesos task
> with cfs isolation enabled. Further, setting up the default working
> directory and user was a pain.
>
> Unified mode (also called mesos containerizer for some reason) solves a
> lot of these issues, though using it with private image repositories was
> not as straightforward as the docker containerizer. I eventually had to use
> an image directory to get that working, cause curl kept throwing vague ssl
> errors(I'm fairly certain this is due to my private image repository not
> having https set up since it's a test environment).
>
> Once I get things set up and cleaned up I'll post a more involved guide on
> how to get this particular use case setup and running, especially a part on
> preparing your container image for use with mesos.
>
> Mark Edgar Hammons II - Research Engineer at BioEmergences
> 0603695656
>
> On 14 Aug 2016, at 18:11, Erik Weathers  > wrote:
>
> What was the problem and how did you overcome it?  (i.e. This would be a
> sad resolution to this thread for someone faced with this same problem in
> the future.)
>
> On Sunday, August 14, 2016, Mark Hammons  > wrote:
>
>> I finally got this working after fiddling with it all night. It works
>> great so far!
>>
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>>
>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>
>> If you're not against running Docker containers without the Docker
>> daemon, try using the Unified containerizer.
>> See the latter half of this document: http://mesos.apache.org/docume
>> ntation/latest/mesos-containerizer/
>> 
>>
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>
>>> Hi All,
>>>
>>>
>>>
>>> I was having a lot of success having mesos force sandboxed programs to
>>> work within cpu and memory constraints, but when I added docker into the
>>> mix, the cpu limitations go out the window (not sure about the memory
>>> limitations. Is there any way to mix these two methods of isolation? I'd
>>> like my executor/algorithm to run inside a docker container, but have that
>>> container's memory and cpu usage controlled by systemd/mesos.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>> --
>>>
>>> Mark Hammons - +33 06 03 69 56 56
>>>
>>> Research Engineer @ BioEmergences
>>> 
>>>
>>> Lab Phone: 01 69 82 34 19
>>>
>>
>>

Re: Using mesos' cfs limits on a docker container?

[Avinash Sridharan]

Hi Mark,
 That would be awesome (user facing documentation forthe
`UnifiedContainerizer`). We have bits and pieces of the unified
containerizer (and what it actually is), but would be great to land a more
comprehensive documentation into its motivation and usability. May be have
a separate `unified-containerized.md` and update the `
https://github.com/apache/mesos/blob/master/docs/mesos-containerizer.md`
with a link to the new 'unified-containerized.md'. Jie ??

Would also be very useful if you file some JIRAs related to the problems
you faced in setting up docker private registries to be used with the
unified-containerizer.

Thanks,
Avinash

On Sun, Aug 14, 2016 at 10:36 AM, Mark Hammons <
mark.hamm...@inaf.cnrs-gif.fr> wrote:

> In specific, I wanted the process control capabilities of a mesos
> framework with custom schedulers and executors, but wanted to run my tasks
> in a framework definable environment (like running my tasks on a copy of
> Ubuntu 14 with certain libs installed). Using mixed-mode containerization
> worked with some fiddling, but it was painful in certain ways. The sandbox
> mounted in a mixed-mode container wasn't accessible from within the
> container thanks to selinux unless I ran the container in privileged mode
> and the cou limits per executor were no longer enforced unlike a mesos task
> with cfs isolation enabled. Further, setting up the default working
> directory and user was a pain.
>
> Unified mode (also called mesos containerizer for some reason) solves a
> lot of these issues, though using it with private image repositories was
> not as straightforward as the docker containerizer. I eventually had to use
> an image directory to get that working, cause curl kept throwing vague ssl
> errors(I'm fairly certain this is due to my private image repository not
> having https set up since it's a test environment).
>
> Once I get things set up and cleaned up I'll post a more involved guide on
> how to get this particular use case setup and running, especially a part on
> preparing your container image for use with mesos.
>
> Mark Edgar Hammons II - Research Engineer at BioEmergences
> 0603695656
>
> On 14 Aug 2016, at 18:11, Erik Weathers  wrote:
>
> What was the problem and how did you overcome it?  (i.e. This would be a
> sad resolution to this thread for someone faced with this same problem in
> the future.)
>
> On Sunday, August 14, 2016, Mark Hammons 
> wrote:
>
>> I finally got this working after fiddling with it all night. It works
>> great so far!
>>
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>>
>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>
>> If you're not against running Docker containers without the Docker
>> daemon, try using the Unified containerizer.
>> See the latter half of this document: http://mesos.apache.org/docume
>> ntation/latest/mesos-containerizer/
>> 
>>
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>
>>> Hi All,
>>>
>>>
>>>
>>> I was having a lot of success having mesos force sandboxed programs to
>>> work within cpu and memory constraints, but when I added docker into the
>>> mix, the cpu limitations go out the window (not sure about the memory
>>> limitations. Is there any way to mix these two methods of isolation? I'd
>>> like my executor/algorithm to run inside a docker container, but have that
>>> container's memory and cpu usage controlled by systemd/mesos.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>> --
>>>
>>> Mark Hammons - +33 06 03 69 56 56
>>>
>>> Research Engineer @ BioEmergences
>>> 
>>>
>>> Lab Phone: 01 69 82 34 19
>>>
>>
>>


-- 
Avinash Sridharan, Mesosphere
+1 (323) 702 5245

Re: Using mesos' cfs limits on a docker container?

[Mark Hammons]

In specific, I wanted the process control capabilities of a mesos framework 
with custom schedulers and executors, but wanted to run my tasks in a framework 
definable environment (like running my tasks on a copy of Ubuntu 14 with 
certain libs installed). Using mixed-mode containerization worked with some 
fiddling, but it was painful in certain ways. The sandbox mounted in a 
mixed-mode container wasn't accessible from within the container thanks to 
selinux unless I ran the container in privileged mode and the cou limits per 
executor were no longer enforced unlike a mesos task with cfs isolation 
enabled. Further, setting up the default working directory and user was a pain.

Unified mode (also called mesos containerizer for some reason) solves a lot of 
these issues, though using it with private image repositories was not as 
straightforward as the docker containerizer. I eventually had to use an image 
directory to get that working, cause curl kept throwing vague ssl errors(I'm 
fairly certain this is due to my private image repository not having https set 
up since it's a test environment).

Once I get things set up and cleaned up I'll post a more involved guide on how 
to get this particular use case setup and running, especially a part on 
preparing your container image for use with mesos.

Mark Edgar Hammons II - Research Engineer at BioEmergences
0603695656

> On 14 Aug 2016, at 18:11, Erik Weathers  wrote:
> 
> What was the problem and how did you overcome it?  (i.e. This would be a sad 
> resolution to this thread for someone faced with this same problem in the 
> future.)
> 
>> On Sunday, August 14, 2016, Mark Hammons  
>> wrote:
>> I finally got this working after fiddling with it all night. It works great 
>> so far!
>> 
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>> 
>>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>> 
>>> If you're not against running Docker containers without the Docker daemon, 
>>> try using the Unified containerizer.  
>>> See the latter half of this document: 
>>> http://mesos.apache.org/documentation/latest/mesos-containerizer/
>>> 
 On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons 
  wrote:
 Hi All,
  
 I was having a lot of success having mesos force sandboxed programs to 
 work within cpu and memory constraints, but when I added docker into the 
 mix, the cpu limitations go out the window (not sure about the memory 
 limitations. Is there any way to mix these two methods of isolation? I'd 
 like my executor/algorithm to run inside a docker container, but have that 
 container's memory and cpu usage controlled by systemd/mesos.
  
 Thanks,
 Mark
 --
 Mark Hammons - +33 06 03 69 56 56
 Research Engineer @ BioEmergences
 Lab Phone: 01 69 82 34 19

Re: Using mesos' cfs limits on a docker container?

[haosdent]

Personally, I suggest to use the approach @Joseph and @Avinash mentioned.
Because zhitao and my patches require Docker >= 1.7.0 .

On Mon, Aug 15, 2016 at 1:27 AM, haosdent  wrote:

> Not sure if this related to https://issues.apache.org/
> jira/browse/MESOS-2154
> So far we have a quick workaround: specify the `cpu-period` and
> `cpu-quota` in the parameters field of `DockerInfo`. Then `Docker::run`
> would delegate this to the docker daemon.
>
> And recently zhitao and me work on the fix for this, we have some under
> reviewing patches. I think it should be fixed shortly once zhitao and my
> patches ready.
>
> On Mon, Aug 15, 2016 at 12:11 AM, Erik Weathers 
> wrote:
>
>> What was the problem and how did you overcome it?  (i.e. This would be a
>> sad resolution to this thread for someone faced with this same problem in
>> the future.)
>>
>>
>> On Sunday, August 14, 2016, Mark Hammons 
>> wrote:
>>
>>> I finally got this working after fiddling with it all night. It works
>>> great so far!
>>>
>>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>>> 0603695656
>>>
>>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>>
>>> If you're not against running Docker containers without the Docker
>>> daemon, try using the Unified containerizer.
>>> See the latter half of this document: http://mesos.apache.org/docume
>>> ntation/latest/mesos-containerizer/
>>> 
>>>
>>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>>
 Hi All,



 I was having a lot of success having mesos force sandboxed programs to
 work within cpu and memory constraints, but when I added docker into the
 mix, the cpu limitations go out the window (not sure about the memory
 limitations. Is there any way to mix these two methods of isolation? I'd
 like my executor/algorithm to run inside a docker container, but have that
 container's memory and cpu usage controlled by systemd/mesos.



 Thanks,

 Mark
 --

 Mark Hammons - +33 06 03 69 56 56

 Research Engineer @ BioEmergences
 

 Lab Phone: 01 69 82 34 19

>>>
>>>
>
>
> --
> Best Regards,
> Haosdent Huang
>



-- 
Best Regards,
Haosdent Huang

Re: Using mesos' cfs limits on a docker container?

[haosdent]

Not sure if this related to https://issues.apache.org/jira/browse/MESOS-2154
So far we have a quick workaround: specify the `cpu-period` and `cpu-quota`
in the parameters field of `DockerInfo`. Then `Docker::run` would delegate
this to the docker daemon.

And recently zhitao and me work on the fix for this, we have some under
reviewing patches. I think it should be fixed shortly once zhitao and my
patches ready.

On Mon, Aug 15, 2016 at 12:11 AM, Erik Weathers 
wrote:

> What was the problem and how did you overcome it?  (i.e. This would be a
> sad resolution to this thread for someone faced with this same problem in
> the future.)
>
>
> On Sunday, August 14, 2016, Mark Hammons 
> wrote:
>
>> I finally got this working after fiddling with it all night. It works
>> great so far!
>>
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>>
>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>
>> If you're not against running Docker containers without the Docker
>> daemon, try using the Unified containerizer.
>> See the latter half of this document: http://mesos.apache.org/docume
>> ntation/latest/mesos-containerizer/
>> 
>>
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>
>>> Hi All,
>>>
>>>
>>>
>>> I was having a lot of success having mesos force sandboxed programs to
>>> work within cpu and memory constraints, but when I added docker into the
>>> mix, the cpu limitations go out the window (not sure about the memory
>>> limitations. Is there any way to mix these two methods of isolation? I'd
>>> like my executor/algorithm to run inside a docker container, but have that
>>> container's memory and cpu usage controlled by systemd/mesos.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>> --
>>>
>>> Mark Hammons - +33 06 03 69 56 56
>>>
>>> Research Engineer @ BioEmergences
>>> 
>>>
>>> Lab Phone: 01 69 82 34 19
>>>
>>
>>


-- 
Best Regards,
Haosdent Huang

Re: Using mesos' cfs limits on a docker container?

[Erik Weathers]

What was the problem and how did you overcome it?  (i.e. This would be a
sad resolution to this thread for someone faced with this same problem in
the future.)

On Sunday, August 14, 2016, Mark Hammons 
wrote:

> I finally got this working after fiddling with it all night. It works
> great so far!
>
> Mark Edgar Hammons II - Research Engineer at BioEmergences
> 0603695656
>
> On 14 Aug 2016, at 04:50, Joseph Wu  > wrote:
>
> If you're not against running Docker containers without the Docker daemon,
> try using the Unified containerizer.
> See the latter half of this document: http://mesos.apache.org/
> documentation/latest/mesos-containerizer/
> 
>
> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
> mark.hamm...@inaf.cnrs-gif.fr
> > wrote:
>
>> Hi All,
>>
>>
>>
>> I was having a lot of success having mesos force sandboxed programs to
>> work within cpu and memory constraints, but when I added docker into the
>> mix, the cpu limitations go out the window (not sure about the memory
>> limitations. Is there any way to mix these two methods of isolation? I'd
>> like my executor/algorithm to run inside a docker container, but have that
>> container's memory and cpu usage controlled by systemd/mesos.
>>
>>
>>
>> Thanks,
>>
>> Mark
>> --
>>
>> Mark Hammons - +33 06 03 69 56 56
>>
>> Research Engineer @ BioEmergences
>> 
>>
>> Lab Phone: 01 69 82 34 19
>>
>
>

Re: Using mesos' cfs limits on a docker container?

[Mark Hammons]

I finally got this working after fiddling with it all night. It works great so 
far!

Mark Edgar Hammons II - Research Engineer at BioEmergences
0603695656

> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
> 
> If you're not against running Docker containers without the Docker daemon, 
> try using the Unified containerizer.  
> See the latter half of this document: 
> http://mesos.apache.org/documentation/latest/mesos-containerizer/
> 
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons 
>>  wrote:
>> Hi All,
>>  
>> I was having a lot of success having mesos force sandboxed programs to work 
>> within cpu and memory constraints, but when I added docker into the mix, the 
>> cpu limitations go out the window (not sure about the memory limitations. Is 
>> there any way to mix these two methods of isolation? I'd like my 
>> executor/algorithm to run inside a docker container, but have that 
>> container's memory and cpu usage controlled by systemd/mesos.
>>  
>> Thanks,
>> Mark
>> --
>> Mark Hammons - +33 06 03 69 56 56
>> Research Engineer @ BioEmergences
>> Lab Phone: 01 69 82 34 19
> 

Re: Using mesos' cfs limits on a docker container?

[Avinash Sridharan]

+1 . Was just about to suggest the same !!

On Sat, Aug 13, 2016 at 7:50 PM, Joseph Wu  wrote:

> If you're not against running Docker containers without the Docker daemon,
> try using the Unified containerizer.
> See the latter half of this document: http://mesos.apache.org/
> documentation/latest/mesos-containerizer/
>
> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>
>> Hi All,
>>
>>
>>
>> I was having a lot of success having mesos force sandboxed programs to
>> work within cpu and memory constraints, but when I added docker into the
>> mix, the cpu limitations go out the window (not sure about the memory
>> limitations. Is there any way to mix these two methods of isolation? I'd
>> like my executor/algorithm to run inside a docker container, but have that
>> container's memory and cpu usage controlled by systemd/mesos.
>>
>>
>>
>> Thanks,
>>
>> Mark
>> --
>>
>> Mark Hammons - +33 06 03 69 56 56
>>
>> Research Engineer @ BioEmergences 
>>
>> Lab Phone: 01 69 82 34 19
>>
>
>


-- 
Avinash Sridharan, Mesosphere
+1 (323) 702 5245

Re: Using mesos' cfs limits on a docker container?

[Joseph Wu]

If you're not against running Docker containers without the Docker daemon,
try using the Unified containerizer.
See the latter half of this document:
http://mesos.apache.org/documentation/latest/mesos-containerizer/

On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons  wrote:

> Hi All,
>
>
>
> I was having a lot of success having mesos force sandboxed programs to
> work within cpu and memory constraints, but when I added docker into the
> mix, the cpu limitations go out the window (not sure about the memory
> limitations. Is there any way to mix these two methods of isolation? I'd
> like my executor/algorithm to run inside a docker container, but have that
> container's memory and cpu usage controlled by systemd/mesos.
>
>
>
> Thanks,
>
> Mark
> --
>
> Mark Hammons - +33 06 03 69 56 56
>
> Research Engineer @ BioEmergences 
>
> Lab Phone: 01 69 82 34 19
>

Using mesos' cfs limits on a docker container?

[Mark Hammons]

Hi All,

I was having a lot of success having mesos force sandboxed programs to work
within cpu and memory constraints, but when I added docker into the mix, the
cpu limitations go out the window (not sure about the memory limitations. Is
there any way to mix these two methods of isolation? I'd like my
executor/algorithm to run inside a docker container, but have that container's
memory and cpu usage controlled by systemd/mesos.

Thanks,
Mark--
Mark Hammons - +33 06 03 69 56 56
Research Engineer @ BioEmergences[1]
Lab Phone: 01 69 82 34 19


[1] bioemergences.iscpif.fr


signature.asc
Description: This is a digitally signed message part.