Re: resources in Mesos UI but none given to Spark

[Peter Figliozzi]

Problem solved-- I was incorrectly specifying the spark directory in the
command-line argument.  It wants the spark root directory, not the bin.

I unpacked mine in /opt, so, the Right way:

spark.mesos.executor.home=/opt/spark-2.0.0-bin-hadoop2.7/

Wrong way:

spark.mesos.executor.home=/opt/spark-2.0.0-bin-hadoop2.7/bin

On Sun, Aug 14, 2016 at 9:09 PM, Peter Figliozzi 
wrote:

>
> I notice the first thing that happens when I run the spark-shell is *three
> failed 'sandbox' tasks* appearing in the Mesos UI.  (One for each agent.)
> Then another three, as if it tried twice.
>
> I attached the Mesos logs from the master and one of the agents.
>
> All of this happens when I run the spark-shell... before any commands in
> the shell.
>
> Summary/Master:
>
>- Received SUBSCRIBE call for spark shell
>- Adds and launches task with 4 cpus and 1408 mem on each agent
>- Something about "Processing ACCEPT call for offers"
>- TASK_RUNNING on all agents
>- Processing ACKNOWLEDGE calls all agents
>- Status update TASK_FAILED from all agents
>
> Summary/Agent:
>
>- "Got assigned" task 1
>- Trying to chown /var/lib/mesos/slaves/. to user 'peter'
>- Launching executor 1 of framework with resources cpus 0.1, mem 32 in
>work directory...
>- Queuing task '1' for executor '1'...
>- docker: No container info found, skipping launch
>- containerizer: starting container...
>- linux_launcher: cloning child process with flags =
>- systemd: assigned child process '17043' to 'mesos_executors.slice'
>- ... some more slave stuff 
>- Handling status update TASK_FAILED for task 1
>
> Could someone explain what's supposed to happen here when running the
> spark shell?
>
> Thanks,
>
> Pete
>
>
> On Sun, Aug 14, 2016 at 6:57 PM, Michael Gummelt 
> wrote:
>
>> Turning on Spark debug logs in conf/log4j.properties may help.  The
>> problem could be any number of things, including that you don't have enough
>> resources for the default executor sizes.
>>
>> On Sun, Aug 14, 2016 at 2:37 PM, Peter Figliozzi <
>> pete.figlio...@gmail.com> wrote:
>>
>>> Hi All, I am new to Mesos.  I set up a cluster this weekend with 3
>>> agents, 1 master, Mesos 1.0.0.  The resources show in the Mesos UI and the
>>> agents are all in the Agents tab.  So everything looks good from that
>>> vantage point.
>>>
>>> Next I installed Spark 2.0.0 on each agent and the master, in the same
>>> path (/opt/spark) on each machine.  I run the spark-shell from the master
>>> like this:
>>>
>>> ./spark-shell --master mesos://zk://moe:2181/mesos -c
>>> spark.mesos.executor.home=`pwd`
>>>
>>> The shell comes up nicely, however, none of the resources get assigned
>>> to the Spark framework (zeros for everything).
>>>
>>> If I try a simple task like
>>>
>>> sc.parallelize(0 to 10, 8).count
>>>
>>> it fails:
>>>
>>> WARN TaskSchedulerImpl: Initial job has not accepted any resources;
>>> check your cluster UI to ensure that workers are registered and have
>>> sufficient resources
>>>
>>>
>>> I'll post my logs in a little bit if need be.  Hopefully it's a common
>>> newb error and simple fix.
>>>
>>> Thank you
>>>
>>
>>
>>
>> --
>> Michael Gummelt
>> Software Engineer
>> Mesosphere
>>
>
>

what's the difference between mesos and yarn?

[Yu Wei]

Hi guys,


What's the difference between yarn and mesos in practice?


If using yarn, does container still needed?


Thanks,


Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: Debugging Scheduler HTTP API Failures

[Dario Rexin]

Oh, sorry, I didn't see you actually set the header (wall of text ;) ). That's 
an interesting issue, do you set the header case sensitive? I know headers 
shouldn't be case sensitive, but maybe there's a bug in the Mesos code. I have 
not seen this issue before.

> On Aug 14, 2016, at 5:58 PM, Zameer Manji  wrote:
> 
> Hey,
> 
> I'm using the Mesos HTTP API for the first time. I am currently encountering 
> an issue where after a successful SUBSCRIBE call and receiving a SUBSCRIBED 
> and HEARTBEAT event, a subsequent TEARDOWN call fails with HTTP 400 with a 
> message of "The stream ID included in this request didn't match the stream ID 
> currently associated with framework ID".
> 
> Here is a detailed breakdown of what happens with logs:
> 
> A new framework sends an SUBSCRIBE call with the following body:
> 
> 
> framework_id {
>   value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
> }
> type: SUBSCRIBE
> subscribe {
>   framework_info {
> user: "user"
> name: "name"
> id {
>   value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
> }
>   }
> }
> 
> 
> It then receives a 200 OK response with the following headers:
> `{content-type=[application/x-protobuf], date=[Sat, 13 Aug 2016 02:42:48 
> GMT], transfer-encoding=[chunked], 
> mesos-stream-id=[71a0294f-e9c4-4efe-b237-fb120836aaf8]}`
> 
> Over this connection it receives a successful subscribed event:
> 
> type: SUBSCRIBED
> subscribed {
>   framework_id {
> value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
>   }
>   heartbeat_interval_seconds: 15.0
> }
> 
> 
> It also receives a single heart beat event.
> 
> Then it tries to send the following request:
> 
> Sending: framework_id {
>   value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
> }
> type: TEARDOWN
> 
> with the following headers:
> `{accept=[application/x-protobuf], accept-encoding=[gzip], 
> mesos-stream-id=[71a0294f-e9c4-4efe-b237-fb120836aaf8]}`
> 
> The response is a 400 with the body: `The stream ID included in this request 
> didn't match the stream ID currently associated with framework ID 
> '0dffbee9-a514-4ffa-87e1-2850dd4dcf00'`.
> 
> 
> The master logs contains:
> 
> I0813 02:42:48.376819 13934 http.cpp:381] HTTP POST for 
> /master/api/v1/scheduler from 192.168.33.1:60780 with 
> User-Agent='Google-HTTP-Java-Client/1.20.0 (gzip)'
> I0813 02:42:48.376998 13934 master.cpp:2146] Received subscription request 
> for HTTP framework 'name'
> I0813 02:42:48.377104 13934 master.cpp:2244] Subscribing framework 'name' 
> with checkpointing disabled and capabilities [  ]
> I0813 02:42:48.377378 13934 hierarchical.cpp:271] Added framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00
> I0813 02:42:49.475163 13929 http.cpp:381] HTTP POST for 
> /master/api/v1/scheduler from 192.168.33.1:60782 with 
> User-Agent='Google-HTTP-Java-Client/1.20.0 (gzip)'
> I0813 02:42:51.133513 13930 master.cpp:1284] Framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name) disconnected
> I0813 02:42:51.133597 13930 master.cpp:2725] Disconnecting framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.133618 13930 master.cpp:2749] Deactivating framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.133644 13930 master.cpp:1297] Giving framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name) 0ns to failover
> I0813 02:42:51.133692 13932 hierarchical.cpp:382] Deactivated framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00
> I0813 02:42:51.137265 13931 master.cpp:5561] Framework failover timeout, 
> removing framework 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.137339 13931 master.cpp:6296] Removing framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.137464 13931 hierarchical.cpp:333] Removed framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00
> 
> Note the immediate disconnection after the second POST is intentional.
> 
> This is with Mesos 1.0.0 on Ubuntu Trusty.
> 
> What can I do to debug this issue? The logs do not provide a lot of 
> information to act on. The stream id generated by mesos is not in the logs, 
> nor anything indicating that an HTTP 400 was sent.
> 
> --
> Zameer Manji

Re: Debugging Scheduler HTTP API Failures

[Dario Rexin]

HinZameer,

when you send the SUBSCRIBE to Mesos, the response will contain a header 
'Mesos-Stream-Id'. You have to send that header with every subsequent call you 
send to Mesos for that framework.

--
Dario

> On Aug 14, 2016, at 5:58 PM, Zameer Manji  wrote:
> 
> Hey,
> 
> I'm using the Mesos HTTP API for the first time. I am currently encountering 
> an issue where after a successful SUBSCRIBE call and receiving a SUBSCRIBED 
> and HEARTBEAT event, a subsequent TEARDOWN call fails with HTTP 400 with a 
> message of "The stream ID included in this request didn't match the stream ID 
> currently associated with framework ID".
> 
> Here is a detailed breakdown of what happens with logs:
> 
> A new framework sends an SUBSCRIBE call with the following body:
> 
> 
> framework_id {
>   value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
> }
> type: SUBSCRIBE
> subscribe {
>   framework_info {
> user: "user"
> name: "name"
> id {
>   value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
> }
>   }
> }
> 
> 
> It then receives a 200 OK response with the following headers:
> `{content-type=[application/x-protobuf], date=[Sat, 13 Aug 2016 02:42:48 
> GMT], transfer-encoding=[chunked], 
> mesos-stream-id=[71a0294f-e9c4-4efe-b237-fb120836aaf8]}`
> 
> Over this connection it receives a successful subscribed event:
> 
> type: SUBSCRIBED
> subscribed {
>   framework_id {
> value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
>   }
>   heartbeat_interval_seconds: 15.0
> }
> 
> 
> It also receives a single heart beat event.
> 
> Then it tries to send the following request:
> 
> Sending: framework_id {
>   value: "0dffbee9-a514-4ffa-87e1-2850dd4dcf00"
> }
> type: TEARDOWN
> 
> with the following headers:
> `{accept=[application/x-protobuf], accept-encoding=[gzip], 
> mesos-stream-id=[71a0294f-e9c4-4efe-b237-fb120836aaf8]}`
> 
> The response is a 400 with the body: `The stream ID included in this request 
> didn't match the stream ID currently associated with framework ID 
> '0dffbee9-a514-4ffa-87e1-2850dd4dcf00'`.
> 
> 
> The master logs contains:
> 
> I0813 02:42:48.376819 13934 http.cpp:381] HTTP POST for 
> /master/api/v1/scheduler from 192.168.33.1:60780 with 
> User-Agent='Google-HTTP-Java-Client/1.20.0 (gzip)'
> I0813 02:42:48.376998 13934 master.cpp:2146] Received subscription request 
> for HTTP framework 'name'
> I0813 02:42:48.377104 13934 master.cpp:2244] Subscribing framework 'name' 
> with checkpointing disabled and capabilities [  ]
> I0813 02:42:48.377378 13934 hierarchical.cpp:271] Added framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00
> I0813 02:42:49.475163 13929 http.cpp:381] HTTP POST for 
> /master/api/v1/scheduler from 192.168.33.1:60782 with 
> User-Agent='Google-HTTP-Java-Client/1.20.0 (gzip)'
> I0813 02:42:51.133513 13930 master.cpp:1284] Framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name) disconnected
> I0813 02:42:51.133597 13930 master.cpp:2725] Disconnecting framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.133618 13930 master.cpp:2749] Deactivating framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.133644 13930 master.cpp:1297] Giving framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name) 0ns to failover
> I0813 02:42:51.133692 13932 hierarchical.cpp:382] Deactivated framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00
> I0813 02:42:51.137265 13931 master.cpp:5561] Framework failover timeout, 
> removing framework 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.137339 13931 master.cpp:6296] Removing framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00 (name)
> I0813 02:42:51.137464 13931 hierarchical.cpp:333] Removed framework 
> 0dffbee9-a514-4ffa-87e1-2850dd4dcf00
> 
> Note the immediate disconnection after the second POST is intentional.
> 
> This is with Mesos 1.0.0 on Ubuntu Trusty.
> 
> What can I do to debug this issue? The logs do not provide a lot of 
> information to act on. The stream id generated by mesos is not in the logs, 
> nor anything indicating that an HTTP 400 was sent.
> 
> --
> Zameer Manji

Re: Mesos loses track of Docker containers

[Paul]

Thank you, Sivaram.

That would seem to be 2 "votes" for upgrading.

-Paul


> On Aug 13, 2016, at 11:47 PM, Sivaram Kannan  wrote:
> 
> 
> I don't remember the condition exactly, but I have faced similar issue in my 
> deployments and have been fixed when I moved to 0.26.0. Upgrade the marathon 
> to compatible version as well.
> 
>> On Wed, Aug 10, 2016 at 9:30 AM, Paul Bell  wrote:
>> Hi Jeff,
>> 
>> Thanks for your reply.
>> 
>> Yeahthat thought occurred to me late last night. But customer is 
>> sensitive to too much churn, so it wouldn't be my first choice. If I knew 
>> with certainty that such a problem existed in the versions they are running 
>> AND that more recent versions fixed it, then I'd do my best to compel the 
>> upgrade. 
>> 
>> Docker version is also old, 1.6.2.
>> 
>> -Paul
>> 
>>> On Wed, Aug 10, 2016 at 9:18 AM, Jeff Schroeder 
>>>  wrote:
>>> Have you considered upgrading Mesos and Marathon? Those are quite old 
>>> versions of both with some fairly glaring problems with the docker 
>>> containerizer if memory serves. Also what version of docker?
>>> 
>>> 
 On Wednesday, August 10, 2016, Paul Bell  wrote:
 Hello,
 
 One of our customers has twice encountered a problem wherein Mesos & 
 Marathon appear to lose track of the application containers that they 
 started. 
 
 Platform & version info:
 
 Ubuntu 14.04 (running under VMware)
 Mesos (master & agent): 0.23.0
 ZK: 3.4.5--1
 Marathon: 0.10.0
 
 The phenomena:
 
 When I log into either the Mesos or Marathon UIs I see no evidence of 
 *any* tasks, active or completed. Yet, in the Linux shell, a "docker ps" 
 command shows the containers up & running. 
 
 I've seen some confusing appearances before, but never this. For example, 
 I've seen what might be described as the reverse of the above phenomena. I 
 mean the case where a customer powers cycles the VM. In such a case you 
 typically see in Marathon's UI the (mere) appearance of the containers up 
 & running, but a "docker ps" command shows no containers running. As folks 
 on this list have explained to me, this is the result of "stale state" and 
 after 10 minutes (by default), Mesos figures out that the supposedly 
 active tasks aren't there and restarts them.
 
 But that's not the case here. I am hard-pressed to understand what 
 conditions/causes might lead to Mesos & Marathon becoming unaware of 
 containers that they started.
 
 I would be very grateful if someone could help me understand what's going 
 on here (so would our customer!).
 
 Thanks.
 
 -Paul
>>> 
>>> 
>>> -- 
>>> Text by Jeff, typos by iPhone
> 
> 
> 
> -- 
> ever tried. ever failed. no matter.
> try again. fail again. fail better.
> -- Samuel Beckett

Re: Using mesos' cfs limits on a docker container?

[Artem Harutyunyan]

Hi Mark,

Good to hear you figured it out. Can you please post curl errors that you
were observing and describe your image repository setup? I'd like to make
sure that we have instructions on how to mitigate those.

Artem.

On Sunday, August 14, 2016, Mark Hammons 
wrote:

> In specific, I wanted the process control capabilities of a mesos
> framework with custom schedulers and executors, but wanted to run my tasks
> in a framework definable environment (like running my tasks on a copy of
> Ubuntu 14 with certain libs installed). Using mixed-mode containerization
> worked with some fiddling, but it was painful in certain ways. The sandbox
> mounted in a mixed-mode container wasn't accessible from within the
> container thanks to selinux unless I ran the container in privileged mode
> and the cou limits per executor were no longer enforced unlike a mesos task
> with cfs isolation enabled. Further, setting up the default working
> directory and user was a pain.
>
> Unified mode (also called mesos containerizer for some reason) solves a
> lot of these issues, though using it with private image repositories was
> not as straightforward as the docker containerizer. I eventually had to use
> an image directory to get that working, cause curl kept throwing vague ssl
> errors(I'm fairly certain this is due to my private image repository not
> having https set up since it's a test environment).
>
> Once I get things set up and cleaned up I'll post a more involved guide on
> how to get this particular use case setup and running, especially a part on
> preparing your container image for use with mesos.
>
> Mark Edgar Hammons II - Research Engineer at BioEmergences
> 0603695656
>
> On 14 Aug 2016, at 18:11, Erik Weathers  > wrote:
>
> What was the problem and how did you overcome it?  (i.e. This would be a
> sad resolution to this thread for someone faced with this same problem in
> the future.)
>
> On Sunday, August 14, 2016, Mark Hammons  > wrote:
>
>> I finally got this working after fiddling with it all night. It works
>> great so far!
>>
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>>
>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>
>> If you're not against running Docker containers without the Docker
>> daemon, try using the Unified containerizer.
>> See the latter half of this document: http://mesos.apache.org/docume
>> ntation/latest/mesos-containerizer/
>> 
>>
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>
>>> Hi All,
>>>
>>>
>>>
>>> I was having a lot of success having mesos force sandboxed programs to
>>> work within cpu and memory constraints, but when I added docker into the
>>> mix, the cpu limitations go out the window (not sure about the memory
>>> limitations. Is there any way to mix these two methods of isolation? I'd
>>> like my executor/algorithm to run inside a docker container, but have that
>>> container's memory and cpu usage controlled by systemd/mesos.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>> --
>>>
>>> Mark Hammons - +33 06 03 69 56 56
>>>
>>> Research Engineer @ BioEmergences
>>> 
>>>
>>> Lab Phone: 01 69 82 34 19
>>>
>>
>>

Re: Using mesos' cfs limits on a docker container?

[Avinash Sridharan]

Hi Mark,
 That would be awesome (user facing documentation forthe
`UnifiedContainerizer`). We have bits and pieces of the unified
containerizer (and what it actually is), but would be great to land a more
comprehensive documentation into its motivation and usability. May be have
a separate `unified-containerized.md` and update the `
https://github.com/apache/mesos/blob/master/docs/mesos-containerizer.md`
with a link to the new 'unified-containerized.md'. Jie ??

Would also be very useful if you file some JIRAs related to the problems
you faced in setting up docker private registries to be used with the
unified-containerizer.

Thanks,
Avinash

On Sun, Aug 14, 2016 at 10:36 AM, Mark Hammons <
mark.hamm...@inaf.cnrs-gif.fr> wrote:

> In specific, I wanted the process control capabilities of a mesos
> framework with custom schedulers and executors, but wanted to run my tasks
> in a framework definable environment (like running my tasks on a copy of
> Ubuntu 14 with certain libs installed). Using mixed-mode containerization
> worked with some fiddling, but it was painful in certain ways. The sandbox
> mounted in a mixed-mode container wasn't accessible from within the
> container thanks to selinux unless I ran the container in privileged mode
> and the cou limits per executor were no longer enforced unlike a mesos task
> with cfs isolation enabled. Further, setting up the default working
> directory and user was a pain.
>
> Unified mode (also called mesos containerizer for some reason) solves a
> lot of these issues, though using it with private image repositories was
> not as straightforward as the docker containerizer. I eventually had to use
> an image directory to get that working, cause curl kept throwing vague ssl
> errors(I'm fairly certain this is due to my private image repository not
> having https set up since it's a test environment).
>
> Once I get things set up and cleaned up I'll post a more involved guide on
> how to get this particular use case setup and running, especially a part on
> preparing your container image for use with mesos.
>
> Mark Edgar Hammons II - Research Engineer at BioEmergences
> 0603695656
>
> On 14 Aug 2016, at 18:11, Erik Weathers  wrote:
>
> What was the problem and how did you overcome it?  (i.e. This would be a
> sad resolution to this thread for someone faced with this same problem in
> the future.)
>
> On Sunday, August 14, 2016, Mark Hammons 
> wrote:
>
>> I finally got this working after fiddling with it all night. It works
>> great so far!
>>
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>>
>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>
>> If you're not against running Docker containers without the Docker
>> daemon, try using the Unified containerizer.
>> See the latter half of this document: http://mesos.apache.org/docume
>> ntation/latest/mesos-containerizer/
>> 
>>
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>
>>> Hi All,
>>>
>>>
>>>
>>> I was having a lot of success having mesos force sandboxed programs to
>>> work within cpu and memory constraints, but when I added docker into the
>>> mix, the cpu limitations go out the window (not sure about the memory
>>> limitations. Is there any way to mix these two methods of isolation? I'd
>>> like my executor/algorithm to run inside a docker container, but have that
>>> container's memory and cpu usage controlled by systemd/mesos.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>> --
>>>
>>> Mark Hammons - +33 06 03 69 56 56
>>>
>>> Research Engineer @ BioEmergences
>>> 
>>>
>>> Lab Phone: 01 69 82 34 19
>>>
>>
>>


-- 
Avinash Sridharan, Mesosphere
+1 (323) 702 5245

Re: Using mesos' cfs limits on a docker container?

[Mark Hammons]

In specific, I wanted the process control capabilities of a mesos framework 
with custom schedulers and executors, but wanted to run my tasks in a framework 
definable environment (like running my tasks on a copy of Ubuntu 14 with 
certain libs installed). Using mixed-mode containerization worked with some 
fiddling, but it was painful in certain ways. The sandbox mounted in a 
mixed-mode container wasn't accessible from within the container thanks to 
selinux unless I ran the container in privileged mode and the cou limits per 
executor were no longer enforced unlike a mesos task with cfs isolation 
enabled. Further, setting up the default working directory and user was a pain.

Unified mode (also called mesos containerizer for some reason) solves a lot of 
these issues, though using it with private image repositories was not as 
straightforward as the docker containerizer. I eventually had to use an image 
directory to get that working, cause curl kept throwing vague ssl errors(I'm 
fairly certain this is due to my private image repository not having https set 
up since it's a test environment).

Once I get things set up and cleaned up I'll post a more involved guide on how 
to get this particular use case setup and running, especially a part on 
preparing your container image for use with mesos.

Mark Edgar Hammons II - Research Engineer at BioEmergences
0603695656

> On 14 Aug 2016, at 18:11, Erik Weathers  wrote:
> 
> What was the problem and how did you overcome it?  (i.e. This would be a sad 
> resolution to this thread for someone faced with this same problem in the 
> future.)
> 
>> On Sunday, August 14, 2016, Mark Hammons  
>> wrote:
>> I finally got this working after fiddling with it all night. It works great 
>> so far!
>> 
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>> 
>>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>> 
>>> If you're not against running Docker containers without the Docker daemon, 
>>> try using the Unified containerizer.  
>>> See the latter half of this document: 
>>> http://mesos.apache.org/documentation/latest/mesos-containerizer/
>>> 
 On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons 
  wrote:
 Hi All,
  
 I was having a lot of success having mesos force sandboxed programs to 
 work within cpu and memory constraints, but when I added docker into the 
 mix, the cpu limitations go out the window (not sure about the memory 
 limitations. Is there any way to mix these two methods of isolation? I'd 
 like my executor/algorithm to run inside a docker container, but have that 
 container's memory and cpu usage controlled by systemd/mesos.
  
 Thanks,
 Mark
 --
 Mark Hammons - +33 06 03 69 56 56
 Research Engineer @ BioEmergences
 Lab Phone: 01 69 82 34 19

Re: Using mesos' cfs limits on a docker container?

[haosdent]

Personally, I suggest to use the approach @Joseph and @Avinash mentioned.
Because zhitao and my patches require Docker >= 1.7.0 .

On Mon, Aug 15, 2016 at 1:27 AM, haosdent  wrote:

> Not sure if this related to https://issues.apache.org/
> jira/browse/MESOS-2154
> So far we have a quick workaround: specify the `cpu-period` and
> `cpu-quota` in the parameters field of `DockerInfo`. Then `Docker::run`
> would delegate this to the docker daemon.
>
> And recently zhitao and me work on the fix for this, we have some under
> reviewing patches. I think it should be fixed shortly once zhitao and my
> patches ready.
>
> On Mon, Aug 15, 2016 at 12:11 AM, Erik Weathers 
> wrote:
>
>> What was the problem and how did you overcome it?  (i.e. This would be a
>> sad resolution to this thread for someone faced with this same problem in
>> the future.)
>>
>>
>> On Sunday, August 14, 2016, Mark Hammons 
>> wrote:
>>
>>> I finally got this working after fiddling with it all night. It works
>>> great so far!
>>>
>>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>>> 0603695656
>>>
>>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>>
>>> If you're not against running Docker containers without the Docker
>>> daemon, try using the Unified containerizer.
>>> See the latter half of this document: http://mesos.apache.org/docume
>>> ntation/latest/mesos-containerizer/
>>> 
>>>
>>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>>
 Hi All,



 I was having a lot of success having mesos force sandboxed programs to
 work within cpu and memory constraints, but when I added docker into the
 mix, the cpu limitations go out the window (not sure about the memory
 limitations. Is there any way to mix these two methods of isolation? I'd
 like my executor/algorithm to run inside a docker container, but have that
 container's memory and cpu usage controlled by systemd/mesos.



 Thanks,

 Mark
 --

 Mark Hammons - +33 06 03 69 56 56

 Research Engineer @ BioEmergences
 

 Lab Phone: 01 69 82 34 19

>>>
>>>
>
>
> --
> Best Regards,
> Haosdent Huang
>



-- 
Best Regards,
Haosdent Huang

Re: Using mesos' cfs limits on a docker container?

[haosdent]

Not sure if this related to https://issues.apache.org/jira/browse/MESOS-2154
So far we have a quick workaround: specify the `cpu-period` and `cpu-quota`
in the parameters field of `DockerInfo`. Then `Docker::run` would delegate
this to the docker daemon.

And recently zhitao and me work on the fix for this, we have some under
reviewing patches. I think it should be fixed shortly once zhitao and my
patches ready.

On Mon, Aug 15, 2016 at 12:11 AM, Erik Weathers 
wrote:

> What was the problem and how did you overcome it?  (i.e. This would be a
> sad resolution to this thread for someone faced with this same problem in
> the future.)
>
>
> On Sunday, August 14, 2016, Mark Hammons 
> wrote:
>
>> I finally got this working after fiddling with it all night. It works
>> great so far!
>>
>> Mark Edgar Hammons II - Research Engineer at BioEmergences
>> 0603695656
>>
>> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
>>
>> If you're not against running Docker containers without the Docker
>> daemon, try using the Unified containerizer.
>> See the latter half of this document: http://mesos.apache.org/docume
>> ntation/latest/mesos-containerizer/
>> 
>>
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
>> mark.hamm...@inaf.cnrs-gif.fr> wrote:
>>
>>> Hi All,
>>>
>>>
>>>
>>> I was having a lot of success having mesos force sandboxed programs to
>>> work within cpu and memory constraints, but when I added docker into the
>>> mix, the cpu limitations go out the window (not sure about the memory
>>> limitations. Is there any way to mix these two methods of isolation? I'd
>>> like my executor/algorithm to run inside a docker container, but have that
>>> container's memory and cpu usage controlled by systemd/mesos.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>> --
>>>
>>> Mark Hammons - +33 06 03 69 56 56
>>>
>>> Research Engineer @ BioEmergences
>>> 
>>>
>>> Lab Phone: 01 69 82 34 19
>>>
>>
>>


-- 
Best Regards,
Haosdent Huang

Re: Using mesos' cfs limits on a docker container?

[Erik Weathers]

What was the problem and how did you overcome it?  (i.e. This would be a
sad resolution to this thread for someone faced with this same problem in
the future.)

On Sunday, August 14, 2016, Mark Hammons 
wrote:

> I finally got this working after fiddling with it all night. It works
> great so far!
>
> Mark Edgar Hammons II - Research Engineer at BioEmergences
> 0603695656
>
> On 14 Aug 2016, at 04:50, Joseph Wu  > wrote:
>
> If you're not against running Docker containers without the Docker daemon,
> try using the Unified containerizer.
> See the latter half of this document: http://mesos.apache.org/
> documentation/latest/mesos-containerizer/
> 
>
> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons <
> mark.hamm...@inaf.cnrs-gif.fr
> > wrote:
>
>> Hi All,
>>
>>
>>
>> I was having a lot of success having mesos force sandboxed programs to
>> work within cpu and memory constraints, but when I added docker into the
>> mix, the cpu limitations go out the window (not sure about the memory
>> limitations. Is there any way to mix these two methods of isolation? I'd
>> like my executor/algorithm to run inside a docker container, but have that
>> container's memory and cpu usage controlled by systemd/mesos.
>>
>>
>>
>> Thanks,
>>
>> Mark
>> --
>>
>> Mark Hammons - +33 06 03 69 56 56
>>
>> Research Engineer @ BioEmergences
>> 
>>
>> Lab Phone: 01 69 82 34 19
>>
>
>

Re: Using mesos' cfs limits on a docker container?

[Mark Hammons]

I finally got this working after fiddling with it all night. It works great so 
far!

Mark Edgar Hammons II - Research Engineer at BioEmergences
0603695656

> On 14 Aug 2016, at 04:50, Joseph Wu  wrote:
> 
> If you're not against running Docker containers without the Docker daemon, 
> try using the Unified containerizer.  
> See the latter half of this document: 
> http://mesos.apache.org/documentation/latest/mesos-containerizer/
> 
>> On Sat, Aug 13, 2016 at 7:02 PM, Mark Hammons 
>>  wrote:
>> Hi All,
>>  
>> I was having a lot of success having mesos force sandboxed programs to work 
>> within cpu and memory constraints, but when I added docker into the mix, the 
>> cpu limitations go out the window (not sure about the memory limitations. Is 
>> there any way to mix these two methods of isolation? I'd like my 
>> executor/algorithm to run inside a docker container, but have that 
>> container's memory and cpu usage controlled by systemd/mesos.
>>  
>> Thanks,
>> Mark
>> --
>> Mark Hammons - +33 06 03 69 56 56
>> Research Engineer @ BioEmergences
>> Lab Phone: 01 69 82 34 19
>