Re: Soft limit for docker tasks

[Jie Yu]

You can try to use Mesos containerizer to launch your Docker containers
https://github.com/apache/mesos/blob/master/docs/containerizer.md
https://github.com/apache/mesos/blob/master/docs/container-image.md

, and build a custom memory isolator to set the soft memory limit for your
task.

- Jie

On Fri, Sep 23, 2016 at 3:38 PM, Jacopo Sabbatini 
wrote:

> Hi all,
>
> I’m exploring Mesos as a scheduler for batch jobs running in docker
> containers.
>
> When accepting offers and submitting tasks Mesos seems to run the
> container with the hard limit `—memory` and I was wondering if it possible
> to run them with soft memory limit instead, the `—memory-reservation`
> parameter.
>
> I’ll explain the use case. In my company we have data intensive jobs that
> could run for hours. These jobs could have memory spikes and we would like
> not to waste hours of computation just for a brief memory spike. Our
> current scheduler let jobs go over their allocated memory unless there is
> overall memory pressure in the machine, i.e. jobs don’t get killed unless
> they all go over their resource at the same time.
>
> What would be a viable solution to this problem in Mesos?
> Oversubscription? Custom executor?
>
> Thanks
>

Re: what is the status on this?

[Jay JN Guo]

Hi Kant,
 
What exactly are you planning to work on? Modularising interfaces or Consul module? Either way, we would welcome you to the community. You could go through the contributor guide here http://mesos.apache.org/community/ and file a PR on github to be added as contributor, see example here: https://github.com/apache/mesos/pull/167 . Also you need to ask some guy to add you to JIRA, then you'll be able to assign tasks.
 
cheers,
/J
 
- Original message -From: kant kodali To: user@mesos.apache.orgCc: Alex Rukletsov , user@mesos.apache.org, Joseph Wu , Dario Rexin , Avinash Sridharan , Jay JN Guo/China/IBM@IBMCNSubject: Re: what is the status on this?Date: Sat, Sep 24, 2016 5:13 AM  

@Joseph Wu: I am ready. Please add me on gchat kanth...@gmail.com and you guys can ping me anytime!  
 
 
On Wed, Sep 21, 2016 9:23 AM, Alex Rukletsov  a...@mesosphere.com wrote:

Kant,
 
we would love to walk new community members through the code! We understand how important it is to have a more experienced member of the community to help out with patches, hence we have "shepherds". Moreover, though technically possible, is not advised to start working without having agreement with your shepherd.
 
Joseph Wu is driving the effort, get in touch with him and I'm sure you'll figure out the plan!
 
On Tue, Sep 13, 2016 at 9:41 PM, kant kodali  wrote:

@Alex Rukletsov I am sorry I took some time to respond. I am very excited since the beginning to have an opportunity to work on this task but I wanted to take my time if I can really commit to the Task and looks I might be able to however I have not contributed to open source before and I would need some help from someone who can point me to the right parts of the code and basically help me navigate through the process and if that is feasible I will be happy to commit some time every week to work on this. please let me know if that works.
 
 
On Tue, Sep 6, 2016 11:59 AM, Dario Rexin  dre...@apple.com wrote:
 
Frameworks would use the redirect mechanism of the HTTP API and in case of unteachable nodes could do round robin on the list of master nodes. 
On Sep 6, 2016, at 11:52 AM, Joseph Wu  wrote: 
And for discovery of other nodes in the Paxos group.The work on modularizing/decoupling Zookeeper is a prerequisite for having the replicated log perform leader election itself.  <- That would merely be another implementation of the interface we will introduce in the process:https://issues.apache.org/jira/browse/MESOS-3574
 
On Tue, Sep 6, 2016 at 11:31 AM, Avinash Sridharan  wrote:
Also, I think, the replicated log itself uses Zookeeper for leader election.
 
On Tue, Sep 6, 2016 at 12:15 PM, Zameer Manji  wrote:
If we use the replicated log for leader election, how will frameworks detect the leading master? Right now the scheduler driver uses the MasterInfo in ZK to discover the leader and detect leadership changes.
 
On Mon, Sep 5, 2016 at 10:18 AM, Dario Rexin  wrote:
 
If we go and change this, why not simply remove any dependencies to external systems and simply use the replicated log for leader election?
On Sep 5, 2016, at 9:02 AM, Alex Rukletsov  wrote: 
Kant—
 
thanks a lot for the feedback! Are you interested in helping out with Consul module once Jay and Joseph are done with modularizing patches?
 
On Mon, Sep 5, 2016 at 8:50 AM, Jay JN Guo  wrote:
Patches are currently under review by @Joseph and can be found at the links provided by @haosdent.
 
I took a quick look at Consul key/value HTTP APIs and they look very similar to Etcd APIs. You could actually reuse our Etcd module implementation once we manage to push the module into Mesos community.
 
The only technical problem I could see for now is that Consul does not support `POST` with incremental key index. We may need to leverage `?cas=` operation in Consul to emulate the behaviour of joining a key group.
 
We could have a discussion on how to implement Consul HA module.
 
cheers,
/J
 
- Original message -From: haosdent To: user Cc: Jay JN Guo/China/IBM@IBMCNSubject: Re: what is the status on this?Date: Sun, Sep 4, 2016 6:10 PM 
Jay has some patches for de-couple Mesos with Zookeeper
 
https://issues.apache.org/jira/browse/MESOS-5828https://issues.apache.org/jira/browse/MESOS-5829
 
I think it should be possible to support consul by custom modules after jay's work done.
 
On Sun, Sep 4, 2016 at 6:02 PM, kant kodali  wrote: