Totally agree that ultimately a single way of dealing with offers is
wanted. Quota does seem like a way forward although its lack of chunkiness
dilutes to what extent it guarantees progress.
On Wed, Jan 18, 2017 at 1:59 AM Povilas Versockas <[email protected]>
wrote:
> Thanks for information and ideas! I think labeled static reservation would
> help if framework could modify labels during runtime.
>
> Personally I think statically reserved resources should be offered only to
> frameworks but look like free (role:*) resources. Then framework developers
> could use labeled dynamic reservations and there won’t be any code
> differences handling static reservations and dynamic reservations. And this
> would make current frameworks like Cassandra or dcos-commons library
> support static reservation.
>
> Given the current situation it looks like quota may be a solution for me.
> The current idea is to create a custom resource on my mesos-agents and set
> a quota.
>
> Example:
>
> Set mesos-agent with flag:
>
> --resources
> cpus(*):8;mem(*):4096;disk(*):4096;ports(*):[31000-32000];mysql(*):1
>
>
> And then using operators API set quota:
>
> {
>
> "role": “role”,
>
> "guarantee": [
>
> {
>
> "name": "cpus",
>
> "type": "SCALAR",
>
> "scalar": { "value": 8 }
>
> },
>
> {
>
> "name": "mem",
>
> "type": "SCALAR",
>
> "scalar": { "value": 4096 }
>
> },
>
> …
>
> {
>
> "name": “mysql”,
>
> "type": "SCALAR",
>
> "scalar": { "value": 1 }
>
> }
>
> ]
>
> }
>
>
> This should make mesos-master save agent’s resources for my role and let
> me use dynamic reservations.
>
>
>
> On Wed, Jan 18, 2017 at 2:29 AM, Greg Mann <[email protected]> wrote:
>
> Thanks Gabriel, that makes sense. It sounds like labels on static
> reservations might be the most expedient path toward a solution to this
> problem, but that is not without its complications, as suggested in the
> related ticket which Neil filed a while back:
> https://issues.apache.org/jira/browse/MESOS-4476
>
> Povilas, also see this related ticket that Gabriel pointed me to:
> https://issues.apache.org/jira/browse/MESOS-6939
>
> It sounds like this is a real issue for stateful framework developers, so
> hopefully we will find some time soon to implement a solution. In the
> meantime, Povilas, I'm afraid to say I don't know exactly what solution to
> recommend. If anybody else in the community has some ideas, it would be
> great to hear them :)
>
> Cheers,
> Greg
>
>
> On Tue, Jan 17, 2017 at 2:52 PM, Gabriel Hartmann <[email protected]>
> wrote:
>
> @Greg: The reason people use static reservation is to enforce that
> particular resources (usually disks) can only be consumed by a particular
> framework. They also don't know when the stateful service is going to be
> installed necessarily so they don't want to race with other frameworks to
> consume those special resources. So static reservation is desirable.
> However, all stateful services also need more information about reserved
> resources than is natively provided by Mesos in the static reservation case
> (i.e. the labels he describes). `dcos-commons` does the same thing.
> Various work arounds exist, but none are able to provide resource
> allocation enforcement because only roles do that. An alternate resource
> allocation enforcement mechanism is needed. Usually this is the part where
> people start talking about quota.
>
> Neither option 1 nor option 2 provided a race proof way to get fully
> labeled reserved resources. It's been proposed in the past that it be
> allowed to add labels to statically reserved resources. That's kind of
> fine except now you have these things that can't really be UNRESERVEd but
> look exactly like dynamic resources which can...
>
> Quota w/ chunks as a step in the deployment of stateful services is very
> desirable in an adversarial environment. However if your'e in a
> cooperative environment (i.e. you're not in an adversarial relationship
> with other frameworks) if you had resources (particularly disk resources)
> with attributes on them you could have frameworks voluntarily choose not to
> consume resources not meant for them.
>
> e.g. Disk resource has attribute `CASSANDRA`. Ok, since I'm a Kafka
> framework I won't go use that disk.
>
> On Tue, Jan 17, 2017 at 11:24 AM Greg Mann <[email protected]> wrote:
>
> Hi Povilas,
> Another approach you could try is to use dynamic reservations only. You
> could either:
>
> 1. Alter your stateful framework to dynamically reserve the resources
> that it needs, or
> 2. Add a script to your cluster tooling that would make use of the
> operator endpoint for dynamic reservations [1]
> <http://mesos.apache.org/documentation/latest/reservation/> to
> dynamically reserve the stateful framework's resources when your cluster is
> initially provisioned. This would have a similar effect to static
> reservations, but would allow you to set labels
>
> Approach #1 makes sense to me; is there a reason that it's not feasible
> for your stateful framework to dynamically reserve its own resources? This
> is the typical workflow that I would recommend. I'm not too familiar with
> Aurora, so perhaps it's adding some complexity that I'm unaware of?
>
> Cheers,
> Greg
>
> [1] http://mesos.apache.org/documentation/latest/reservation/
>
>
> On Tue, Jan 17, 2017 at 12:28 AM, Povilas Versockas <[email protected]
> > wrote:
>
> Hey,
>
> Thanks for writing me back!
>
> Maybe there is some other method to solve this problem on statically
> reserved cluster? The solution could be making agent's resources appear as
> unreserved resources to only selected framework. I can see that mesos-agent
> has --acls flag, so maybe tinkering with this could help me. Of course it
> is possible to implement this in the framework scheduler, but this will add
> way more clunkiness to the code. It feels like this kind of resource
> management should be part of Mesos. Maybe I'm missing something?
>
>
>
> On Mon, Jan 16, 2017 at 4:58 PM, haosdent <[email protected]> wrote:
>
> Hi, @Povilas It is possible to dynamic reserve unreserved resources on
> those agents.
>
> On Fri, Jan 13, 2017 at 2:47 PM, Povilas Versockas <[email protected]>
> wrote:
>
> Hi,
>
> Maybe someone can help me with a problem I'm having. Short version of the
> question is:
> Is it possible to use dynamic reservation on statically reserved Mesos
> agents?
>
> The current situation is that we have Mesos cluster which runs many
> frameworks (aurora, spark, cassandra) and we are developing a custom
> framework for stateful tasks. Our framework manages stateful tasks for many
> users. Currently we statically reserved our hardware which has good disks
> only to be used by our framework (via --resources flag on Mesos Agents).
>
> The problem we are facing is that if one stateful task fails we would like
> to relaunch it on the same host with the same port, cpu, disk and memory.
> With dynamic reservations we would put a label with task id on a
> reservation and on failure would just simply reuse the reserved offer.
> On the other hand with statically reserved Mesos agents we cannot put any
> labels and so we cannot distinguish offers which should have been reserved
> for a task and a new offer.
> This leaves us in the situation that if one stateful task fails and there
> are new stateful tasks, the new tasks can be scheduled on failed task's
> Mesos agent, filling it up and taking it's port, cpu and memory.
>
>
> --
> Regards
> Povilas Versockas
>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
>
>
>
> --
> Pagarbiai
> Povilas Versockas
>
>
>
>
>
>
> --
> Pagarbiai
> Povilas Versockas
>
