Re: Cgroups v2 + Python 3 + Upstreaming X/Twitter Patches

[Benjamin Mahler]

Just an update here, in late January we finished upstreaming our internal
patches that were upstreamable. This amounted to 35 patches.

The cgroups v2 work is ongoing, we're hoping to be mostly code complete by
the end of this month.

On Fri, Jan 12, 2024 at 6:01 PM Benjamin Mahler  wrote:

> +user@
>
> On Fri, Jan 12, 2024 at 5:55 PM Benjamin Mahler 
> wrote:
>
>> As part of upgrading to CentOS 9 at X/Twitter, Shatil / Devin (cc'ed)
>> will be working on:
>>
>> * Upgrading to Python 3
>> * Cgroups v2 support
>>
>> We will attempt to upstream this work for the benefit of other users.
>>
>> In addition, we have several long-standing internal patches that should
>> have been upstreamed but weren't. We currently deploy from an internal
>> 1.9.x branch with these additional patches on top of OSS 1.9.x. To simplify
>> the above work since the code changes will be substantial (especially for
>> cgroups v2), we'll try to work off master (or 1.11.x) and upstream our
>> long-standing internal patches to minimize the delta between OSS and our
>> internal branch.
>>
>> Let me know if folks have any questions. IMO it would be good for users
>> to hold off on going into the attic so that we can land this work at least.
>>
>> Ben
>>
>

Re: Cgroups v2 + Python 3 + Upstreaming X/Twitter Patches

[Benjamin Mahler]

> >
> https://urldefense.com/v3/__https://sydneyscientific.org__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbSXruBk3U$
> > >
> > >> | consultancy
> > >> <
> >
> https://urldefense.com/v3/__https://offscale.io__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbSQf69uCY$
> > >
> > >> | open-source
> > >> <
> >
> https://urldefense.com/v3/__https://github.com/offscale__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbSR5dWm9A$
> > >
> > >> | LinkedIn
> > >> <
> >
> https://urldefense.com/v3/__https://linkedin.com/in/samuelmarks__;!!PWjfaQ!uCG12YNoPBcCb2z5v__gdjEEdxzUQsTntmDTSOVDMeG_sbq2zT58dS9IisTDUqIVlh18jPbS0q7dfzk$
> > >
> > >>
> > >>
> > >> On Fri, Jan 12, 2024 at 6:01 PM Benjamin Mahler 
> > >> wrote:
> > >>
> > >>> +user@
> > >>>
> > >>> On Fri, Jan 12, 2024 at 5:55 PM Benjamin Mahler 
> > >>> wrote:
> > >>>
> > >>> > As part of upgrading to CentOS 9 at X/Twitter, Shatil / Devin
> (cc'ed)
> > >>> will
> > >>> > be working on:
> > >>> >
> > >>> > * Upgrading to Python 3
> > >>> > * Cgroups v2 support
> > >>> >
> > >>> > We will attempt to upstream this work for the benefit of other
> users.
> > >>> >
> > >>> > In addition, we have several long-standing internal patches that
> > should
> > >>> > have been upstreamed but weren't. We currently deploy from an
> > internal
> > >>> > 1.9.x branch with these additional patches on top of OSS 1.9.x. To
> > >>> simplify
> > >>> > the above work since the code changes will be substantial
> (especially
> > >>> for
> > >>> > cgroups v2), we'll try to work off master (or 1.11.x) and upstream
> > our
> > >>> > long-standing internal patches to minimize the delta between OSS
> and
> > >>> our
> > >>> > internal branch.
> > >>> >
> > >>> > Let me know if folks have any questions. IMO it would be good for
> > >>> users to
> > >>> > hold off on going into the attic so that we can land this work at
> > >>> least.
> > >>> >
> > >>> > Ben
> > >>> >
> > >>>
> > >>
> >
>

Re: Cgroups v2 + Python 3 + Upstreaming X/Twitter Patches

[Benjamin Mahler]

+user@

On Fri, Jan 12, 2024 at 5:55 PM Benjamin Mahler  wrote:

> As part of upgrading to CentOS 9 at X/Twitter, Shatil / Devin (cc'ed) will
> be working on:
>
> * Upgrading to Python 3
> * Cgroups v2 support
>
> We will attempt to upstream this work for the benefit of other users.
>
> In addition, we have several long-standing internal patches that should
> have been upstreamed but weren't. We currently deploy from an internal
> 1.9.x branch with these additional patches on top of OSS 1.9.x. To simplify
> the above work since the code changes will be substantial (especially for
> cgroups v2), we'll try to work off master (or 1.11.x) and upstream our
> long-standing internal patches to minimize the delta between OSS and our
> internal branch.
>
> Let me know if folks have any questions. IMO it would be good for users to
> hold off on going into the attic so that we can land this work at least.
>
> Ben
>

Re: Next steps for Mesos

[Benjamin Mahler]

Also if you are still a user of mesos, please chime in.
Qian, it might be worth having a more explicit email asking users to chime
in as this email was tailored more for contributors.

Twitter is still using mesos heavily, we upgraded from a branch based off
of 1.2.x to 1.9.x in 2021, but haven't upgraded to 1.11.x yet. We do have a
lot of patches carried on our branch that have not been upstreamed. I would
like to upstream them to avoid relying on many custom patches and to
get closer to HEAD, but it will take time and quite a bit of work, and it's
not a priority at the moment.

On the contribution side, at this point if I were to continue contributing,
it would be on a volunteer basis, and I can't guarantee having enough time
to do so.

On Fri, Mar 17, 2023 at 9:57 PM Qian Zhang  wrote:

> Hi all,
>
> I'd like to restart the discussion around the future of the Mesos project.
> As you may already be aware, the Mesos community has been inactive for the
> last few years, there were only 3 contributors last year, that's obviously
> not enough to keep the project moving forward. I think we need at least 3
> active committers/PMC members and some active contributors to keep the
> project alive, or we may have to move it to attic
> .
>
> Call for action: If you are the current committer/PMC member and still
> have the capacity to maintain the project, or if you are willing to
> actively contribute to the project as a contributor, please reply to this
> email, thanks!
>
>
> Regards,
> Qian Zhang
>

Re: [VOTE] Move Apache Mesos to Attic

[Benjamin Mahler]

+1 (binding)

Thanks to all who contributed to the project.

On Mon, Apr 5, 2021 at 1:58 PM Vinod Kone  wrote:

> Hi folks,
>
> Based on the recent conversations
> <
> https://lists.apache.org/thread.html/raed89cc5ab78531c48f56aa1989e1e7eb05f89a6941e38e9bc8803ff%40%3Cuser.mesos.apache.org%3E
> >
> on our mailing list, it seems to me that the majority consensus among the
> existing PMC is to move the project to the attic <
> https://attic.apache.org/>
> and let the interested community members collaborate on a fork in Github.
>
> I would like to call a vote to dissolve the PMC and move the project to the
> attic.
>
> Please reply to this thread with your vote. Only binding votes from
> PMC/committers count towards the final tally but everyone in the community
> is encouraged to vote. See process here
> .
>
> Thanks,
>

Re: Feature requests for Mesos

[Benjamin Mahler]

I think the key issues have been brought up by Benjamin and Renan.

Just to add to Benjamin's comments above, achieving those key markers of
a healthy project requires serious corporate backing such that people are
being
employed to primarily work on Mesos. It takes a lot of work to keep the
project
running along and if people are doing that on the side or as a hobby it is
going
to be unsustainable.

In the initial phase of the project, there was a team dedicated to Mesos at
Twitter,
which is how the system was productionized and core features were developed.
The second phase of the project, in which Twitter was largely absent, was
driven
by Mesosphere (now called D2iQ) along with a few people making sporadic
contributions. With the shift in D2iQs priorities away from Mesos, the
project is
now without corporate backing.

Ultimately, I think with a project like Mesos those corporate backers tend
to be
vendors rather than users, because vendors can justify the investment as a
core
part of their business whereas users cannot. The vendors in this space have
consolidated around kubernetes.

An alternative model to vendor driven development would be to have a
foundation that allows users to fund development in the project, but this
model
requires donations which seems prone to persistent underfunding. (Reading
the wikipedia sections on freebsd and openbsd funding shows examples of
these alternative models).

So I see two options related to how the project is backed:

- Vendor and user companies are employing engineers to work on Mesos
because either they have products based on Mesos, or use it heavily.

- Users / Companies / Other bodies donate to a foundation which houses the
project and drives the development using these donations / resources.

The current model of Apache + maybe some folks stepping up to help I
don't think will work.

On Sun, Mar 7, 2021 at 6:39 PM Marc  wrote:

>
> * csi slrp unmanaged driver
>   (unless it is possible to work around enabling SYS_ADMIN in
> bounding_capabilities for all tasks)
> * csi serp
>
>
> http://mesos.apache.org/documentation/latest/csi/#limitations
> https://issues.apache.org/jira/browse/MESOS-8371
>
>

Re: Slow communications between components

[Benjamin Mahler]

Which version?

I'm not sure what you're observing but slower responses is usually due to
backlogging from expensive requests (like /state), however we made several
changes that have made it much less of a potential problem (see the blog
posts).

How much CPU is the master consuming? What kind of latency are you seeing
when you make a request to /health? What does "connections slowing down"
mean?

Assuming it's a cpu load problem, you can grab and share a flame graph per
the performance docs on the website, so we can see where the master is
spending time.

On Sat, Nov 7, 2020 at 10:17 PM Renan DelValle  wrote:

> Hi all,
>
> We've been noticing connections slowing down between our elected master
> and other components in the cluster the like the agents, frameworks,
> executor, etc.
>
>  From a high level view, it looks like the master is too busy doing
> other tasks to reply to messages and we've seen ACKs from our exectuor
> get delayed to the point where a new request has been sent by the retry
> mechanism.
>
> My initial suspicion is that we have some metric collectors that are
> hitting expensive endpoints (/metrics/snapshot, /master/state) too
> frequently and causing the master process to get bogged down.
>
> I was wondering if anyone had any experience with this and could confirm
> whether I'm on the right track with this.
>
> If this hunch is right, it would also be great if anyone could chime
> with a rough estimate of tasks and agents at which we should avoid
> hitting the Web UI directly since that generates a call to
> /metrics/snapshot at an interval.
>
> Thanks!
>
> -Renan
>
>

Re: Changing logging timestamp

[Benjamin Mahler]

Mesos uses Google's glog library for logging:
https://github.com/google/glog

I believe it just prints the local time. You can see that it's produced by
a call to gettimeofday(, NULL) and localtime_r (not gmtime_r):

https://github.com/google/glog/blob/v0.4.0/src/logging.cc#L1267
https://github.com/google/glog/blob/v0.4.0/src/utilities.cc#L221

There appears to be no way to configure it.

We find most users set up their servers to use UTC as the local time.

On Sun, Sep 20, 2020 at 11:06 AM Marc Roos  wrote:

>
> I have default remote syslog setup on centos all applications and server
> log the same timestamp (zone), except mesos and marathon tasks. I assume
> UTC times are send from them. How can I set this back to the 'hosts
> default'?
>
>
>
>
>

Re: No offers are being made -- how to debug Mesos?

[Benjamin Mahler]

Don't worry about that "Ignoring" message on the agent. When the framework
information is updated, the master broadcasts it to the agents, and in this
case the agent doesn't know about the framework since it has no tasks for
it, and so it ignores the updated information.

I can't quite tell from the log snippet you provided. Assuming this is the
only scheduler registered, it should receive offers for all the agents for
the scheduler's roles (in this case, should just be the '*' role).

Some reasons offers might not be sent:

-Framework doesn't have capability required to be offered the agent (e.g.
scheduler doesn't have GPU_RESOURCES when the agent has GPUs).
-Framework suppressed its role(s) (doesn't seem to be the case from the log
snippet)
-The role has insufficient quota (e.g. if you have set a quota limit for
that role, or if other roles have quota guarantees overcommitting the
cluster)
-The agent's resources are reserved to a role.

Can you show us the scheduler code? Can you give us complete logs, along
with results of the agent and master /state endpoints?


On Sat, Jun 6, 2020 at 8:00 AM Benjamin Wulff 
wrote:

> So with logging_level set to INFO (and master and slave restarted) I
> noticed in /var/log/mesos.INFO on the agent the following line:
>
> I0606 13:46:41.393455 206117 slave.cpp:4222] Ignoring info update for
> framework 2777de92-bc91-4e48-9960-bbab05694665- because it does not
> exist
>
> That is indeed the ID of the framework I’d like to run my task. In the web
> UI the framework is listed. So why is the agent saying that it doesn’t
> exist? What is the semantic of this message?
>
> On the master in /var/log/mesos-master.INFO the last relevant log lines
> (after that comes HTTP requests) are:
>
> I0606 13:50:11.710996 52025 http.cpp:1115] HTTP POST for
> /master/api/v1/scheduler from 172.30.0.8:41378 with
> User-Agent='python-requests/2.23.0'
> I0606 13:50:11.720903 52025 master.cpp:2670] Received subscription request
> for HTTP framework 'Go-Docker'
> I0606 13:50:11.721153 52025 master.cpp:2742] Subscribing framework
> 'Go-Docker' with checkpointing disabled and capabilities [  ]
> I0606 13:50:11.721993 52025 master.cpp:10847] Adding framework
> 2777de92-bc91-4e48-9960-bbab05694665- (Go-Docker) with roles {  }
> suppressed
> I0606 13:50:11.722084 52025 master.cpp:8300] Updating framework
> 2777de92-bc91-4e48-9960-bbab05694665- (Go-Docker) with roles {  }
> suppressed
> I0606 13:50:11.722514 52030 hierarchical.cpp:605] Added framework
> 2777de92-bc91-4e48-9960-bbab05694665-
> I0606 13:50:11.722573 52030 hierarchical.cpp:711] Deactivated framework
> 2777de92-bc91-4e48-9960-bbab05694665-
> I0606 13:50:11.722625 52030 hierarchical.cpp:1552] Suppressed offers for
> roles {  } of framework 2777de92-bc91-4e48-9960-bbab05694665-
> I0606 13:50:11.722657 52030 hierarchical.cpp:1592] Unsuppressed offers and
> cleared filters for roles {  } of framework
> 2777de92-bc91-4e48-9960-bbab05694665-
> I0606 13:50:11.722703 52030 hierarchical.cpp:681] Activated framework
> 2777de92-bc91-4e48-9960-bbab05694665-
>
> From my novice perspective it seems the framework is registered..
>
> Thanks,
> Ben
>
>
> > On 6. Jun 2020, at 13:40, Marc Roos  wrote:
> >
> >
> >
> >
> > You already put these on debug?
> >
> > [@ ]# cat /etc/mesos-master/logging_level
> > WARNING
> > [@ ]# cat /etc/mesos-slave/logging_level
> > WARNING
> >
> >
> >
> >
> > -Original Message-
> > From: Benjamin Wulff [mailto:benjamin.wulff...@ieee.org]
> > Sent: zaterdag 6 juni 2020 13:36
> > To: user@mesos.apache.org
> > Subject: No offers are being made -- how to debug Mesos?
> >
> > Hi all,
> >
> > I’m in the process of setting up my first Mesos cluster with 1x master
> > and 3x slaves on CentOS 8.
> >
> > So far set up Zookeepr and Mesos-master on the master and Mesos-slave on
> > one of the compute nodes. Mesos-master communicates with ZK and becomes
> > leader. Then I started memos-slave on the compute node and can see in
> > the log that it registers at the master with the correct resources
> > reported. The agent and its resources are also displayed in the web UI
> > of the master. So is the framework that I want to use.
> >
> > The crux is that no tasks I schedule in the framework are executed. And
> > I suppose this is because the framework never receives an offer. I can
> > see in the web UI that no offers are made and that all resources remain
> > idle.
> >
> > Now, I’m new to Mesos and I don’t really have an idea how to debug my
> > setup at this point.
> >
> > There is a page called ‘Debugging with the new CLI’ in the
> > documentation but it only explains how to configure  the CLI command.
> >
> > Any directions how to debug in my situation in general or on how to use
> > the CLI for debugging would be highly welcome! :)
> >
> > Thanks and best regards,
> > Ben
> >
> >
> >
>
>

Re: Subject: [VOTE] Release Apache Mesos 1.10.0 (rc1)

[Benjamin Mahler]

+1 (binding)

On Mon, May 18, 2020 at 4:36 PM Andrei Sekretenko 
wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.10.0.
>
> 1.10.0 includes the following major improvements:
>
> 
> * support for resource bursting (setting task resource limits separately
> from requests) on Linux
> * ability for an executor to communicate with an agent via Unix domain
> socket instead of TCP
> * ability for operators to modify reservations via the RESERVE_RESOURCES
> master API call
> * performance improvements of V1 operator API read-only calls bringing
> them on par with V0 HTTP endpoints
> * ability for a scheduler to expect that effects of calls sent through the
> same connection will not be reordered/interleaved by master
>
> NOTE: 1.10.0 includes a breaking change for custom authorizer modules.
> Now, `ObjectApprover`s may be stored by Mesos indefinitely and must be
> kept up-to-date by an authorizer throughout their lifetime.
> This allowed for several bugfixes and performance improvements.
>
> The CHANGELOG for the release is available at:
>
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.10.0-rc1
>
> 
>
> The candidate for Mesos 1.10.0 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.10.0-rc1/mesos-1.10.0.tar.gz
>
> The tag to be voted on is 1.10.0-rc1:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.10.0-rc1
>
> The SHA512 checksum of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.10.0-rc1/mesos-1.10.0.tar.gz.sha512
>
> The signature of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.10.0-rc1/mesos-1.10.0.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1259
>
> Please vote on releasing this package as Apache Mesos 1.10.0!
>
> The vote is open until Fri, May 21, 19:00 CEST  and passes if a majority
> of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.10.0
> [ ] -1 Do not release this package because ...
>
> Thanks,
> Andrei Sekretenko
>

Re: [VOTE] Release Apache Mesos 1.7.3 (rc1)

[Benjamin Mahler]

+1 (binding)

On Mon, May 4, 2020 at 1:48 PM Greg Mann  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.7.3.
>
> The CHANGELOG for the release is available at:
>
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.7.3-rc1
>
> 
>
> The candidate for Mesos 1.7.3 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.7.3-rc1/mesos-1.7.3.tar.gz
>
> The tag to be voted on is 1.7.3-rc1:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.7.3-rc1
>
> The SHA512 checksum of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.7.3-rc1/mesos-1.7.3.tar.gz.sha512
>
> The signature of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.7.3-rc1/mesos-1.7.3.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1258
>
> Please vote on releasing this package as Apache Mesos 1.7.3!
>
> The vote is open until Thu, May 7, 11:00 PDT 2020, and passes if a majority
> of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.7.3
> [ ] -1 Do not release this package because ...
>
> Thanks,
> Greg Mann
>

Re: Found no roles suitable for revive repetition.

[Benjamin Mahler]

See: https://mesosphere.github.io/marathon/support.html

I suppose the 'marathon' slack channel should be added to this page as well.

On Wed, Mar 18, 2020 at 2:51 PM Marc Roos  wrote:

>
> Hi Benjamin,
>
> Do you have a subscribe email address of this mailing list of marathon?
>
> Thanks,
> Marc
>
>
>
> -----Original Message-
> From: Benjamin Mahler [mailto:bmah...@apache.org]
> Sent: 18 March 2020 18:32
> To: user
> Subject: Re: Found no roles suitable for revive repetition.
>
> Hi Marc, can you contact the marathon mailing list or slack channel.
>
> Also, if there is a question here or some more context, please include
> that so they know what you need help with.
>
> On Wed, Mar 18, 2020 at 9:46 AM Marc Roos 
> wrote:
>
>
>
>
> Marathon is stuck on 'loading applications'
>
>
> Mar 18 14:43:48 m01 marathon: [2020-03-18 14:43:48,646] INFO
> Received
> fake heartbeat task-status update
> (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-30
> )
> Mar 18 14:43:53 m01 marathon: [2020-03-18 14:43:53,321] INFO
> Found
> no
> roles suitable for revive repetition.
> (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$
> Reviv
> eRepeaterLogic:marathon-akka.actor.default-dispatcher-9)
> Mar 18 14:43:58 m01 marathon: [2020-03-18 14:43:58,324] INFO
> Found
> no
> roles suitable for revive repetition.
> (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$
> Reviv
> eRepeaterLogic:marathon-akka.actor.default-dispatcher-6)
> Mar 18 14:44:03 m01 marathon: [2020-03-18 14:44:03,321] INFO
> Found
> no
> roles suitable for revive repetition. (mesosphe
>
>
>
>

Re: registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime.

[Benjamin Mahler]

Same here, please reach out to marathon support channels and include
additional context.

On Wed, Mar 18, 2020 at 12:27 PM Marc Roos  wrote:

>
> I am having these, has been reported already on Jira long time ago. How
> to fix these?
>
>
>
> der mesosphere.marathon.api.v2.PodsResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,785] WARN  A provider
> mesosphere.marathon.api.v2.AppsResource registered in SERVER runtime
> does not implement any provider interfaces applicable in the SERVER
> runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.AppsResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,787] WARN  A provider
> mesosphere.marathon.api.v2.DeploymentsResource registered in SERVER
> runtime does not implement any provider interfaces applicable in the
> SERVER runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.DeploymentsResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,789] WARN  A provider
> mesosphere.marathon.api.v2.TasksResource registered in SERVER runtime
> does not implement any provider interfaces applicable in the SERVER
> runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.TasksResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,792] WARN  A provider
> mesosphere.marathon.api.v2.QueueResource registered in SERVER runtime
> does not implement any provider interfaces applicable in the SERVER
> runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.QueueResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,798] WARN  A provider
> mesosphere.marathon.api.v2.InfoResource registered in SERVER runtime
> does not implement any provider interfaces applicable in the SERVER
> runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.InfoResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,800] WARN  A provider
> mesosphere.marathon.api.v2.LeaderResource registered in SERVER runtime
> does not implement any provider interfaces applicable in the SERVER
> runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.LeaderResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,803] WARN  A provider
> mesosphere.marathon.api.v2.PluginsResource registered in SERVER runtime
> does not implement any provider interfaces applicable in the SERVER
> runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.PluginsResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,805] WARN  A provider
> mesosphere.marathon.api.SystemResource registered in SERVER runtime does
> not implement any provider interfaces applicable in the SERVER runtime.
> Due to constraint configuration problems the provider
> mesosphere.marathon.api.SystemResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
> Mar 18 16:38:21 m01 marathon: [2020-03-18 16:38:21,805] WARN  A provider
> mesosphere.marathon.api.v2.GroupsResource registered in SERVER runtime
> does not implement any provider interfaces applicable in the SERVER
> runtime. Due to constraint configuration problems the provider
> mesosphere.marathon.api.v2.GroupsResource will be ignored.
> (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService
> STARTING)
>

Re: Found no roles suitable for revive repetition.

[Benjamin Mahler]

Hi Marc, can you contact the marathon mailing list or slack channel.

Also, if there is a question here or some more context, please include that
so they know what you need help with.

On Wed, Mar 18, 2020 at 9:46 AM Marc Roos  wrote:

>
>
> Marathon is stuck on 'loading applications'
>
>
> Mar 18 14:43:48 m01 marathon: [2020-03-18 14:43:48,646] INFO  Received
> fake heartbeat task-status update
> (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-30)
> Mar 18 14:43:53 m01 marathon: [2020-03-18 14:43:53,321] INFO  Found no
> roles suitable for revive repetition.
> (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$Reviv
> eRepeaterLogic:marathon-akka.actor.default-dispatcher-9)
> Mar 18 14:43:58 m01 marathon: [2020-03-18 14:43:58,324] INFO  Found no
> roles suitable for revive repetition.
> (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$Reviv
> eRepeaterLogic:marathon-akka.actor.default-dispatcher-6)
> Mar 18 14:44:03 m01 marathon: [2020-03-18 14:44:03,321] INFO  Found no
> roles suitable for revive repetition. (mesosphe
>

Re: Kill task, but not restarted

[Benjamin Mahler]

There's not enough information to understand the situation.

How did you kill the task? Did the task get correctly marked as killed? Did
the killed notification get correctly acknowledged?

On Sun, Feb 2, 2020 at 9:04 AM Marc Roos  wrote:

>
>
>
> Because the instance was not showing in the marathon gui. I have killed
> a task with kill -KILL, assuming it would restart, yet it did not.
>
> I think it has to do with these messages. Why do I have these even, when
> I can just ping them?
>
> W0202 14:46:51.215673 359364 process.cpp:1480] Failed to link to
> '192.168.122.253:35071', connect: Failed connect: connection closed
> W0202 14:46:51.217136 359364 process.cpp:1480] Failed to link to
> '192.168.122.95:41400', connect: Failed connect: connection closed
> W0202 14:46:51.217594 359364 process.cpp:1480] Failed to link to
> '192.168.122.94:41974', connect: Failed connect: connection closed
> W0202 14:46:51.218037 359364 process.cpp:1480] Failed to link to
> '192.168.122.13:33447', connect: Failed connect: connection closed
>
>
>
> [@mesos]# ping -c 2 192.168.122.95
> PING 192.168.122.95 (192.168.122.95) 56(84) bytes of data.
> 64 bytes from 192.168.122.95: icmp_seq=1 ttl=64 time=0.062 ms
> 64 bytes from 192.168.122.95: icmp_seq=2 ttl=64 time=0.051 ms
>
> [@mesos]# ping -c 2 192.168.122.94
> PING 192.168.122.94 (192.168.122.94) 56(84) bytes of data.
> 64 bytes from 192.168.122.94: icmp_seq=1 ttl=64 time=0.053 ms
> 64 bytes from 192.168.122.94: icmp_seq=2 ttl=64 time=0.045 ms
>
> [@mesos]# ping -c 2 192.168.122.13
> PING 192.168.122.13 (192.168.122.13) 56(84) bytes of data.
> 64 bytes from 192.168.122.13: icmp_seq=1 ttl=64 time=0.069 ms
> 64 bytes from 192.168.122.13: icmp_seq=2 ttl=64 time=0.051 ms
>
>

Welcome Andrei Sekretenko as a new committer and PMC member!

[Benjamin Mahler]

Please join me in welcoming Andrei Sekretenko as the newest committer and
PMC member!

Andrei has been active in the project for almost a year at this point and
has been a productive and collaborative member of the community.

He has helped out a lot with allocator work, both with code and
investigations of issues. He made improvements to multi-role framework
scalability (which includes the addition of the UPDATE_FRAMEWORK call), and
exposed metrics for per-role quota consumption.

He has also investigated, identified, and followed up on important bugs.
One such example is the message re-ordering issue he is currently working
on: https://issues.apache.org/jira/browse/MESOS-10023

Thanks for all your work so far Andrei, I'm looking forward to more of your
contributions in the project.

Ben

Re: Task Pinning

[Benjamin Mahler]

It's easier to do something custom for your own needs than to bring generic
support into the project.

For example, in kubernetes, as far as I can tell they offer two modes for
the agent: "static" (i.e. pinning for integer requests) and "none" (regular
shares / limit model).
https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/#static-policy

This means the user has to choose which model they want. If they choose the
"static" model, their utilization may go down given that some CPUs become
exclusive, which prevents other containers from "bursting" up and using
them when the owning container doesn't make use of them. We're about to add
this type of bursting on a per container basis:
https://issues.apache.org/jira/browse/MESOS-9916

The kubernetes approach is inline with some of the previous proposals for
Mesos, and I think it could be brought in first class to the project and
has no impact on the API. It will be up to operators to decide how they
want to run things and they may use attributes to mark which nodes have the
cpu pinning isolation on, to target scheduling there.

However, a complementary and potentially preferred approach that's been
proposed is to have it opt-in on a per container basis. When a task is
being launched it could state that it is latency sensitive and/or
explicitly specify the cpus it wants. Most tasks would not bother with
this, only those that have these special needs.

Ben

On Mon, Oct 21, 2019 at 10:41 AM Abel Souza  wrote:

> Hi,
>
> Does anyone know if pinning capabilities will ever be available to Mesos?
>
> Someone registered an issue at Jira
> (https://issues.apache.org/jira/browse/MESOS-5342), started an
> implementation (https://github.com/ct-clmsn/mesos-cpusets), but
> apparently it never went through mainline. I successfully compiled it in
> my testbed and loaded it into the Mesos master agent, but it keeps
> crashing the master during the submission process.
>
> So before moving on into potential fixes to these crashes, I would like
> to know if someone knows about possible updates to this specific
> capability in future Mesos releases.
>
> Thank you,
>
> /Abel
>
>

Re: large task scheduling on multi-framework cluster

[Benjamin Mahler]

Note that with the newest marathon that is capable of handling multiple
roles, you would not need to run a dedicated marathon instance.

On Tue, Oct 1, 2019 at 8:17 AM Grégoire Seux  wrote:

> Hello,
>
> I'm wondering how other mesos users deal with scheduling of large tasks
> (using all resources offered by most agents).
>
> On our cluster, we have various application launched mainly by marathon.
> Some of those applications have large instances (30 cpus) which use all
> resources from agents (most of our agents expose 30 cpus to mesos). Beyond
> these large applications (many instances, many resource per instance) we
> have a lot more applications whose instances are of various size (from 1 to
> 10 cpus).
>
> Our issue lies with scheduling, since marathon uses offers from mesos as
> they come and it creates fragmentation: most agents have small tasks
> running which prevents big tasks to be scheduled. In an ideal world, mesos
> (or marathon) would make sure some apps (let's say frameworks if mesos
> takes that responsibility) have guarantees on large offers. We also have
> non-marathon in-house frameworks which have similar needs to launch large
> tasks.
>
> Our current solution is to:
>
>- use a dedicated marathon instance (and a dedicated role) for those
>big applications
>- dedicate agents to this role
>
> Of course, this require extra work since our mesos clusters are now
> sharded (it creates additional toil in term of maintenance & capacity
> planning).
> Our thinking is that mesos allocator might be improved to distribute
> offers with a better heuristic than currently (offers are randomly sorted).
> A bit similar to what was suggested on
> http://mail-archives.apache.org/mod_mbox/mesos-user/201906.mbox/%3cCAHReGaiY0nJ0AevMvKbxAZsy2Xc=jmtszcucdxryzbvwkvv...@mail.gmail.com%3e,
> we could imagine to sort offers (offers from most used slaves first).
>
> So I'm curious on how other users handle this kind of needs!
>
> Regards,
>
> -- ​
> Grégoire Seux
>

Re: reservations from terminated frameworks

[Benjamin Mahler]

Hi Hendrik, currently reservations are tied to a role, not framework. In
this case, it's a static reservation which means you need to update the
agent configuration and restart it destructively (we don't currently
support a non-destructive non-additive agent resources change). If it was a
dynamic reservation, you could send an unreserve operation to the master
instead.

On Fri, Sep 27, 2019 at 11:50 AM Hendrik Haddorp 
wrote:

> Hi,
>
> I have a custom framework running on Mesos 1.8 using static reservation
> so that resources are reserved for a specific role. For some time I had
> two instances of my framework running, each using its own principal.
> Each instance reserved resources and created persistent volumes. Then I
> terminated one of the frameworks. But in the resource offers I get for
> the still running instance I still see reservations done by the
> principal of the now terminated framework. Why is that and how can get
> rid of those resources?
>
> thanks,
> Hendrik
>

Re: Attach Shared Volume to new tasks

[Benjamin Mahler]

Can you show the full resource information from the offer?

On Tue, Sep 10, 2019 at 6:50 AM Harold Molina-Bulla 
wrote:

> Hi everybody,
>
> We are implementing a Scheduler for Mesos in python, and we need to attach
> a preconfigured shared volume to a new Task. The Shared volume is now
> offered by the agents, we can get all information from the offer, but we do
> not know how to attach this volume to a new created task.
>
> We are trying with the follow code:
>
>   for off_volume in offers_disks :
> volume = task.resources.add()
> volume.name = "disk"
> volume.type = mesos_pb2.Value.SCALAR
> volume.scalar.value = off_volume.scalar.value
> volume.disk.persistence.id =
> off_volume.disk.persistence.id
> volume.disk.volume.container_path =
> off_volume.disk.volume.container_path
> volume.disk.volume.mode = off_volume.disk.volume.mode
>
> Where offer_disks are the list of offered shared volumes (as we said
> previously, this volume was created before to try to launch the Tasks, with
> the field “shared” defined); task is the new task created using
> mesos_pb2.TaskInfo().
>
> When  we try to create this new task, the system says:
>
> "Task uses invalid resources: Invalid DiskInfo: Persistent volumes cannot
> be created from unreserved resources."
>
> How can I attach this shared volume to my new Task?
>
> Thanks in advance.
>
> H.
>
>
> --
> “En una época de mentira universal, decir la verdad constituye un acto
> revolucionario”
> George Orwell (1984)
>
> --
> Recuerda: PRISM te está vigilando!!! X)
>
> --
> Harold Molina-Bulla
> h.mol...@tsc.uc3m.es
> Clave GnuPG: 189D5144
>
>
>
>
>
>
>
>

Re: [VOTE] Release Apache Mesos 1.9.0 (rc1)

[Benjamin Mahler]

> We upgraded the version of the bundled boost very late in the release
cycle

Did we? We still bundle boost 1.65.0, just like we did during 1.8.x. We
just adjusted our special stripped bundle to include additional headers.

On Tue, Aug 27, 2019 at 1:39 PM Vinod Kone  wrote:

> -1
>
> We upgraded the version of the bundled boost very late in the release cycle
> which doesn't give downstream customers (who also depend on boost) enough
> time to vet any compatibility/perf/other issues. I propose we revert the
> boost upgrade (and the corresponding code changes depending on the upgrade)
> in 1.9.x branch but keep it in the master branch.
>
> On Tue, Aug 27, 2019 at 4:18 AM Qian Zhang  wrote:
>
> > Hi all,
> >
> > Please vote on releasing the following candidate as Apache Mesos 1.9.0.
> >
> >
> > 1.9.0 includes the following:
> >
> >
> 
> > * Agent draining
> > * Support configurable /dev/shm and IPC namespace.
> > * Containerizer debug endpoint.
> > * Add `no-new-privileges` isolator.
> > * Client side SSL certificate verification in Libprocess.
> >
> > The CHANGELOG for the release is available at:
> >
> >
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.9.0-rc1
> >
> >
> 
> >
> > The candidate for Mesos 1.9.0 release is available at:
> >
> https://dist.apache.org/repos/dist/dev/mesos/1.9.0-rc1/mesos-1.9.0.tar.gz
> >
> > The tag to be voted on is 1.9.0-rc1:
> > https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.9.0-rc1
> >
> > The SHA512 checksum of the tarball can be found at:
> >
> >
> https://dist.apache.org/repos/dist/dev/mesos/1.9.0-rc1/mesos-1.9.0.tar.gz.sha512
> >
> > The signature of the tarball can be found at:
> >
> >
> https://dist.apache.org/repos/dist/dev/mesos/1.9.0-rc1/mesos-1.9.0.tar.gz.asc
> >
> > The PGP key used to sign the release is here:
> > https://dist.apache.org/repos/dist/release/mesos/KEYS
> >
> > The JAR is in a staging repository here:
> > https://repository.apache.org/content/repositories/orgapachemesos-1255
> >
> > Please vote on releasing this package as Apache Mesos 1.9.0!
> >
> > The vote is open until Friday, April 30 and passes if a majority of at
> > least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Mesos 1.9.0
> > [ ] -1 Do not release this package because ...
> >
> >
> > Thanks,
> > Qian and Gilbert
> >
>

[Performance / Resource Management WG] August Update

[Benjamin Mahler]

Can't make today's meeting, so sending out some notes:

On the performance front:

- Long Fei reported a slow master, and perf data indicates a lot of time is
spent handling executor churn, this can be easily improved:
https://issues.apache.org/jira/browse/MESOS-9948

On the resource management front:

- Work on quota limits is wrapping up. The functionality is ready for
release, but some pieces are missing (e.g. there is no quota consumption
metric, it's only exposed in the /roles endpoint for now):
http://mesos.apache.org/documentation/latest/quota/

- The allocation cycle times in the LargeAndSmallQuota benchmark appears to
be ~2-3x worse from 1.8.1 as a result of implementing quota limits. The
non-quota cycle times appear similar. We will be attempting to bring this
back down closer to 1.8.1 but it's tight with the release planning to be
cut this week.

If folks have anything else to mention, or any comments or questions,
please chime in.

Ben

Re: Mesos 1.9.0 release

[Benjamin Mahler]

Thanks for taking this on Qian!

I seem to be unable to view the dashboard.
Also, when are we aiming to make the cut?

On Tue, Aug 13, 2019 at 10:58 PM Qian Zhang  wrote:

> Folks,
>
> It is time for Mesos 1.9.0 release and I am the release manager. Here is
> the dashboard:
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12334125
>
> Please start to wrap up patches if you are contributing or shepherding any
> issues. If you expect any particular JIRA for this new release, please set
> "Target Version" as "1.9.0" and mark it as "Blocker" priority.
>
>
> Regards,
> Qian Zhang
>

Re: Should mesos 1.8 (and marathon 1.8) drain/migrate tasks or not?

[Benjamin Mahler]

(had to join the marathon-framework group to post to it, re-sending)

On Tue, Aug 13, 2019 at 1:26 PM Benjamin Mahler  wrote:

> > I know DRAIN_AGENT is only for mesos 1.9. But what use it to post a
> > maintenance schedule, see the node being marked as draining, and nothing
> > happens with the tasks?
>
> The maintenance schedules require that schedulers implement support for
> them. Nothing happens if the scheduler does not have support for the
> maintenance schedules.
> The DRAIN_AGENT in Mesos 1.9 does not require scheduler support (Mesos
> will kill the tasks). As a result, it is up to the operator issuing the
> DRAIN_AGENT request to not violate higher level SLAs.
> See this thread for context:
> https://lists.apache.org/thread.html/c0414897debdea2faffbffc81fa36c435e109e7ff80c72c544b4d135@%3Cdev.mesos.apache.org%3E
>
> As to what support marathon has for maintenance schedules, the ticket you
> pointed to what closed in favor of
> https://jira.mesosphere.com/browse/DCOS-54915.
> This new ticket is only about DRAIN_AGENT as far as I can tell. If so,
> then there is a gap in the marathon ticketing for supporting SLA aware
> maintenance.
>
> On Thu, Aug 8, 2019 at 4:22 PM Marc Roos  wrote:
>
>>
>> I don’t get from this page
>> http://mesos.apache.org/documentation/latest/maintenance/ if mesos
>> should be 'moving' tasks to another node when it is marked as draining.
>> I know DRAIN_AGENT is only for mesos 1.9. But what use it to post a
>> maintenance schedule, see the node being marked as draining, and nothing
>> happens with the tasks?
>>
>>
>> On the marathon page the say "draining is not yet implemented", yet they
>> refer to an issue that has been resolved.
>> https://mesosphere.github.io/marathon/docs/maintenance-mode.html
>>
>>
>> On stackoverflow there is the same question, and again referencing issue
>> that have been resolved.
>>
>> https://stackoverflow.com/questions/37194123/marathon-tasks-not-migrating-off-mesos-node-goes-into-draining-mode
>> https://jira.mesosphere.com/browse/MARATHON-3216
>> https://phabricator.mesosphere.com/D1069
>>
>>
>>
>> -Original Message-
>> From: Vinod Kone [mailto:vinodk...@apache.org]
>> Sent: donderdag 8 augustus 2019 0:35
>> To: user
>> Subject: Re: Draining: Failed to validate master::Call: Expecting 'type'
>> to be present
>>
>> Please read the "maintenace primitives" section in this doc
>> http://mesos.apache.org/documentation/latest/maintenance/ and let us
>> know if you have unanswered questions.
>>
>> On Wed, Aug 7, 2019 at 4:59 PM Marc Roos 
>> wrote:
>>
>>
>>
>>  I seem to be able to add a maintenance schedule, and get also a
>> report
>> on '{"down_machines":[{"hostname":"m02.local"}]}' but I do not
>> see
>> tasks
>> migrate to other hosts. Or is this not the purpose of maintenance
>> mode
>> in 1.8? Just to make sure no new tasks will be launched on hosts
>> scheduled for maintenance?
>>
>>
>>
>> -Original Message-
>> From: Chun-Hung Hsiao [mailto:chhs...@apache.org]
>> Sent: woensdag 7 augustus 2019 22:59
>> To: user
>> Subject: Re: Draining: Failed to validate master::Call: Expecting
>> 'type'
>> to be present
>>
>> Hi Marc.
>>
>> Agent draining is a Mesos 1.9 feature and is only available on
>> the
>> current Mesos master branch.
>> Please see https://issues.apache.org/jira/browse/MESOS-9814.
>>
>> Best,
>> Chun-Hung
>>
>> On Wed, Aug 7, 2019 at 1:35 PM Marc Roos <
>> m.r...@f1-outsourcing.eu>
>>
>> wrote:
>>
>>
>>
>> Should this be working in mesos 1.8?
>>
>> [@m01 ~]# curl --user test:x -X POST \
>> >   https://m01.local:5050/api/v1 \
>> >   --cacert /etc/pki/ca-trust/source/ca.crt \
>> >   -H 'Accept: application/json' \
>> >   -H 'content-type: application/json' -d '{
>> >   "type": "DRAIN_AGENT",
>> >   "drain_agent": {"agent_id": {
>> > "value":"53336fcb-7756-4673-b9c7-177e04f34c3b-S1"
>> >   }}}'
>>
>> Failed to validate master::Call: Expecting 'type' to be
>> present
>>
>>
>>
>>
>>
>>
>>
>>

Re: Should mesos 1.8 (and marathon 1.8) drain/migrate tasks or not?

[Benjamin Mahler]

> I know DRAIN_AGENT is only for mesos 1.9. But what use it to post a
> maintenance schedule, see the node being marked as draining, and nothing
> happens with the tasks?

The maintenance schedules require that schedulers implement support for
them. Nothing happens if the scheduler does not have support for the
maintenance schedules.
The DRAIN_AGENT in Mesos 1.9 does not require scheduler support (Mesos will
kill the tasks). As a result, it is up to the operator issuing the
DRAIN_AGENT request to not violate higher level SLAs.
See this thread for context:
https://lists.apache.org/thread.html/c0414897debdea2faffbffc81fa36c435e109e7ff80c72c544b4d135@%3Cdev.mesos.apache.org%3E

As to what support marathon has for maintenance schedules, the ticket you
pointed to what closed in favor of
https://jira.mesosphere.com/browse/DCOS-54915.
This new ticket is only about DRAIN_AGENT as far as I can tell. If so, then
there is a gap in the marathon ticketing for supporting SLA aware
maintenance.

On Thu, Aug 8, 2019 at 4:22 PM Marc Roos  wrote:

>
> I don’t get from this page
> http://mesos.apache.org/documentation/latest/maintenance/ if mesos
> should be 'moving' tasks to another node when it is marked as draining.
> I know DRAIN_AGENT is only for mesos 1.9. But what use it to post a
> maintenance schedule, see the node being marked as draining, and nothing
> happens with the tasks?
>
>
> On the marathon page the say "draining is not yet implemented", yet they
> refer to an issue that has been resolved.
> https://mesosphere.github.io/marathon/docs/maintenance-mode.html
>
>
> On stackoverflow there is the same question, and again referencing issue
> that have been resolved.
>
> https://stackoverflow.com/questions/37194123/marathon-tasks-not-migrating-off-mesos-node-goes-into-draining-mode
> https://jira.mesosphere.com/browse/MARATHON-3216
> https://phabricator.mesosphere.com/D1069
>
>
>
> -Original Message-
> From: Vinod Kone [mailto:vinodk...@apache.org]
> Sent: donderdag 8 augustus 2019 0:35
> To: user
> Subject: Re: Draining: Failed to validate master::Call: Expecting 'type'
> to be present
>
> Please read the "maintenace primitives" section in this doc
> http://mesos.apache.org/documentation/latest/maintenance/ and let us
> know if you have unanswered questions.
>
> On Wed, Aug 7, 2019 at 4:59 PM Marc Roos 
> wrote:
>
>
>
>  I seem to be able to add a maintenance schedule, and get also a
> report
> on '{"down_machines":[{"hostname":"m02.local"}]}' but I do not see
> tasks
> migrate to other hosts. Or is this not the purpose of maintenance
> mode
> in 1.8? Just to make sure no new tasks will be launched on hosts
> scheduled for maintenance?
>
>
>
> -Original Message-
> From: Chun-Hung Hsiao [mailto:chhs...@apache.org]
> Sent: woensdag 7 augustus 2019 22:59
> To: user
> Subject: Re: Draining: Failed to validate master::Call: Expecting
> 'type'
> to be present
>
> Hi Marc.
>
> Agent draining is a Mesos 1.9 feature and is only available on the
> current Mesos master branch.
> Please see https://issues.apache.org/jira/browse/MESOS-9814.
>
> Best,
> Chun-Hung
>
> On Wed, Aug 7, 2019 at 1:35 PM Marc Roos 
>
>
> wrote:
>
>
>
> Should this be working in mesos 1.8?
>
> [@m01 ~]# curl --user test:x -X POST \
> >   https://m01.local:5050/api/v1 \
> >   --cacert /etc/pki/ca-trust/source/ca.crt \
> >   -H 'Accept: application/json' \
> >   -H 'content-type: application/json' -d '{
> >   "type": "DRAIN_AGENT",
> >   "drain_agent": {"agent_id": {
> > "value":"53336fcb-7756-4673-b9c7-177e04f34c3b-S1"
> >   }}}'
>
> Failed to validate master::Call: Expecting 'type' to be
> present
>
>
>
>
>
>
>
>

Re: Mesos-dns srv weight

[Benjamin Mahler]

Please seek support through the mesos DNS channels:
https://github.com/mesosphere/mesos-dns#contact

On Fri, Jul 26, 2019 at 9:50 AM Marc Roos  wrote:

>
> Is it possible to configure a task with srv record weight?
>
>
> [@ mesos-cni]# dig +short @192.168.10.151 _webchat._tcp.marathon.mesos
> SRV
> 0 0 8181 webchat-jygxm-s1.marathon.mesos.
> 0 0 8181 webchat-h3q8o-s1.marathon.mesos.
>
>
> {
>   "id": "/webchat",
>   "user": "nobody",
>   "cpus": 1,
>   "mem": 256,
>   "disk": 0,
>   "instances": 1,
>   "acceptedResourceRoles": ["*"],
>   "upgradeStrategy": { "minimumHealthCapacity": 0,
> "maximumOverCapacity": 0 },
>   "constraints": [["hostname","CLUSTER","c04.local"]],
>   "backoffSeconds": 10,
>   "env": { "TOMCAT_HTTP_PORT": "$PORT0" },
>   "networks": [
> { "mode": "container", "name": "cni-apps" }
>   ],
>   "container": {
> "type": "MESOS",
> "docker": {
> "image": "webchat:4.2.0",
> "credential": null,
> "forcePullImage": true
> },
> "portMappings": [
> { "containerPort": 8181, "protocol": "tcp", "name": "http"}
> ]
>   },
>   "labels": {
> "HAPROXY_GROUP": "webchat",
> "HAPROXY_0_BACKEND_HTTP_HEALTHCHECK_OPTIONS": "n httpchk HEAD
> /webchat/ HTTP/1.1\r\nHost:localhost"
>   },
>   "healthChecks": [
> {
>   "gracePeriodSeconds": 300,
>   "intervalSeconds": 30,
>   "timeoutSeconds": 5,
>   "maxConsecutiveFailures": 3,
>   "portIndex": 0,
>   "path": "/webchat/",
>   "protocol": "MESOS_HTTP"
>
> }
>   ]
> }
>
>
>

[Performance / Resource Management WG] July Update

[Benjamin Mahler]

On the resource management front, Meng Zhu, Andrei Sekretenko, and myself
have been working on quota limits and enhancing multi-role framework
support:

- A memory leak in the allocator was fixed: MESOS-9852

- Support for quota limits work is well underway, and at this point the
major pieces are there and within the next few days it can be tried out.
The plan here is to try to push users towards using only quota limits, as
future support for optimistic offers will likely not support quota
guarantees.

- The /roles endpoint was fixed to expose all cases of "known" roles:
MESOS-9888 , MESOS-9890
.

- The /roles endpoint and roles table in the webui has been updated to
display quota consumption, as well as breakdowns of allocated, offered, and
reserved resources, see gif here:
https://reviews.apache.org/r/71059/file/1894/

- Several bugs were fixed for MULTI_ROLE framework support: MESOS-9856
, MESOS-9870


- The v0 scheduler driver and java/python bindings are being updated to
support multiple roles.


On the performance front:

- MESOS-9755 : William
Mahler and I looked into updating protobuf to 3.7.x from our existing 3.5.x
in order to attempt to use protobuf arenas in the master API and we noticed
a performance regression in the v0 /state endpoint. After looking into it,
it appears to be a performance regression in the protobuf reflection code
that we use to convert from our in-memory protobuf to json. No issue is
filed yet with the protobuf community, but it's worth also trying out
protobuf's built in json conversion code to see how that compares (see
MESOS-9896 ).


Feel free to reply with any questions or additional comments!

Ben

Re: Design doc: Agent draining and deprecation of maintenance primitives

[Benjamin Mahler]

> With the new proposal, it's going to be as difficult as before to have
SLA-aware maintenances because it will need cooperation from the frameworks
anyway and we know this is rarely a priority for them. We will also lose
the ability to signal future maintenance in order to optimize allocations.

Personally, I think right now we should solve the basic need of draining a
node. The plan to add SLA-awareness into draining was to introduce a
capability that schedulers opt into that enables them to (1) take control
over the killing of tasks when an agent is put into the draining state and
(2) still get offers when an agent is the draining state in case the
scheduler needs to restart a task that *must* run. This allows an SLA-aware
scheduler to avoid killing during a drain if its task(s) will have SLAs
violated.

Perhaps this functionality can live alongside the maintenance schedule
information we currently support, without being coupled together. As far as
I'm aware that's something we hadn't considered (we considered integrating
into the maintenance schedules or replacing them).

> For example I had this idea to improve the allocator (or write a custom
one) that would offer resources from agents with no maintenance planned in
priority, and then sort agents by maintenance date in decremasing order.

Right now there is no meaning to the order of offers. Adding some meaning
to the ordering of offers quickly becomes an issue for us as soon as there
are multiple criteria that need to be evaluated. For example, if you want
to incorporate maintenance, load spreading, fault domain spreading, etc
across machines, it becomes less clear how offers should be ordered. One
could try to build some scoring model in mesos for ordering, but it will be
woefully inadequate since Mesos does not know anything about the pending
workloads: it's ultimately the schedulers that are best positioned to make
these decisions. This is why we are going to move towards an "optimistic
concurrency" model where schedulers can choose what they want and Mesos
enforces constraints (e.g. quota limits), thereby eliminating the
multi-scheduler scalability issues of the current offer model.

And as somewhat of an aside, the lack of built-in scheduling has been bad
for the Mesos ecosystem. The vast majority of users just need to schedule:
services, jobs and cron jobs. These have a pretty standard look and feel
(including the SLA aspect of them!). Many of the existing schedulers could
be thinner "orchestrators" that know when to submit something to be
scheduled by a common scheduler, rather than reimplementing all of the
typical scheduling primitives (constraints, SLA awareness, dealing with the
low level mesos scheduling API). My point here is that we ask too much of
frameworks and it hurts users. I would love to see scheduling become more
standardized and built into Mesos.

On Thu, Jun 6, 2019 at 10:52 AM Greg Mann  wrote:

> Maxime,
> Thanks for the feedback, it's much appreciated. I agree that it would be
> possible to evolve the existing primitives to accomplish something similar
> to the proposal. That is one option that was considered before writing the
> design doc, but after some discussion, I thought that it seems more
> appropriate to start over with a simpler model that accomplishes what we
> perceive to be the predominant use case: the automated draining of agent
> nodes, without the concept of a maintenance window or designated
> maintenance time in the future. However, perhaps this perception is
> incorrect?
>
> Using maintenance metadata to alter the sorting order in the allocator is
> an interesting idea; currently, the allocator does not have access to
> information about maintenance, but it's conceivable that we could extend
> the allocator interface to accommodate this. While the currently-proposed
> design would not allow this, it would allow operators to deactivate nodes,
> which is an extreme version of this, since deactivated agents would never
> have their resources offered to frameworks. This provides a blunt mechanism
> to prevent scheduling on nodes which have upcoming maintenance, although it
> sounds like you see some benefit to a more subtle notion of scheduling
> priority based on upcoming maintenance? Do you think that maintenance-aware
> sorting would provide much more benefit to you over agent deactivation? Do
> you make use of the existing maintenance primitives to signal upcoming
> maintenance on agents?
>
> Thanks!
> Greg
>
> On Thu, Jun 6, 2019 at 9:37 AM Maxime Brugidou 
> wrote:
>
>> Hi,
>>
>> As a Mesos operator, I am really surprised by this proposal.
>>
>> The main advantage of the proposed design is that we can finally set
>> nodes down for maintenance with a configurable kill grace period and a
>> proper task status (with maintenance primitives, it was TASK_LOST I think)
>> without any specific cooperation from the frameworks.
>>
>> I think that this could be just an evolution of the current primitives.
>>
>> 

[Performance / Resource management WG] Notes in lieu of tomorrow's meeting

[Benjamin Mahler]

I'm out of the country and so I'm sending out notes in lieu of tomorrow's
performance / resource management meeting.

Resource Management:

- Work is underway for adding the UPDATE_FRAMEWORK scheduler::Call.
- Some fixes and small performance improvements landed for the random
sorter.
- Perf data from actual workload testing showed that the work we were doing
for incremental sorting wouldn't provide a significant impact, so that work
has been shelved for now.
- Logging was added to show the before and after state of quota headroom
during each allocation cycle.
- We will be getting back to the quota limits work shortly.

Performance:

- Some benchmarking was done for command health checks, which are rather
expensive since they create nested containers.
https://issues.apache.org/jira/browse/MESOS-9509

Please let us know if you have questions.

Ben

Re: Slack upgrade to Standard plan. Thanks Criteo

[Benjamin Mahler]

Thank you Criteo!

On Tue, Apr 23, 2019 at 1:12 PM Vinod Kone  wrote:

> Hi folks,
>
> As you probably realized today, we got our Slack upgraded from "free" plan
> to "standard" plan, which allows us to have unlimited message history and
> better analytics among other things! This would be great for our community.
>
> This upgrade has been made possible due to a general contribution/donation
> from folks at Criteo . Criteo has been a long
> time user of Apache Mesos and luckily for us, they wanted to contribute
> back to the ecosystem. We will update the website with the thanks shortly.
>
> Hope you'll take advantage of the standard plan.
>
> Thanks,
> Vinod
>
>
>

Performance / Resource Management Update

[Benjamin Mahler]

In lieu of today's meeting, this is an email update:

The 1.8 release process is underway, and it includes a few performance
related changes:

- Parallel reads for the v0 API have been extended to all other v0 read
only endpoints (e.g. /state-summary, /roles, etc). Whereas in 1.7.0, only
/state had the parallel read support. Also, requests are de-duplicated by
principal so that we don't perform redundant construction of responses if
we know they will be the same.

- The allocator performance has improved significantly when quota is in
use, benchmarking shows allocation cycle time reduced ~40% for a small size
cluster and up to ~70% for larger clusters.

- A per-framework (and per-role) override of the global
--min_allocatable_resources filter has been introduced. This lets
frameworks specify the minimum size of offers they want to see for their
roles, and improves scheduling efficiency by reducing the number of offers
declined for having insufficient resource quantities.

In the resource management area, we're currently working on the following
near term items:

- Investigating whether we can make some additional performance
improvements to the sorters (e.g. incremental sorting).
- Finishing the quota limits work, which will allow setting of limits
separate from guarantees.
- Adding an UPDATE_FRAMEWORK call to allow multi-role frameworks to change
their roles without re-subscribing.
- Exposing quota consumption via the API and UI (note that we currently
expose "allocation", but reservations are also considered against quota
consumption!)

There's lots more in the medium term, but I'll omit them here unless folks
are curious.

In the performance area, the following seem like the most pressing short
term items to me:

- Bring the v0 parallel read functionality to the v1 read-only calls.
- Bring v1 endpoint performance closer to v0.

Please chime in if there are any questions or comments,
Ben

Re: Subject: [VOTE] Release Apache Mesos 1.8.0 (rc1)

[Benjamin Mahler]

The CHANGELOG highlights seem a bit lacking?

- For some reason, the task CLI command is listed in a performance section?
- The parallel endpoint serving changes are in the longer list of items,
seems like we highlight them in the performance section? Maybe we could be
specific too about what we did additionally vs 1.7.0, since we already
announced parallel state reads in 1.7.0?
- We also have some additional allocator related performance improvements
in 1.8.0 that we need to add.
- Do we want to say something w.r.t.
https://issues.apache.org/jira/browse/MESOS-9211?

Not sure to what degree we care about an accurate CHANGELOG in the 1.8.0
tag vs updating the 1.8.0 CHANGELOG on master?

On Mon, Apr 15, 2019 at 2:26 PM Benno Evers  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.8.0.
>
>
> 1.8.0 includes the following:
>
> 
>  * Operation feedback for v1 schedulers.
>  * Per-framework minimum allocatable resources.
>  * New CLI subcommands `task attach` and `task exec`.
>  * New `linux/seccomp` isolator.
>  * Support for Docker v2 Schema2 manifest format.
>  * XFS quota for persistent volumes.
>  * **Experimental** Support for the new CSI v1 API.
>
> The CHANGELOG for the release is available at:
>
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.8.0-rc1
>
> 
>
> The candidate for Mesos 1.8.0 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.8.0-rc1/mesos-1.8.0.tar.gz
>
> The tag to be voted on is 1.8.0-rc1:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.8.0-rc1
>
> The SHA512 checksum of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.8.0-rc1/mesos-1.8.0.tar.gz.sha512
>
> The signature of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.8.0-rc1/mesos-1.8.0.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1251
>
> Please vote on releasing this package as Apache Mesos 1.8.0!
>
> The vote is open until Thursday, April 18 and passes if a majority of at
> least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.8.0
> [ ] -1 Do not release this package because ...
>
> Thanks,
> Benno and Joseph
>

Re: Mesos Master Crashes when Task launched with LAUNCH_GROUP fails

[Benjamin Mahler]

For posterity: https://issues.apache.org/jira/browse/MESOS-9619

On Thu, Feb 28, 2019 at 6:02 PM Meng Zhu  wrote:

> Hi Nimi:
>
> Thanks for reporting this.
>
> From the log snippet, looks like, when de-allocating resources, the agent
> does not have the port resources that is supposed to have been allocated.
> Can you provide the master log (which at least covers the period from when
> the resources on the agent is offered to the crash point)? Also, can you
> create a JIRA ticket and upload the log to there? (
> https://issues.apache.org/jira/projects/MESOS/issues)
>
> -Meng
>
> On Thu, Feb 28, 2019 at 1:58 PM Nimi W  wrote:
>
>> Hi,
>>
>> Mesos: 1.7.1
>>
>> I'm trying to debug an issue where if I launch a task using the
>> LAUNCH_GROUP method,
>> and the task fails to start, the mesos master will crash. I am using a
>> custom framework
>> I've built using the HTTP Scheduler API.
>>
>> When my framework received an offer - I return with an ACCEPT with this
>> JSON:
>>
>> https://gist.github.com/nemosupremo/3b23c4e1ca0ab241376aa5b975993270
>>
>> I then receive the following UPDATE events:
>>
>> TASK_STARTING
>> TASK_RUNNING
>> TASK_FAILED
>>
>> My framework then immediately tries to relaunch the task on the next
>> OFFERS:
>>
>> https://gist.github.com/nemosupremo/2b02443241c3bd002f04be034d8e64f7
>>
>> But between sometime when I get that event and try to acknowledge the
>> TASK_FAILED event,
>> the mesos master crashes with:
>>
>> Feb 28 21:34:02 master03 mesos-master[7124]: F0228 21:34:02.118693  7142
>> sorter.hpp:357] Check failed: resources.at(slaveId).contains(toRemove)
>> Resources disk(allocated: faust)(reservations: [(STATIC,faust)]):1;
>> cpus(allocated: faust)(reservations: [(STATIC,faust)]):0.1; mem(allocated:
>> faust)(reservations: [(STATIC,faust)]):64 at agent
>> 643078ba-8cb8-4582-b9c3-345d602506c8-S0 does not contain cpus(allocated:
>> faust)(reservations: [(STATIC,faust)]):0.1; mem(allocated:
>> faust)(reservations: [(STATIC,faust)]):64; disk(allocated:
>> faust)(reservations: [(STATIC,faust)]):1; ports(allocated:
>> faust)(reservations: [(STATIC,faust)]):[-]
>> Feb 28 21:34:02 master03 mesos-master[7124]: *** Check failure stack
>> trace: ***
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd935e48d
>> google::LogMessage::Fail()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd9360240
>> google::LogMessage::SendToLog()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd935e073
>> google::LogMessage::Flush()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd9360c69
>> google::LogMessageFatal::~LogMessageFatal()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd83d85f8
>> mesos::internal::master::allocator::DRFSorter::unallocated()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd83a78af
>> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::untrackAllocatedResources()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd83ba281
>> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::recoverResources()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd92a6631
>> process::ProcessBase::consume()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd92c878a
>> process::ProcessManager::resume()
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd92cc4d6
>> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd6289c80
>> (unknown)
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd5da56ba
>> start_thread
>> Feb 28 21:34:02 master03 mesos-master[7124]: @ 0x7f1fd5adb41d
>> (unknown)
>> Feb 28 21:34:02 master03 systemd[1]: mesos-master.service: Main process
>> exited, code=killed, status=6/ABRT
>> Feb 28 21:34:02 master03 systemd[1]: mesos-master.service: Unit entered
>> failed state.
>> Feb 28 21:34:02 master03 systemd[1]: mesos-master.service: Failed with
>> result 'signal'.
>>
>> The entire process works with the older LAUNCH API (for some reason the
>> docker task crashes with filesystem permission issues when using
>> LAUNCH_GROUPS)
>>
>

Re: Enabling framework authentication Loaded deprecated flag 'authenticate'

[Benjamin Mahler]

The --authenticate master flag has been renamed:

https://github.com/apache/mesos/blob/1.7.1/src/master/flags.cpp#L221

So yes, the documentation you linked to needs an update.

On Fri, Feb 15, 2019 at 12:26 PM Marc Roos  wrote:

>
> [@]# cat /etc/mesos-master/authenticate
> true
>
> Is this page old?
> http://mesos.apache.org/documentation/latest/authentication/
>
>
>
>
>
>
>
>

Re: centos7/el7 newer marathon rpms

[Benjamin Mahler]

Hi Marc,

You can reach out to the marathon community to get this question answered:
https://mesosphere.github.io/marathon/support.html

Ben

On Fri, Feb 8, 2019 at 6:33 PM Marc Roos  wrote:

>
> Where can I get newer marathon rpms, currently I am getting them from
> here
>
>
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.3.7-1.0.565.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.3.8-1.0.566.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.3.9-1.0.576.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.0-1.0.631.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.10-1.0.675.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.1-1.0.633.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.1-1.0.635.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.11-1.0.676.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.12-1.0.681.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.13-1.0.683.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.2-1.0.647.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.3-1.0.649.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.4-1.0.653.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.5-1.0.654.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.6-1.0.656.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.7-1.0.657.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.8-1.0.660.el7.x86_64.rpm
>
> http://repos.mesosphere.com/el/7/x86_64/RPMS/marathon-1.4.9-1.0.668.el7.x86_64.rpm
>

Re: Check failed: reservationScalarQuantities.contains(role)

[Benjamin Mahler]

Thanks for reporting this, we can help investigate this with you in JIRA.

On Tue, Feb 5, 2019 at 5:40 PM Jeff Pollard  wrote:

> Thanks for the info. I did find the "Removed agent" line as you suspected,
> but not much else in logging looked promising. I opened a JIRA to track
> from here on out https://issues.apache.org/jira/browse/MESOS-9555.
>
> On Tue, Feb 5, 2019 at 2:03 PM Joseph Wu  wrote:
>
>> From the stack, it looks like the master is attempting to remove an agent
>> from the master's in-memory state.  In the master's logs you should find a
>> line shortly before the exit, like:
>>
>>  master.cpp:] Removed agent : 
>>
>> The agent's ID should at least give you some pointer to which agent is
>> causing the problem.  Feel free to create a JIRA (
>> https://issues.apache.org/jira/) with any information you can glean.
>> This particular type of failure, a CHECK-failure, means some invariant has
>> been violated and usually means we missed a corner case.
>>
>> On Tue, Feb 5, 2019 at 12:04 PM Jeff Pollard 
>> wrote:
>>
>>> We recently upgraded our Mesos  cluster from version 1.3 to 1.5, and
>>> since then have been getting periodic master crashes due to this error:
>>>
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: F0205
>>> 15:53:57.385118  8434 hierarchical.cpp:2630] Check failed:
>>> reservationScalarQuantities.contains(role)
>>>
>>> Full stack trace is at the end of this email. When the master fails, we
>>> automatically restart it and it rejoins the cluster just fine. I did some
>>> initial searching and was unable to find any existing bug reports or other
>>> people experiencing this issue. We run a cluster of 3 masters, and see
>>> crashes on all 3 instances.
>>>
>>> Hope to get some guidance on what is going on and/or where to start
>>> looking for more information.
>>>
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e9170a7d  google::LogMessage::Fail()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e9172830  google::LogMessage::SendToLog()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e9170663  google::LogMessage::Flush()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e9173259  google::LogMessageFatal::~LogMessageFatal()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e8443cbd
>>> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::untrackReservations()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e8448fcd
>>> mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::removeSlave()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e90c4f11  process::ProcessBase::consume()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e90dea4a  process::ProcessManager::resume()
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e90e25d6
>>> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e6700c80  (unknown)
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e5f136ba  start_thread
>>> Feb  5 15:53:57 ip-10-0-16-140 mesos-master[8414]: @
>>>  0x7f87e5c4941d  (unknown)
>>>
>>

Re: Welcome Benno Evers as committer and PMC member!

[Benjamin Mahler]

Welcome Benno! Thanks for all the great contributions

On Wed, Jan 30, 2019 at 6:21 PM Alex R  wrote:

> Folks,
>
> Please welcome Benno Evers as an Apache committer and PMC member of the
> Apache Mesos!
>
> Benno has been active in the project for more than a year now and has made
> significant contributions, including:
>   * Agent reconfiguration, MESOS-1739
>   * Memory profiling, MESOS-7944
>   * "/state" performance improvements, MESOS-8345
>
> I have been working closely with Benno, paired up on, and shepherded some
> of his work. Benno has very strong technical knowledge in several areas and
> he is willing to share it with others and help his peers.
>
> Benno, thanks for all your contributions so far and looking forward to
> continuing to work with you on the project!
>
> Alex.
>

Re: [VOTE] Release Apache Mesos 1.7.1 (rc1)

[Benjamin Mahler]

+1 (binding)

make check passes on macOS 10.14.2

$ clang++ --version
Apple LLVM version 10.0.0 (clang-1000.10.44.4)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

$ ./configure CC=clang CXX=clang++ CXXFLAGS="-Wno-deprecated-declarations"
--disable-python --disable-java --with-apr=/usr/local/opt/apr/libexec
--with-svn=/usr/local/opt/subversion && make check -j12
...
[  PASSED  ] 1956 tests.

On Fri, Dec 21, 2018 at 5:48 PM Chun-Hung Hsiao  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.7.1.
>
>
> 1.7.1 includes the following:
>
> 
> * This is a bug fix release. Also includes performance and API
>   improvements:
>
>   * **Allocator**: Improved allocation cycle time substantially
> (see MESOS-9239 and MESOS-9249). These reduce the allocation
> cycle time in some benchmarks by 80%.
>
>   * **Scheduler API**: Improved the experimental `CREATE_DISK` and
> `DESTROY_DISK` operations for CSI volume recovery (see MESOS-9275
> and MESOS-9321). Storage local resource providers now return disk
> resources with the `source.vendor` field set, so frameworks needs to
> upgrade the `Resource` protobuf definitions.
>
>   * **Scheduler API**: Offer operation feedbacks now present their agent
> IDs and resource provider IDs (see MESOS-9293).
>
>
> The CHANGELOG for the release is available at:
>
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.7.1-rc1
>
> 
>
> The candidate for Mesos 1.7.1 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.7.1-rc1/mesos-1.7.1.tar.gz
>
> The tag to be voted on is 1.7.1-rc1:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.7.1-rc1
>
> The SHA512 checksum of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.7.1-rc1/mesos-1.7.1.tar.gz.sha512
>
> The signature of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/1.7.1-rc1/mesos-1.7.1.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is in a staging repository here:
>
> https://repository.apache.org/content/repositories/releases/org/apache/mesos/mesos/1.7.1-rc1/
>
> Please vote on releasing this package as Apache Mesos 1.7.1!
>
> To accommodate for the holidays, the vote is open until Mon Dec 31
> 14:00:00 PST 2018 and passes if a majority of at least 3 +1 PMC votes are
> cast.
>
> [ ] +1 Release this package as Apache Mesos 1.7.1
> [ ] -1 Do not release this package because ...
>
> Thanks,
> Chun-Hung & Gaston
>

Re: New scheduler API proposal: unsuppress and clear_filter

[Benjamin Mahler]

I think we're agreed:

-There are no schedulers modeling the existing per-agent time-based
filters that mesos is tracking, and we shouldn't go in a direction that
encourages frameworks to try to model and manage these. So, we should be
very careful in considering something like CLEAR_FILTERS. We're probably
also agreed that the current filters aren't so great. :)
-Letting a scheduler have more explicit control over the offers it gets
(both in shape of the offers and overall quantity of resources) is a good
direction to go in to reduce the inefficiency in the pessimistic offer
model.
-Combining matchers of model (2) with REVIVE may eliminate the need for
CLEAR_FILTERS. I think once you have global matchers in play, it eliminates
the need for the existing decline filters to involve resource subsets and
we may be able to move new schedulers forward with a better model without
breaking old schedulers.

I don’t think model (1) was understood as intended. Schedulers would not be
expressing limits, they would be expressing a "request" equivalent to “how
much more they want”. The internal effective limit (equal to
allocation+request) is just an implementation detail here that demonstrates
how it fits cleanly into the allocation algorithm. So, if a scheduler needs
to run 10 tasks with [1 cpu, 10GB mem], they would express a request of
[10cpus ,100GB mem] regardless of how much else is already allocated at
that role/scheduler node.

>From a scheduler's perspective the difference between the two models is:

(1) expressing "how much more" you need
(2) expressing an offer "matcher"

So:

(1) covers the middle part of the demand quantity spectrum we currently
have: unsuppressed -> infinite additional demand, suppressed -> 0
additional demand, and now also unsuppressed w/ request of X -> X
additional demand

(2) is a global filtering mechanism to avoid getting offers in an unusable
shape

They both solve inefficiencies we have, and they're complementary: a
"request" could actually consist of (1) and (2), e.g. "I need an additional
10 cpus, 100GB mem, and I want offers to contain [1cpu, 10GB mem]".

I'll schedule a meeting to discuss further. We should also make sure we
come back to the original problem in this thread around REVIVE retries.

On Mon, Dec 10, 2018 at 11:58 AM Benjamin Bannier <
benjamin.bann...@mesosphere.io> wrote:

> Hi Ben et al.,
>
> I'd expect frameworks to *always* know how to accept or decline offers in
> general. More involved frameworks might know how to suppress offers. I
> don't expect that any framework models filters and their associated
> durations in detail (that's why I called them a Mesos implementation
> detail) since there is not much benefit to a framework's primary goal of
> running tasks as quickly as possible.
>
> > I couldn't quite tell how you were imagining this would work, but let me
> spell out the two models that I've been considering, and you can tell me if
> one of these matches what you had in mind or if you had a different model
> in mind:
>
> > (1) "Effective limit" or "give me this much more" ...
>
> This sounds more like an operator-type than a framework-type API to me.
> I'd assume that frameworks would not worry about their total limit the way
> an operator would, but instead care about getting resources to run a
> certain task at a point in time. I could also imagine this being easy to
> use incorrectly as frameworks would likely need to understand their total
> limit when issuing the call which could require state or coordination among
> internal framework components (think: multi-purpose frameworks like
> Marathon or Aurora).
>
> > (2) "Matchers" or "give me things that look like this": when a scheduler
> expresses its "request" for a role, it would act as a "matcher" (opposite
> of filter). When mesos is allocating resources, it only proceeds if
> (requests.matches(resources) && !filters.filtered(resources)). The open
> ended aspect here is what a matcher would consist of. Consider a case where
> a matcher is a resource quantity and multiple are allowed; if any matcher
> matches, the result is a match. This would be equivalent to letting
> frameworks specify their own --min_allocatable_resources for a role (which
> is something that has been considered). The "matchers" could be more
> sophisticated: full resource objects just like filters (but global), full
> resource objects but with quantities for non-scalar resources like ports,
> etc.
>
> I was thinking in this direction, but what you described is more involved
> than what I had in mind as a possible first attempt. I'd expect that
> frameworks currently use `REVIVE` as a proxy for `REQUEST_RESOURCES`, not
> as a way to manage their filter state tracked in the allocator. Assuming we
> have some way to express resource quantities (i.e., MESOS-9314), we should
> be able to improve on `REVIVE` by providing a `REQUEST_RESOURCES` which
> clears all filters for resource containing the requested resources 

Re: New scheduler API proposal: unsuppress and clear_filter

[Benjamin Mahler]

Thanks for bringing REQUEST_RESOURCES up for discussion, it's one of the
mechanisms that we've been considering for further scaling pessimistic
offers before we make the migration to optimistic offers. It's also been
referred to as "demand" rather than "request", but for the sake of this
discussion consider them the same.

I couldn't quite tell how you were imagining this would work, but let me
spell out the two models that I've been considering, and you can tell me if
one of these matches what you had in mind or if you had a different model
in mind:

(1) "Effective limit" or "give me this much more": when a scheduler
expresses its "request" for a role, it would be equivalent to setting an
"effective limit" on the framework leaf node underneath the role node (i.e.
.../role/). The effective limit would probably be set to
(request + existing .../role/ wrote:

> Hi Meng,
>
> thanks for the proposal, I agree that the way these two aspects are
> currently entangled is an issue (e.g., for master/allocator performance
> reasons). At the same time, the workflow we currently expect frameworks to
> follow is conceptually not hard to grasp,
>
> (1) If framework has work then
> (i) put framework in unsuppressed state,
> (ii) decline not matching offers with a long filter duration.
> (2) If an offer matches, accept.
> (3) If there is no more work, suppress. GOTO (1).
>
> Here the framework does not need to track its filters across allocation
> cycles (they are an unexposed implementation detail of the hierarchical
> allocator anyway) which e.g., allows metaschedulers like Marathon or Apache
> Aurora to decouple the scheduling of different workloads. A downside of
> this interface is that
>
> * there is little incentive for frameworks to use SUPPRESS in addition to
> filters, and
> * unsupression is all-or-nothing, forcing the master to send potentially
> all unused resources to one framework, even if it is only interested in a
> fraction. This can cause, at least temporal, non-optimal allocation
> behavior.
>
> It seems to me that even though adding UNSUPPRESS and CLEAR_FILTERS would
> give frameworks more control, it would only be a small improvement. In
> above framework workflow we would allow a small improvement if the
> framework knows that a new workload matches a previously running workflow
> (i.e., it can infer that no filters for the resources it is interested in
> is active) so that it can issue UNSUPPRESS instead of CLEAR_FILTERS.
> Incidentally, there seems little local benefit for frameworks to use these
> new calls as they’d mostly help the master and I’d imagine we wouldn’t want
> to imply that clearing filters would unsuppress the framework. This seems
> too little to me, and we run the danger that frameworks would just always
> pair UNSUPPRESS and CLEAR_FILTERS (or keep using REVIVE) to simplify their
> workflow. If we’d model the interface more along framework needs, there
> would be clear benefit which would help adoption.
>
> A more interesting call for me would be REQUEST_RESOURCES. It maps very
> well onto framework needs (e.g., “I want to launch a task requiring these
> resources”), and clearly communicates a requirement to the master so that
> it e.g., doesn’t need to remove all filters for a framework. It also seems
> to fit the allocator model pretty well which doesn’t explicitly expose
> filters. I believe implementing it should not be too hard if we'd restrict
> its semantics to only communicate to the master that a framework _is
> interested in a certain resource_ without promising that the framework
> _will get them in any amount of time_ (i.e., no need to rethink DRF
> fairness semantics in the hierarchical allocator). I also feel that if we
> have REQUEST_RESOURCES we would have some freedom to perform further
> improvements around filters in the master/allocator (e.g., filter
> compatification, work around increasing the default filter duration, …).
>
>
> A possible zeroth implementation for REQUEST_RESOURCES with the
> hierarchical allocator would be to have it remove any filters containing
> the requested resource and likely to unsuppress the framework. A
> REQUEST_RESOURCES call would hold an optional resource and an optional
> AgentID; the case where both are empty would map onto CLEAR_FILTERS.
>
>
> That being said, it might still be useful to in the future expose a
> low-level knob for framework allowing them to explicitly manage their
> filters.
>
>
> Cheers,
>
> Benjamin
>
>
> On Dec 4, 2018, at 5:44 AM, Meng Zhu  wrote:
> >
> > See my comments inline.
> >
> > On Mon, Dec 3, 2018 at 5:43 PM Vinod Kone  wrote:
> >
> >> Thanks Meng for the explanation.
> >>
> >> I imagine most frameworks do not remember what stuff they filtered much
> >> less figure out how previously filtered stuff  can satisfy new
> operations.
> >> That sounds complicated!
> >>
> >
> > Frameworks do not need to remember what filters they currently have. Only
> > knowing
> > the resource profiles of the current 

Re: [API WG] Proposals for dealing with master subscriber leaks.

[Benjamin Mahler]

>- We can add heartbeats to the SUBSCRIBE call.
> This would need to be
>  part of a separate operator Call, because one platform (browsers) that
> might subscribe to the master does not support two-way streaming.

This doesn't make sense to me, the heartbeats should still be part of the
same connection (request and response are infinite and heartbeating) by
default. Splitting into a separate call is messy and shouldn't be what we
force everyone to do, it should only be done in cases that it's impossible
to use a single connection (e.g. browsers).

On Sat, Nov 10, 2018 at 12:03 AM Joseph Wu  wrote:

> Hi all,
>
> During some internal scale testing, we noticed that, when Mesos streaming
> endpoints are accessed via certain proxies (or load balancers), the proxies
> might not close connections after they are complete.  For the Mesos master,
> which only has the /api/v1 SUBSCRIBE streaming endpoint, this can generate
> unnecessary authorization requests and affects performance.
>
> We are considering a few potential solutions:
>
>- We can add heartbeats to the SUBSCRIBE call.  This would need to be
>part of a separate operator Call, because one platform (browsers) that
>might subscribe to the master does not support two-way streaming.
>- We can add (optional) arguments to the SUBSCRIBE call, which tells the
>master to disconnect it after a while.  And the client would have to
> remake
>the connection every so often.
>- We can change the master to hold subscribers in a circular buffer, and
>disconnect the oldest ones if there are too many connections.
>
> We're tracking progress on this issue here:
> https://issues.apache.org/jira/browse/MESOS-9258
> Some prototypes of the code changes involved are also linked in the JIRA.
>
> Please chime in if you have any suggestions or if any of these options
> would be undesirable/bad,
> ~Joseph
>

Re: Rhythm - time-based job scheduler

[Benjamin Mahler]

Thanks for sharing Michał! could you tell us how you (or your employer) are
using it?

On Tue, Oct 30, 2018 at 10:34 AM Michał Łowicki  wrote:

> Hey!
>
> I would like to announce project I've been working on recently -
> https://github.com/mlowicki/rhythm. It's a Cron-like scheduler with
> couple of additional features:
> * ACLs backend by either LDAP or GitLab
> * Integration with Vault by HashiCorp for secrets management
> * Support for both Docker and Mesos containerizers
>
> Feature requests / ideas / bug reports are more than welcome o/
>
> --
> BR,
> Michał Łowicki
>

Welcome Meng Zhu as PMC member and committer!

[Benjamin Mahler]

Please join me in welcoming Meng Zhu as a PMC member and committer!

Meng has been active in the project for almost a year and has been very
productive and collaborative. He is now one of the few people of
understands the allocator code well, as well as the roadmap for this area
of the project. He has also found and fixed bugs, and helped users in slack.

Thanks for all your work so far Meng, I'm looking forward to more of your
contributions in the project.

Ben

Re: Dedup mesos agent status updates at framework

[Benjamin Mahler]

The timeout behavior sounds like a dangerous scalability tripwire. Consider
revisiting that approach.

On Sun, Oct 28, 2018 at 10:42 PM Varun Gupta 
wrote:

> Mesos Version: 1.6
>
> scheduler has 250k events in its queue: Master master sends status updates
> to scheduler, and scheduler stores them in the queue. Scheduler process in
> FIFO, and once processed (includes persisting to DB) it ack the update.
> These updates are processed asynchronously with a thread pool of 1000 size.
> We are using explicit reconciliation.
> If the ack to Mesos Master is timing out, due to high CPU usage then next
> ack will likely fail too. It slows down processing on Scheduler side,
> meanwhile Mesos Master continuous to send status updates (duplicate ones,
> since old status updates are not ack). This leads to building up of status
> updates at Scheduler to be processed, and we have seen it to grow upto 250k
> status updates.
>
> Timeout is the explicit ack request from Scheduler to Mesos Master.
>
> Mesos Master profiling: Next time, when this issue occurs, I will take the
> dump.
>
> Deduplication is for the status updates present in the queue for scheduler
> to process, idea is to dedup duplicate status updates such that scheduler
> only processes same status update pending in queue once, and ack to Mesos
> Master also ones. It will reduce the load for both Scheduler and Mesos
> Master. After the ack (success/fail) scheduler will remove the status
> update from the queue, and in case of failure, Mesos Master will send
> status update again.
>
>
>
> On Sun, Oct 28, 2018 at 10:15 PM Benjamin Mahler 
> wrote:
>
> > Which version of mesos are you running?
> >
> > > In framework, event updates grow up to 250k
> >
> > What does this mean? The scheduler has 250k events in its queue?
> >
> > > which leads to cascading effect on higher latency at Mesos Master (ack
> > requests with 10s timeout)
> >
> > Can you send us perf stacks of the master during such a time window so
> > that we can see if there are any bottlenecks?
> > http://mesos.apache.org/documentation/latest/performance-profiling/
> >
> > Where is this timeout coming from and how is it used?
> >
> > > simultaneously explore if dedup is an option
> >
> > I don't know what you're referring to in terms of de-duplication. Can you
> > explain how the scheduler's status update processing works? Does it use
> > explicit acknowledgements and process batches asynchronously? Aurora
> > example: https://reviews.apache.org/r/33689/
> >
> > On Sun, Oct 28, 2018 at 8:58 PM Varun Gupta 
> > wrote:
> >
> >> Hi Benjamin,
> >>
> >> In our batch workload use case, number of tasks churn is pretty high. We
> >> have seen 20-30k tasks launch within 10 second window and 100k+ tasks
> >> running.
> >>
> >> In framework, event updates grow up to 250k, which leads to cascading
> >> effect on higher latency at Mesos Master (ack requests with 10s timeout)
> >> as
> >> well as blocking framework to process new since there are too many left
> to
> >> be acknowledged.
> >>
> >> Reconciliation is every 30 mins which also adds pressure on event stream
> >> if
> >> too many unacknowledged.
> >>
> >> I am thinking to experiment with default backoff period from 10s -> 30s
> or
> >> 60s, and simultaneously explore if dedup is an option.
> >>
> >> Thanks,
> >> Varun
> >>
> >> On Sun, Oct 28, 2018 at 6:49 PM Benjamin Mahler 
> >> wrote:
> >>
> >> > Hi Varun,
> >> >
> >> > What problem are you trying to solve precisely? There seems to be an
> >> > implication that the duplicate acknowledgements are expensive. They
> >> should
> >> > be low cost, so that's rather surprising. Do you have any data related
> >> to
> >> > this?
> >> >
> >> > You can also tune the backoff rate on the agents, if the defaults are
> >> too
> >> > noisy in your setup.
> >> >
> >> > Ben
> >> >
> >> > On Sun, Oct 28, 2018 at 4:51 PM Varun Gupta  wrote:
> >> >
> >> > >
> >> > > Hi,
> >> > >>
> >> > >> Mesos agent will send status updates with exponential backoff until
> >> ack
> >> > >> is received.
> >> > >>
> >> > >> If processing of events at framework and sending ack to Master is
> >> > running
> >> > >> slow then it builds a back pressure at framework due to duplicate
> >> > updates
> >> > >> for same status.
> >> > >>
> >> > >> Has someone explored the option to dedup same status update event
> at
> >> > >> framework or is it even advisable to do. End goal is to dedup all
> >> events
> >> > >> and send only one ack back to Master.
> >> > >>
> >> > >> Thanks,
> >> > >> Varun
> >> > >>
> >> > >>
> >> > >>
> >> >
> >>
> >
>

Re: Dedup mesos agent status updates at framework

[Benjamin Mahler]

Which version of mesos are you running?

> In framework, event updates grow up to 250k

What does this mean? The scheduler has 250k events in its queue?

> which leads to cascading effect on higher latency at Mesos Master (ack
requests with 10s timeout)

Can you send us perf stacks of the master during such a time window so that
we can see if there are any bottlenecks?
http://mesos.apache.org/documentation/latest/performance-profiling/

Where is this timeout coming from and how is it used?

> simultaneously explore if dedup is an option

I don't know what you're referring to in terms of de-duplication. Can you
explain how the scheduler's status update processing works? Does it use
explicit acknowledgements and process batches asynchronously? Aurora
example: https://reviews.apache.org/r/33689/

On Sun, Oct 28, 2018 at 8:58 PM Varun Gupta  wrote:

> Hi Benjamin,
>
> In our batch workload use case, number of tasks churn is pretty high. We
> have seen 20-30k tasks launch within 10 second window and 100k+ tasks
> running.
>
> In framework, event updates grow up to 250k, which leads to cascading
> effect on higher latency at Mesos Master (ack requests with 10s timeout) as
> well as blocking framework to process new since there are too many left to
> be acknowledged.
>
> Reconciliation is every 30 mins which also adds pressure on event stream if
> too many unacknowledged.
>
> I am thinking to experiment with default backoff period from 10s -> 30s or
> 60s, and simultaneously explore if dedup is an option.
>
> Thanks,
> Varun
>
> On Sun, Oct 28, 2018 at 6:49 PM Benjamin Mahler 
> wrote:
>
> > Hi Varun,
> >
> > What problem are you trying to solve precisely? There seems to be an
> > implication that the duplicate acknowledgements are expensive. They
> should
> > be low cost, so that's rather surprising. Do you have any data related to
> > this?
> >
> > You can also tune the backoff rate on the agents, if the defaults are too
> > noisy in your setup.
> >
> > Ben
> >
> > On Sun, Oct 28, 2018 at 4:51 PM Varun Gupta  wrote:
> >
> > >
> > > Hi,
> > >>
> > >> Mesos agent will send status updates with exponential backoff until
> ack
> > >> is received.
> > >>
> > >> If processing of events at framework and sending ack to Master is
> > running
> > >> slow then it builds a back pressure at framework due to duplicate
> > updates
> > >> for same status.
> > >>
> > >> Has someone explored the option to dedup same status update event at
> > >> framework or is it even advisable to do. End goal is to dedup all
> events
> > >> and send only one ack back to Master.
> > >>
> > >> Thanks,
> > >> Varun
> > >>
> > >>
> > >>
> >
>

Re: Dedup mesos agent status updates at framework

[Benjamin Mahler]

Hi Varun,

What problem are you trying to solve precisely? There seems to be an
implication that the duplicate acknowledgements are expensive. They should
be low cost, so that's rather surprising. Do you have any data related to
this?

You can also tune the backoff rate on the agents, if the defaults are too
noisy in your setup.

Ben

On Sun, Oct 28, 2018 at 4:51 PM Varun Gupta  wrote:

>
> Hi,
>>
>> Mesos agent will send status updates with exponential backoff until ack
>> is received.
>>
>> If processing of events at framework and sending ack to Master is running
>> slow then it builds a back pressure at framework due to duplicate updates
>> for same status.
>>
>> Has someone explored the option to dedup same status update event at
>> framework or is it even advisable to do. End goal is to dedup all events
>> and send only one ack back to Master.
>>
>> Thanks,
>> Varun
>>
>>
>>

Re: Proposal: Adding health check definitions to master state output

[Benjamin Mahler]

> It's worth mentioning that I believe the original intention of the 'Task'
> message was to contain most information contained in 'TaskInfo', except
for
> those fields which could grow very large, like the 'data' field.

+1 all task / executor metadata should be exposed IMO. I look at the 'data'
field as a payload delivered to the executor / task rather than it being
part of the metadata. Based on this, if one wanted to have custom metadata
that gets exposed, labels would be used instead.

On Thu, Oct 18, 2018 at 3:21 PM Greg Mann  wrote:

> Hi all,
> In addition to the health check API change proposal that I recently sent
> out, we're considering adding a task's health check definition (when
> present) to the 'Task' protobuf message so that it appears in the master's
> '/state' endpoint response, as well as the v1 GET_STATE response and the
> TASK_ADDED event. This will allow operators to detect the presence and
> configuration of health checks on tasks via the operator API, which they
> are currently unable to do:
>
> message Task {
>   . . .
>
>   optional HealthCheck health_check = 15;
>
>   . . .
> }
>
> I wanted to check in with the community regarding this change, since for
> very large clusters it could have a non-negligible impact on the size of
> the master's state output.
>
> It's worth mentioning that I believe the original intention of the 'Task'
> message was to contain most information contained in 'TaskInfo', except for
> those fields which could grow very large, like the 'data' field.
>
> Please reply if you foresee this change having a negative impact on your
> deployments, or if you have any other thoughts/concerns!
>
> Thanks,
> Greg
>

1.7.x Performance Improvements Blog Post

[Benjamin Mahler]

We published a blog post highlighting the performance improvements in Mesos
1.7.x, take a look!

https://twitter.com/ApacheMesos/status/1049740950359044096

Ben

Re: Vote now for MesosCon 2018 proposals!

[Benjamin Mahler]

Voted! Thanks Jörg and the PC!

On Thu, Sep 20, 2018 at 9:51 AM Jörg Schad  wrote:

> Dear Mesos Community,
>
> Please take a few minutes over the next few days and review what members
> of the community have submitted for MesosCon 2018
>  (which will be held in San Francisco between
> November 5th-7th)!
> To make voting easier, we structured the voting following the different
> tracks.
> Please visit the following links and submit your responses. Look through
> as few or as many talks as you'd like to, and give us your feedback on
> these talks.
>
> Core Track: https://www.surveymonkey.com/r/mesoscon18-core
> Ecosystem Track: https://www.surveymonkey.com/r/mesoscon18-ecosystem
> DC/OS Track: https://www.surveymonkey.com/r/mesoscon18-dcos
> Frameworks Track: https://www.surveymonkey.com/r/mesoscon18-frameworks
> Operations Tracks: https://www.surveymonkey.com/r/mesoscon18-operations
> Misc Track: https://www.surveymonkey.com/r/mesoscon18-misc
> User Track: https://www.surveymonkey.com/r/mesoscon18-users
>
> Please submit your votes until Wednesday, Sept 26th 11:59 PM PDT, so you
> have one week to vote and make your voice heard!
>
> Thank you for your help and looking forward to a great MesosCon!
> Your MesosCon PC
>

Re: [VOTE] Release Apache Mesos 1.4.2 (rc1)

[Benjamin Mahler]

+1 (binding)

make check passes on macOS 10.13.6 with Apple LLVM version 9.1.0
(clang-902.0.39.2).

Thanks Kapil!

On Wed, Aug 8, 2018 at 3:06 PM, Kapil Arya  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.4.2.
>
> 1.4.2 is a bug fix release. The CHANGELOG for the release is available at:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.4.2-rc1
>
> The candidate for Mesos 1.4.2 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/mesos-1.4.2.tar.gz
>
> The tag to be voted on is 1.4.2-rc1:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.4.2-rc1
>
> The SHA512 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/
> mesos-1.4.2.tar.gz.sha512
>
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/
> mesos-1.4.2.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1231
>
> Please vote on releasing this package as Apache Mesos 1.4.2!
>
> The vote is open until Sat Aug 11 11:59:59 PDT 2018 and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.4.2
> [ ] -1 Do not release this package because ...
>
> Thanks,
> Anand and Kapil
>

Re: [VOTE] Release Apache Mesos 1.4.2 (rc1)

[Benjamin Mahler]

This was fixed in
https://github.com/apache/mesos/commit/02ad5c8cdd644ee8eec83bf887daa98bb163637d,
I don't recall there being any issues due to it.

On Mon, Aug 13, 2018 at 4:50 PM, Benjamin Mahler  wrote:

> Hm.. I ran make check on macOS saw the following:
>
> [ RUN  ] AwaitTest.AwaitSingleDiscard
> src/tests/collect_tests.cpp:275: Failure
> Value of: promise.future().hasDiscard()
>   Actual: false
> Expected: true
> [  FAILED  ] AwaitTest.AwaitSingleDiscard (0 ms)
>
> On Wed, Aug 8, 2018 at 3:06 PM, Kapil Arya  wrote:
>
>> Hi all,
>>
>> Please vote on releasing the following candidate as Apache Mesos 1.4.2.
>>
>> 1.4.2 is a bug fix release. The CHANGELOG for the release is available at:
>> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain
>> ;f=CHANGELOG;hb=1.4.2-rc1
>>
>> The candidate for Mesos 1.4.2 release is available at:
>> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/mesos-1.4.2.tar.gz
>>
>> The tag to be voted on is 1.4.2-rc1:
>> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.4.2-rc1
>>
>> The SHA512 checksum of the tarball can be found at:
>> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/mesos
>> -1.4.2.tar.gz.sha512
>>
>> The signature of the tarball can be found at:
>> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/mesos
>> -1.4.2.tar.gz.asc
>>
>> The PGP key used to sign the release is here:
>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>>
>> The JAR is in a staging repository here:
>> https://repository.apache.org/content/repositories/orgapachemesos-1231
>>
>> Please vote on releasing this package as Apache Mesos 1.4.2!
>>
>> The vote is open until Sat Aug 11 11:59:59 PDT 2018 and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Mesos 1.4.2
>> [ ] -1 Do not release this package because ...
>>
>> Thanks,
>> Anand and Kapil
>>
>
>

Re: [VOTE] Release Apache Mesos 1.4.2 (rc1)

[Benjamin Mahler]

Hm.. I ran make check on macOS saw the following:

[ RUN  ] AwaitTest.AwaitSingleDiscard
src/tests/collect_tests.cpp:275: Failure
Value of: promise.future().hasDiscard()
  Actual: false
Expected: true
[  FAILED  ] AwaitTest.AwaitSingleDiscard (0 ms)

On Wed, Aug 8, 2018 at 3:06 PM, Kapil Arya  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.4.2.
>
> 1.4.2 is a bug fix release. The CHANGELOG for the release is available at:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.4.2-rc1
>
> The candidate for Mesos 1.4.2 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/mesos-1.4.2.tar.gz
>
> The tag to be voted on is 1.4.2-rc1:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.4.2-rc1
>
> The SHA512 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/
> mesos-1.4.2.tar.gz.sha512
>
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.4.2-rc1/
> mesos-1.4.2.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1231
>
> Please vote on releasing this package as Apache Mesos 1.4.2!
>
> The vote is open until Sat Aug 11 11:59:59 PDT 2018 and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.4.2
> [ ] -1 Do not release this package because ...
>
> Thanks,
> Anand and Kapil
>

Re: Understand fixed resource estimator to get oversubscribe resources

[Benjamin Mahler]

The fixed resource estimator provides a fixed size revocable pool: if you
tell it to create a 24 cpu revocable pool, there will be a 24 cpu revocable
pool. It is not looking at utilization slack.

On Mon, Aug 6, 2018 at 2:28 PM, Varun Gupta  wrote:

> Hi,
>
> I was reading the code
> 
> for fixed resource estimator and want to understand the behavior for it.
>
> For example, CPU is the only revocable resource for the Mesos Cluster. I
> have one machine with 24 CPUs as non-revocable, and correspondingly 24
> CPU's are revocable resources (hardcoded for fixed resource estimator).
>
> Below are two scenario's, I want to confirm what is the actual behavior or
> it is completely different.
>
> 1) Static behavior, were on 24 CPU revocable and 24 CPU non-revocable, 12
> CPU non-revocable task is launched with 100% utilization. In this scenario,
> resources offered 12 CPU non-revocable + 24 CPU revocable resources.
>
> 2) Same as above scenario, and reading the document
> 
> for Revocable resources for Aurora. *"A revocable CPU resource value will
> include unallocated + unused resource"*. For above scenario, fixed
> resource estimator should return 12 CPU non-revocable and 12 CPU revocable
> (as 12 CPU non-revocable are running on 100% utilization).
> Following up this scenario, suppose 12 CPU non-revocable task is launched
> and of that 8 CPU are effectively used. Thereby, offer generated will be 12
> CPU non-revocable and 12 (available) + 4 (usage slack) CPU revocable
> resources.
>
> Thanks,
> Varun
>
>

Re: Backport Policy

[Benjamin Mahler]

uch cases to a
>>> release
>>> > >> manager in my opinion will help us enforce the strategy of minimal
>>> > number
>>> > >> backports. As a bonus, the release manager will have a much better
>>> > >> understanding of what's going on with the release, keyword: "more
>>> > >> ownership".
>>> > >>
>>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>>> > >> and...@schwartzmeyer.com> wrote:
>>> > >>
>>> > >>> I believe I fall somewhere between Alex and Ben.
>>> > >>>
>>> > >>> As for deciding what to backport or not, I lean toward Alex's view
>>> of
>>> > >>> backporting as little as possible (and agree with his criteria). My
>>> > >>> reasoning is that all changes can have unforeseen consequences,
>>> which I
>>> > >>> believe is something to be actively avoided in already released
>>> > versions.
>>> > >>> The reason for backporting patches to fix regressions is the same
>>> as
>>> > the
>>> > >>> reason to avoid backporting as much as possible: keep behavior
>>> > consistent
>>> > >>> (and safe) within a release. With that as the goal of a branch in
>>> > >>> maintenance mode, it makes sense to fix regressions, and make
>>> > exceptions to
>>> > >>> fix CVEs and other critical/blocking issues.
>>> > >>>
>>> > >>> As for who should decide what to backport, I lean toward Ben's
>>> view of
>>> > >>> the burden being on the committer. I don't think we should add more
>>> > work
>>> > >>> for release managers, and I think the committer/shepherd obviously
>>> has
>>> > the
>>> > >>> most understanding of the context around changes proposed for
>>> backport.
>>> > >>>
>>> > >>> Here's an example of a recent bugfix which I backported:
>>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>>> > >>>
>>> > >>> While normally I believe this change falls under "avoid due to
>>> > >>> unforeseen consequences," I made an exception as the bug was old,
>>> circa
>>> > >>> 2015, (indicating it had been an issue for others), and was causing
>>> > >>> recurring failures in testing. The fix itself was very small,
>>> meaning
>>> > it
>>> > >>> was easier to evaluate for possible side effects, so I felt a
>>> little
>>> > safer
>>> > >>> in that regard. The effect of not having the fix was a fatal and
>>> > undesired
>>> > >>> crash, which furthermore left troublesome side effects on the
>>> system
>>> > (you
>>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>>> > (DC/OS)
>>> > >>> wanted it in their next bump, which necessitated backporting to the
>>> > release
>>> > >>> they were pulling in.
>>> > >>>
>>> > >>> I think in general we should backport only as necessary, and leave
>>> it
>>> > on
>>> > >>> the committers to decide if backporting a particular change is
>>> > necessary.
>>> > >>>
>>> > >>>
>>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>>> > >>>
>>> > >>>> This is exactly where our views differ, Ben : )
>>> > >>>>
>>> > >>>> Ideally, I would like a release manager to have more ownership and
>>> > less
>>> > >>>> manual work. In my imagination, a release manager has more power
>>> and
>>> > >>>> control about dates, features, backports and everything that is
>>> > related
>>> > >>>> to
>>> > >>>> "their" branch. I would also like us to back port as little as
>>> > >>>> possible, to
>>> > >>>> simplify testing and releasing patch versions.
>>> > >>>>
>>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>>> bmah...@apache.org>
&g

Mesos 1.7.x and JSON clients

[Benjamin Mahler]

TLDR: If you use a spec-compliant JSON parser, you will observe no change
in Mesos 1.7.x and everything will continue to work as before.

Longer version:

JSON allows strings to be encoded in several different ways. For example,
"/" can be encoded directly as "/", or "\/", "\u002F", or "\u002f". A
spec-compliant parser [1] must handle these and produce the same result for
each.

In Mesos 1.7.0, we're adopting rapidjson [2] in order to provide
performance improvements to /state serving as well as all other json
serving and serialization paths. Rapidjson will encode strings in a
spec-compliant way, but the choices differ from the original JSON
serialization code in Mesos (e.g. rapidjson encodes "/" unescaped whereas
we previously encoded the escaped form "\/").

Please be sure that your JSON libraries used for parsing comply with one of
the JSON specs. There are several updates to RFC 4627, but they are all
equivalent from the perspective of string encoding and parser requirements
for strings FWICT.

There are some other topics that warrant discussion at some point, like
number precision, but we're not changing anything in Mesos 1.7.0 related to
this.

Ben

[1] https://tools.ietf.org/html/rfc8259#section-9
[2] http://rapidjson.org/

Re: Backport Policy

[Benjamin Mahler]

+user, I probably it would be good to hear from users as well.

Please see the original proposal as well as Alex's proposal and let us know
your thoughts.

To continue the discussion from where Alex left off:

> Other bugs and significant improvements, e.g., performance, may be back 
> ported,
the release manager should ideally be the one who decides on this.

I'm a little puzzled by this, why is the release manager involved? As we
already document, backports occur when the bug is fixed, so this happens in
the steady state of development, not at release time. The release manager
only comes in at the time of the release itself, at which point all
backports have already happened and the release manager handles the release
process. Only blocker level issues can stop the release and while the
release manager has a strong say, we should generally agree on what
consists of a release blocking issue.

Just to clarify my workflow, I generally backport every bug fix I commit
that applies cleanly, right after I commit it to master (with the
exceptions I listed below).

On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov  wrote:

> I would like to back port as little as possible. I suggest the following
> criteria:
>
> * By default, regressions are back ported to existing release branches. A
> bug is considered a regression if the functionality is present in the
> previous minor or patch version and is not affected by the bug there.
>
> * Critical and blocker issues, e.g., a CVE, can be back ported.
>
> * Other bugs and significant improvements, e.g., performance, may be back
> ported, the release manager should ideally be the one who decides on this.
>
> On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone  wrote:
>
> > Ben, thanks for the clarification. I'm in agreement with the points you
> > made.
> >
> > Once we have consensus, would you mind updating the doc?
> >
> > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler 
> > wrote:
> >
> > > I realized recently that we aren't all on the same page with
> backporting.
> > > We currently only document the following:
> > >
> > > "Typically the fix for an issue that is affecting supported releases
> > lands
> > > on the master branch and is then backported to the release branch(es).
> In
> > > rare cases, the fix might directly go into a release branch without
> > landing
> > > on master (e.g., fix / issue is not applicable to master)." [1]
> > >
> > > This leaves room for interpretation about what lies outside of
> "typical".
> > > Here's the simplest way I can explain what I stick to, and I'd like to
> > hear
> > > what others have in mind:
> > >
> > > * By default, bug fixes at any level should be backported to existing
> > > release branches if it affects those releases. Especially important:
> > > crashes, bugs in non-experimental features.
> > >
> > > * Exceptional cases that can omit backporting: difficult to backport
> > fixes
> > > (especially if the bugs are deemed of low priority), bugs in
> experimental
> > > features.
> > >
> > > * Exceptional non-bug cases that can be backported: performance
> > > improvements.
> > >
> > > I realize that there is a ton of subtlety here (even in terms of which
> > > things are defined as bugs). But I hope we can lay down a policy that
> > gives
> > > everyone the right mindset for common cases and then discuss corner
> cases
> > > on-demand in the future.
> > >
> > > [1] http://mesos.apache.org/documentation/latest/versioning/
> > >
> >
>

Re: Normalization of metric keys

[Benjamin Mahler]

Do we also want:

3. Has an unambiguous decoding.

Replacing '/' with '#%$' means I don't know if the user actually supplied
'#%$' or '/'. But using something like percent-encoding would have property
3.

On Fri, Jul 6, 2018 at 10:25 AM, Greg Mann  wrote:

> Thanks for the reply Ben!
>
> Yea I suspect the lack of normalization there was not intentional, and it
> means that you can no longer reliably split on '/' unless you apply some
> external controls to user input. Yep, this is bad :)
>
> One thing we should consider when normalizing metadata embedded in metric
> keys (like framework name/ID) is that operators will likely want to
> de-normalize this information in their metrics tooling. For example,
> ideally something like the 'mesos_exporter' [1] could expose the framework
> name/ID as tags which could be easily consumed by the cluster's metrics
> infrastructure.
>
> To accommodate de-normalization, any substitutions we perform while
> normalizing should be:
>
>1. Unique - we should substitute a single, unique string for each
>disallowed character
>2. Verbose - we should substitute strings which are unlikely to appear
>in user input. (Examples: 

Re: Normalization of metric keys

[Benjamin Mahler]

I don't think the lack of principal normalization was intentional. Why
spread that further? Don't we also have some normalization today?

Having slashes show up in components complicates parsing (can no longer
split on '/'), no? For example, if we were to introduce the ability to
query a subset of metrics with a simple matcher (e.g.
/frameworks/*/messages_received), then this would be complicated by the
presence of slashes in the principal or other user supplied strings.

On Tue, Jul 3, 2018 at 3:17 PM, Greg Mann  wrote:

> Hi all!
> I'm currently working on adding a suite of new per-framework metrics to
> help schedulers better debug unexpected/unwanted behavior (MESOS-8842
> ). One issue that has
> come up during this work is how we should handle strings like the framework
> name or role name in metric keys, since those strings may contain
> characters like '/' which already have a meaning in our metrics interface.
> I intend to place the framework name and ID in the keys for the new
> per-framework metrics, delimited by a sufficiently-unique separator so that
> operators can decode the name/ID in their metrics tooling. An example
> per-framework metric key:
>
> master/frameworks/###/tasks/task_running
>
>
> I recently realized that we actually already allow the '/' character in
> metric keys, since we include the framework principal in these keys:
>
> frameworks//messages_received
> frameworks//messages_processed
>
> We don't disallow any characters in the principal, so anything could
> appear in those keys.
>
> *Since we don't normalize the principal in the above keys, my proposal is
> that we do not normalize the framework name at all when constructing the
> new per-framework metric keys.*
>
>
> Let me know what you think!
>
> Cheers,
> Greg
>

Re: [VOTE] Release Apache Mesos 1.3.3 (rc1)

[Benjamin Mahler]

+1 (binding)

Make check passes on macOS 10.13.4 with Apple LLVM version 9.1.0
(clang-902.0.39.1).

On Wed, May 23, 2018 at 10:00 PM, Michael Park  wrote:

> The tarball has been fixed, please vote now!
>
> 'twas BSD `tar` issues... :(
>
> Thanks,
>
> MPark
>
> On Wed, May 23, 2018 at 11:39 AM, Michael Park  wrote:
>
>> Huh... 樂 Super weird. I'll look into it.
>>
>> Thanks for checking!
>>
>> MPark
>>
>> On Wed, May 23, 2018 at 11:34 AM Vinod Kone  wrote:
>>
>>> It's empty for me too!
>>>
>>> On Wed, May 23, 2018 at 11:32 AM, Benjamin Mahler 
>>> wrote:
>>>
>>>> Thanks Michael!
>>>>
>>>> Looks like the tar.gz is empty, is it just me?
>>>>
>>>> On Tue, May 22, 2018 at 10:09 PM, Michael Park 
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Please vote on releasing the following candidate as Apache Mesos 1.3.3.
>>>>>
>>>>> The CHANGELOG for the release is available at:
>>>>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_p
>>>>> lain;f=CHANGELOG;hb=1.3.3-rc1
>>>>> 
>>>>> 
>>>>>
>>>>> The candidate for Mesos 1.3.3 release is available at:
>>>>> https://dist.apache.org/repos/dist/dev/mesos/1.3.3-rc1/mesos
>>>>> -1.3.3.tar.gz
>>>>>
>>>>> The tag to be voted on is 1.3.3-rc1:
>>>>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit
>>>>> ;h=1.3.3-rc1
>>>>>
>>>>> The SHA512 checksum of the tarball can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/mesos/1.3.3-rc1/mesos
>>>>> -1.3.3.tar.gz.sha512
>>>>>
>>>>> The signature of the tarball can be found at:
>>>>> https://dist.apache.org/repos/dist/dev/mesos/1.3.3-rc1/mesos
>>>>> -1.3.3.tar.gz.asc
>>>>>
>>>>> The PGP key used to sign the release is here:
>>>>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>>>>>
>>>>> The JAR is up in Maven in a staging repository here:
>>>>> https://repository.apache.org/content/repositories/orgapachemesos-1226
>>>>>
>>>>> Please vote on releasing this package as Apache Mesos 1.3.3!
>>>>>
>>>>> The vote is open until Fri May 25 22:07:39 PDT 2018 and passes if a
>>>>> majority of at least 3 +1 PMC votes are cast.
>>>>>
>>>>> [ ] +1 Release this package as Apache Mesos 1.3.3
>>>>> [ ] -1 Do not release this package because ...
>>>>>
>>>>> Thanks,
>>>>>
>>>>> MPark
>>>>>
>>>>
>>>>
>>>
>

Re: [VOTE] Release Apache Mesos 1.5.1 (rc1)

[Benjamin Mahler]

+1 (binding)

make check passes on macOS 10.13.4 with Apple LLVM version 9.1.0
(clang-902.0.39.1)

On Fri, May 11, 2018 at 12:35 PM, Gilbert Song  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.5.1.
>
> 1.5.1 includes the following:
> 
> 
> * [MESOS-1720] - Slave should send exited executor message when the
> executor is never launched.
> * [MESOS-7742] - Race conditions in IOSwitchboard: listening on unix socket
> and premature closing of the connection.
> * [MESOS-8125] - Agent should properly handle recovering an executor when
> its pid is reused.
> * [MESOS-8411] - Killing a queued task can lead to the command executor
> never terminating.
> * [MESOS-8416] - CHECK failure if trying to recover nested containers but
> the framework checkpointing is not enabled.
> * [MESOS-8468] - `LAUNCH_GROUP` failure tears down the default executor.
> * [MESOS-8488] - Docker bug can cause unkillable tasks.
> * [MESOS-8510] - URI disk profile adaptor does not consider plugin type for
> a profile.
> * [MESOS-8536] - Pending offer operations on resource provider resources
> not properly accounted for in allocator.
> * [MESOS-8550] - Bug in `Master::detected()` leads to coredump in
> `MasterZooKeeperTest.MasterInfoAddress`.
> * [MESOS-8552] - CGROUPS_ROOT_PidNamespaceForward and
> CGROUPS_ROOT_PidNamespaceBackward tests fail.
> * [MESOS-8565] - Persistent volumes are not visible in Mesos UI when
> launching a pod using default executor.
> * [MESOS-8569] - Allow newline characters when decoding base64 strings in
> stout.
> * [MESOS-8574] - Docker executor makes no progress when 'docker inspect'
> hangs.
> * [MESOS-8575] - Improve discard handling for 'Docker::stop' and
> 'Docker::pull'.
> * [MESOS-8576] - Improve discard handling of 'Docker::inspect()'.
> * [MESOS-8577] - Destroy nested container if
> `LAUNCH_NESTED_CONTAINER_SESSION` fails.
> * [MESOS-8594] - Mesos master stack overflow in libprocess socket send
> loop.
> * [MESOS-8598] - Allow empty resource provider selector in
> `UriDiskProfileAdaptor`.
> * [MESOS-8601] - Master crashes during slave reregistration after failover.
> * [MESOS-8604] - Quota headroom tracking may be incorrect in the presence
> of hierarchical reservation.
> * [MESOS-8605] - Terminal task status update will not send if 'docker
> inspect' is hung.
> * [MESOS-8619] - Docker on Windows uses `USERPROFILE` instead of `HOME` for
> credentials.
> * [MESOS-8624] - Valid tasks may be explicitly dropped by agent due to race
> conditions.
> * [MESOS-8631] - Agent should be able to start a task with every CPU on a
> Windows machine.
> * [MESOS-8641] - Event stream could send heartbeat before subscribed.
> * [MESOS-8646] - Agent should be able to resolve file names on open files.
> * [MESOS-8651] - Potential memory leaks in the `volume/sandbox_path`
> isolator.
> * [MESOS-8741] - `Add` to sequence will not run if it races with sequence
> destruction.
> * [MESOS-8742] - Agent resource provider config API calls should be
> idempotent.
> * [MESOS-8786] - CgroupIsolatorProcess accesses subsystem processes
> directly.
> * [MESOS-8787] - RP-related API should be experimental.
> * [MESOS-8876] - Normal exit of Docker container using rexray volume
> results in TASK_FAILED.
> * [MESOS-8881] - Enable epoll backend in libevent integration.
> * [MESOS-8885] - Disable libevent debug mode.
>
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.5.1-rc1
> 
> 
>
> The candidate for Mesos 1.5.1 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.5.1-rc1/mesos-1.5.1.tar.gz
>
> The tag to be voted on is 1.5.1-rc1:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.1-rc1
>
> The SHA512 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.5.1-rc1/
> mesos-1.5.1.tar.gz.sha512
>
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.5.1-rc1/
> mesos-1.5.1.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1224
>
> Please vote on releasing this package as Apache Mesos 1.5.1!
>
> The vote is open until Wed May 16 12:31:02 PDT 2018 and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.5.1
> [ ] -1 Do not release this package because ...
>
> Thanks,
> Gilbert
>

Re: Mesos Roles | Min or Max ?

[Benjamin Mahler]

Currently a role either has no guarantee and no limit, or a guarantee and
limit set to the same amount of resources. The work is underway to allow
setting limit distinct from guarantee:

https://issues.apache.org/jira/browse/MESOS-8068

On Mon, May 21, 2018 at 4:17 PM Ken Sipe  wrote:

> Hey Trevor!
>
> Quota in mesos is used for both… it is a guarantee of offers up to a Max
> and it defines the Max.
>
> Ken
>
> On May 21, 2018, at 4:34 PM, Trevor Powell  wrote:
>
> Hello everyone,
>
> Reading up on http://mesos.apache.org/documentation/latest/roles/ and
> http://mesos.apache.org/documentation/latest/quota/
> Its not clear to me if there is a way to control how much a role is
> allowed to use a cluster. The MAX.  I think roles and quotas are more for
> minimum guarantees of resources??
>
> —
>
> 
>
> *Trevor Alexander Powell*
> Product Owner, Release+Platform Engineering
> 7575 Gateway Blvd. Newark, CA 94560
> 
> M: +1.650.325.7467
>
> https://github.com/tpowell-rms
> https://www.linkedin.com/in/trevorapowell
> http://www.rms.com
>
>

Re: Operator ReadFile API

[Benjamin Mahler]

Yes, it's base64 encoded.

The protobuf schema defines this field of type "bytes":
https://github.com/apache/mesos/blob/1.5.0/include/mesos/v1/agent/agent.proto#L460

When converted to JSON, this follows the standard protobuf -> JSON
conversion by converting "bytes" fields into base64 encoded strings:

https://developers.google.com/protocol-buffers/docs/proto3#json

Ben

On Sat, May 5, 2018 at 3:42 AM Pascal Gillet  wrote:

> Hi All,
>
> I try to read the stderr/stdout files from the Mesos sandbox with the
> READ_FILE call in the Operator HTTP API.
>
> I get a well-formatted json response as follows:
>
> {
>'type': 'READ_FILE',
>'read_file': {
>   'data': 'SDFtyfytgh4TR67RFY8',
>   'size': 19
>}
> }
>
> But I'm just getting a sequence of letters and numbers that has nothing to
> do with the file contents.
>
> Is the returned data encoded or compressed?
>
> Thanks,
>
> Pascal GILLET
>
>

Re: mesos-slave Failed to initialize: Failed to bind on 0.0.0.0:0: Address already in use: Address already in use [98]

[Benjamin Mahler]

>From the man page for bind:

*EADDRINUSE*
  (Internet domain sockets) The port number was specified as
  zero in the socket address structure, but, upon attempting to
  bind to an ephemeral port, it was determined that all port
  numbers in the ephemeral port range are currently in use.  See
  the discussion of */proc/sys/net/ipv4/ip_local_port_range*
  ip(7) .


On Thu, May 3, 2018 at 10:11 AM Srikanth Viswanathan 
wrote:

> fwiw, I've seen this type of error in the past when the system runs out of
> ephemeral ports. Not saying this definitely the same issue, but I suggest
> checking to see if you have ephemeral ports available.
>
> On Thu, May 3, 2018 at 8:57 AM, Zhitao Li  wrote:
>
>> Can you paste the command line of how you started the Mesos agent process?
>>
>> On Wed, May 2, 2018 at 9:21 PM, Luke Adolph  wrote:
>>
>>> Hi all：
>>> When mesos slave run task, the stderr file shows
>>> I0503 04:01:20.488590  9110 logging.cpp:188] INFO level logging started!
>>> I0503 04:01:20.489073  9110 fetcher.cpp:424] Fetcher Info:
>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/2bcc032f-950b-4c36-bff4-b5552c193dc9-S1\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"file:\/\/\/etc\/.dockercfg"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/2bcc032f-950b-4c36-bff4-b5552c193dc9-S1\/docker\/links\/b4eabcbb-5769-49f0-9324-b25c3cda8b8c","user":"root"}
>>> I0503 04:01:20.491297  9110 fetcher.cpp:379] Fetching URI
>>> 'file:///etc/.dockercfg'
>>> I0503 04:01:20.491325  9110 fetcher.cpp:250] Fetching directly into the
>>> sandbox directory
>>> I0503 04:01:20.491348  9110 fetcher.cpp:187] Fetching URI
>>> 'file:///etc/.dockercfg'
>>> I0503 04:01:20.491367  9110 fetcher.cpp:167] Copying resource with
>>> command:cp '/etc/.dockercfg'
>>> '/tmp/mesos/slaves/2bcc032f-950b-4c36-bff4-b5552c193dc9-S1/docker/links/b4eabcbb-5769-49f0-9324-b25c3cda8b8c/.dockercfg'
>>> W0503 04:01:20.495400  9110 fetcher.cpp:272] Copying instead of
>>> extracting resource from URI with 'extract' flag, because it does not seem
>>> to be an archive: file:///etc/.dockercfg
>>> I0503 04:01:20.495728  9110 fetcher.cpp:456] Fetched
>>> 'file:///etc/.dockercfg' to
>>> '/tmp/mesos/slaves/2bcc032f-950b-4c36-bff4-b5552c193dc9-S1/docker/links/b4eabcbb-5769-49f0-9324-b25c3cda8b8c/.dockercfg'
>>> F0503 04:01:21.990416  9202 process.cpp:889] Failed to initialize:
>>> Failed to bind on 0.0.0.0:0: Address already in use: Address already in
>>> use [98]
>>> *** Check failure stack trace: ***
>>> @ 0x7f95fc6ef86d  google::LogMessage::Fail()
>>> @ 0x7f95fc6f169d  google::LogMessage::SendToLog()
>>> @ 0x7f95fc6ef45c  google::LogMessage::Flush()
>>> @ 0x7f95fc6ef669  google::LogMessage::~LogMessage()
>>> @ 0x7f95fc6f05d2  google::ErrnoLogMessage::~ErrnoLogMessage()
>>> @ 0x7f95fc6955d9  process::initialize()
>>> @ 0x7f95fc696be2  process::ProcessBase::ProcessBase()
>>> @   0x430e9a
>>> mesos::internal::docker::DockerExecutorProcess::DockerExecutorProcess()
>>> @   0x41916b  main
>>> @ 0x7f95fa60ff45  (unknown)
>>> @   0x419c77  (unknown)
>>>
>>> When mesos slave initialize, it runs into "Failed to bind on 0.0.0.0:0:
>>> Address already in use", I run `netstat -nlp`, But there is no port "0" is
>>> used, full output is
>>> root@10:~# netstat -nlp
>>> Active Internet connections (only servers)
>>> Proto Recv-Q Send-Q Local Address   Foreign Address
>>>  State   PID/Program name
>>> tcp0  0 0.0.0.0:22  0.0.0.0:*
>>>  LISTEN  1153/sshd
>>> tcp0  0 0.0.0.0:37786   0.0.0.0:*
>>>  LISTEN  20042/mesos-docker-
>>> tcp0  0 0.0.0.0:50510.0.0.0:*
>>>  LISTEN  12701/mesos-slave
>>> tcp0  0 0.0.0.0:37084   0.0.0.0:*
>>>  LISTEN  19765/mesos-docker-
>>> tcp0  0 0.0.0.0:24220   0.0.0.0:*
>>>  LISTEN  28584/ruby
>>> tcp0  0 0.0.0.0:87650.0.0.0:*
>>>  LISTEN  28353/nginx
>>> tcp0  0 0.0.0.0:24224   0.0.0.0:*
>>>  LISTEN  28584/ruby
>>> tcp0  0 127.0.0.1:24225 0.0.0.0:*
>>>  LISTEN  28584/ruby
>>> tcp0  0 0.0.0.0:46690   0.0.0.0:*
>>>  LISTEN  28932/mesos-docker-
>>> tcp0  0 0.0.0.0:42437   0.0.0.0:*
>>>  LISTEN  32184/mesos-docker-
>>> tcp0  0 0.0.0.0:34695   0.0.0.0:*
>>>  LISTEN  25862/mesos-docker-
>>> tcp0  0 0.0.0.0:37039   0.0.0.0:*
>>>  LISTEN  21273/mesos-docker-
>>> tcp0  0 0.0.0.0:46001   0.0.0.0:*
>>>  LISTEN  710/mesos-docker-ex
>>> tcp6   0  0 :::31765:::*
>>> LISTEN  20160/docker-proxy
>>> tcp6   0  

Re: Reason of cascaded kill in a group

[Benjamin Mahler]

Are you saying that there was no reason previously, and there would be a
reason after the change? If so, adding a reason where one did not exist is
safe from a backwards compatibility perspective.

On Mon, Apr 9, 2018 at 10:32 AM, Zhitao Li  wrote:

> Hi,
>
> We are considering adding a new reason to StatusUpdate::Reason
> ,
> to reflect the case when a task in a task group is killed cascaded:
>
> Currently, if a task fails in a task group, other active tasks in the same
> group will see *TASK_KILLED* without any custom reason. We would like to
> provide a custom reason like *REASON_TASK_GROUP_KILLED* to distinguish
> whether the task is killed upon request of scheduler or upon a cascaded
> failure.
>
>
> Question to framework maintainer: does any framework depends the value of
> this reason? If not, we probably can just change the reason without a
> opt-in mechanism from framework (i.e, a new framework capability).
>
> Please let me know if your framework as such a dependency.
>
> Thanks!
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Troubleshooting Mesos SSL setup

[Benjamin Mahler]

Are there bugs here? Is there anything that mesos could have logged /
handled better?

On Fri, Mar 16, 2018 at 11:46 AM, Renan DelValle 
wrote:

> Follow up,  we weren't able to get our wildcard certificate working but we
> did get it to work when we used a certificate for a single hostname.
>
> Also our hostname was too long (over 64 bytes).
>
> Hope that helps someone else who runs into this issue.
>
> -Renan
>
> On Fri, Mar 16, 2018 at 10:36 AM, Renan DelValle  > wrote:
>
>> Hi all,
>>
>> We're trying to set up Mesos with SSL. We've compiled Mesos with SSL
>> support and deployed it to the right boxes.
>>
>> Unfortunately, after setting up all the correct environmental variables,
>> we get the following error:
>>
>> I0315 17:48:30.54186520 libevent_ssl_socket.cpp:1105] Could not
>>> determine hostname of peer: Unknown error
>>> I0315 17:48:30.54193720 libevent_ssl_socket.cpp:1120] Failed accept,
>>> verification error: Cannot verify peer certificate: peer hostname unknown
>>> * GnuTLS recv error (-110): The TLS connection was non-properly
>>> terminated.
>>> * Closing connection 0
>>> curl: (56) GnuTLS recv error (-110): The TLS connection was non-properly
>>> terminated.
>>
>>
>> Any chance someone knows what these errors mean and how we can fix the
>> underlying issue?
>>
>> Thanks!
>>
>> -Renan
>>
>
>

Re: Support deadline for tasks

[Benjamin Mahler]

Ah, I was more curious about why they need to be killed after a timeout.
E.g. After a particular deadline the work is useless (in Zhitao's case).

On Fri, Mar 23, 2018 at 6:22 PM Sagar Sadashiv Patwardhan <sag...@yelp.com>
wrote:

> Hi Benjamin,
> We have a few tasks that should be killed after
> some timeout. We currently have some logic in our scheduler to kill these
> tasks. Would be nice to delegate this to the executor.
>
> - Sagar
>
> On Fri, Mar 23, 2018 at 3:29 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
> > Sagar, could you share your use case? Or is it exactly the same as
> > Zhitao's?
> >
> > On Fri, Mar 23, 2018 at 3:15 PM, Sagar Sadashiv Patwardhan <
> > sag...@yelp.com>
> > wrote:
> >
> > > +1
> > >
> > > This will be useful for us(Yelp) as well.
> > >
> > > On Fri, Mar 23, 2018 at 1:31 PM, Benjamin Mahler <bmah...@apache.org>
> > > wrote:
> > >
> > > > Also, it's advantageous for mesos to be aware of a hard deadline when
> > it
> > > > comes to resource allocation. We know that some resources will free
> up
> > > and
> > > > can make better decisions when it comes to pre-emption, for example.
> > > > Currently, mesos doesn't know if a task will run forever or will run
> to
> > > > completion.
> > > >
> > > > On Fri, Mar 23, 2018 at 10:07 AM, James Peach <jpe...@apache.org>
> > wrote:
> > > >
> > > > >
> > > > >
> > > > > > On Mar 23, 2018, at 9:57 AM, Renan DelValle <
> > > renanidelva...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Zhitao,
> > > > > >
> > > > > > Since this is something that could potentially be handled by the
> > > > > executor and/or framework, I was wondering if you could speak to
> the
> > > > > advantages of making this a TaskInfo primitive vs having the
> executor
> > > (or
> > > > > even the framework) handle it.
> > > > >
> > > > > There's some discussion around this on https://issues.apache.org/
> > > > > jira/browse/MESOS-8725.
> > > > >
> > > > > My take is that delegating too much to the scheduler makes
> schedulers
> > > > > harder to write and exacerbates the complexity of the system. If 4
> > > > > different schedulers implement this feature, operators are likely
> to
> > > need
> > > > > to understand 4 different ways of doing the same thing, which would
> > be
> > > > > unfortunate.
> > > > >
> > > > > J
> > > >
> > >
> >
>

Re: Support deadline for tasks

[Benjamin Mahler]

Sagar, could you share your use case? Or is it exactly the same as Zhitao's?

On Fri, Mar 23, 2018 at 3:15 PM, Sagar Sadashiv Patwardhan <sag...@yelp.com>
wrote:

> +1
>
> This will be useful for us(Yelp) as well.
>
> On Fri, Mar 23, 2018 at 1:31 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
> > Also, it's advantageous for mesos to be aware of a hard deadline when it
> > comes to resource allocation. We know that some resources will free up
> and
> > can make better decisions when it comes to pre-emption, for example.
> > Currently, mesos doesn't know if a task will run forever or will run to
> > completion.
> >
> > On Fri, Mar 23, 2018 at 10:07 AM, James Peach <jpe...@apache.org> wrote:
> >
> > >
> > >
> > > > On Mar 23, 2018, at 9:57 AM, Renan DelValle <
> renanidelva...@gmail.com>
> > > wrote:
> > > >
> > > > Hi Zhitao,
> > > >
> > > > Since this is something that could potentially be handled by the
> > > executor and/or framework, I was wondering if you could speak to the
> > > advantages of making this a TaskInfo primitive vs having the executor
> (or
> > > even the framework) handle it.
> > >
> > > There's some discussion around this on https://issues.apache.org/
> > > jira/browse/MESOS-8725.
> > >
> > > My take is that delegating too much to the scheduler makes schedulers
> > > harder to write and exacerbates the complexity of the system. If 4
> > > different schedulers implement this feature, operators are likely to
> need
> > > to understand 4 different ways of doing the same thing, which would be
> > > unfortunate.
> > >
> > > J
> >
>

Re: Support deadline for tasks

[Benjamin Mahler]

Also, it's advantageous for mesos to be aware of a hard deadline when it
comes to resource allocation. We know that some resources will free up and
can make better decisions when it comes to pre-emption, for example.
Currently, mesos doesn't know if a task will run forever or will run to
completion.

On Fri, Mar 23, 2018 at 10:07 AM, James Peach  wrote:

>
>
> > On Mar 23, 2018, at 9:57 AM, Renan DelValle 
> wrote:
> >
> > Hi Zhitao,
> >
> > Since this is something that could potentially be handled by the
> executor and/or framework, I was wondering if you could speak to the
> advantages of making this a TaskInfo primitive vs having the executor (or
> even the framework) handle it.
>
> There's some discussion around this on https://issues.apache.org/
> jira/browse/MESOS-8725.
>
> My take is that delegating too much to the scheduler makes schedulers
> harder to write and exacerbates the complexity of the system. If 4
> different schedulers implement this feature, operators are likely to need
> to understand 4 different ways of doing the same thing, which would be
> unfortunate.
>
> J

Re: Mesos scalability

[Benjamin Mahler]

Hi Karan,

Only one master can be elected leader in the current architecture. It's
unlikely we're at a point where we need to balance work across masters to
push scalability further. That comes with a lot of complexity, and we still
have a lot of room for performance improvements on a single leader
architecture.

There are successful clusters sized beyond 35k machines as well, so
performance generally depend on the characteristics of the workloads. I
believe folks running clusters this large generally use a high core count
server (e.g. 24 cores), as to whether this is necessary is not clear to me
and again depends on the workloads, but certainly the master will continue
to improve its ability to leverage more cores for better performance.

In terms of the performance issues you've experienced, if you could provide
some data (e.g. flamegraphs of a backlogged master) that would help us
start to get a better sense of what's happening.

Ben

On Thu, Mar 22, 2018 at 2:26 PM, Karan Pradhan 
wrote:

> Hi All,
>
> I had the following questions:
> 1.
> I was wondering if it is possible to have multiple Mesos masters as
> elected masters in a Mesos cluster so that the load can be balanced amongst
> the masters. Is there a way to achieve this?
> In general, can there be a load balancer for the Mesos masters?
>
> 2.
> I have seen spikes in the Mesos event queues while running spark SQL
> workloads with multiple stages. So I was wondering what is a better way to
> handle these scalability issues. I noticed that compute intensive machines
> were able to deal with those workloads better. Is there a particular
> hardware requirement or requirement for the number of masters for scaling a
> Mesos cluster horizontally? After reading success stories which mention
> that Mesos is deployed for ~10K machines, I was curious about the hardware
> used and the number of masters in this case.
>
> It would be awesome if I could get some insight into these questions.
>
> Thanks,
> Karan
>
>

Re: Mesos on OS X

[Benjamin Mahler]

MacOS is a supported platform, you can see the supported versions here:
http://mesos.apache.org/documentation/latest/building/

The containerization maintainers could probably chime in to elaborate on
the isolation caveats. For example, you won't have many of the resource
isolators available and the launcher cannot prevent processes from
"escaping" from the "container".

On Wed, Mar 21, 2018 at 11:37 AM Ken Sipe  wrote:

> I don’t have long running experience but I would expect it to work fine…
> the thing to be aware of is that under OSX there are no cgroup constraints…
>  you also may want to review the APPLE difference:
> https://github.com/apache/mesos/search?utf8=%E2%9C%93=__APPLE__=
> 
>
> Ken
>
>
> On Mar 21, 2018, at 1:25 PM, Sunil Shah  wrote:
>
> Hey all,
>
> We're contemplating setting up a small OS X Mesos cluster for running iOS
> tests. I know Mesos technically builds on Macs, but has anyone ever had
> experience with a long running cluster on OS X? Is it possible?
> Recommended? Not recommended?
>
> Thanks,
>
> Sunil
>
>
>

Re: 答复: 答复: Status update: task 1 is in state TASK_ERROR

[Benjamin Mahler]

What kind of tasks are you trying to run?

If you want to run commands or containers, you can just use the built-in
DEFAULT executor:
https://github.com/apache/mesos/blob/1.5.0/include/mesos/v1/mesos.proto#L713-L725

If you need a custom executor because your tasks are not commands or
containers, then you can implement your own custom executor:
https://github.com/apache/mesos/blob/1.5.0/include/mesos/v1/mesos.proto#L727-L730

In the latter case, you will have to implement your own executor or use an
existing third party executor. If implementing your own, you need to speak
the v1 protocol to the agent. We maintain a listing of known executor API
libraries here:
http://mesos.apache.org/documentation/latest/api-client-libraries/#executor-api

On Thu, Mar 15, 2018 at 2:32 AM, 罗 辉 <luo...@zetyun.com> wrote:

> Hi guys:
>
> For more info, my framework app’s log and master/agent logs are attached.
>
> My app fails as the end of log described:
>
> The message of current task is :Executor did not register within 1mins
>
> Status update: task 1 is in state TASK_FAILED
>
> Aborting because task 1 is in unexpected state TASK_FAILED with reason
> 'REASON_EXECUTOR_REGISTRATION_TIMEOUT' from source 'SOURCE_AGENT' with
> message 'Executor did not register within 1mins'
>
>
>
> My opinion about this failure:
>
> 1.I guess there should be an V1 version executor class , with a register
> method to register the executor onto the agent?
>
> 2.I studied V0’s executor implementation and tried to implement a V1
> version executor ,which supposed to extend from executor interface, and
> implement the abstract methods including register, reregister and etc.
> However I didn’t find the V1 executor interface java API. Does that mean I
> am in the wrong direction？
>
>
>
> In one word, any ideas about the REASON_EXECUTOR_REGISTRATION_TIMEOUT
> failure?
>
>
>
> San
>
>
>
> *发件人:* 罗 辉 <luo...@zetyun.com>
> *发送时间:* 2018年3月14日 15:29
> *收件人:* user <user@mesos.apache.org>
> *主题:* 答复: 答复: Status update: task 1 is in state TASK_ERROR
>
>
>
> Thanks Benjamin,
>
> I tried to understand the missing reservation metadata and look up
> relative docs about resource reservation, however i didn't find to much
> document about it.
>
> I solved this problem by adding a method like below in my scheduler:
>
>   def luanchtask(offer: Offer, task: TaskInfo): Call = {
> Call.newBuilder()
>   .setFrameworkId(frameworkId)
>   .setType(Call.Type.ACCEPT)
>   .setAccept(
> Call.Accept.newBuilder()
>   .addOfferIds(offer.getId)
>   .addOperations(
> Offer.Operation.newBuilder()
>   .setType(Offer.Operation.Type.LAUNCH)
>   .setLaunch(
> Offer.Operation.Launch.newBuilder()
>   .addTaskInfos(task.build()
>   }
>
>
>
> And after that I met another problem: my task is always in staging, and
> terminates after 1min due to timeout. I think there are many mini process
> in a scheduler app including callbacks, such as connect, register, get
> offers list,accpet offer and etc. Is there a detail programming guide in V1
> framework developing?
>
>
>
> Thank you.
>
>
>
>
>
> San
>
>
> --
>
> *发件人**:* Benjamin Mahler <bmah...@apache.org>
> *发送时间**:* 2018年3月10日 9:00:55
> *收件人**:* user
> *主题**:* Re: 答复: Status update: task 1 is in state TASK_ERROR
>
>
>
> The message clarifies it, the task+executor have some unreserved
> resources:
>
> cpus(allocated: controller):6; mem(allocated: controller):8000
>
>
>
> But the resources offered were reserved:
>
> cpus(allocated: controller)(reservations: [(STATIC,controller)]):6;
> mem(allocated: controller)(reservations: [(STATIC,controller)]):8000; +
> disk + ports
>
>
>
> The scheduler needs to provide resources that are contained in the offer,
> in this case it needs to include the missing reservation metadata.
>
>
>
> On Thu, Mar 8, 2018 at 6:57 PM, 罗 辉 <luo...@zetyun.com> wrote:
>
> yes,I modified my code like below:
>
>   def acknowledgeTaskMessage(taskStatus: TaskStatus): String = {
> taskStatus.getMessage
>   }
>
> def update(mesos: Mesos, status: TaskStatus) = {
> val message = acknowledgeTaskMessage(status)
> println("The message of current task is :" + message)
> println("Status update: task " + status.getTaskId().getValue() + " is
> in state " + status.getState().getValueDescriptor().getName())
>
>
> ..
>
>
>
> And I got below log as attched file line 231:
>
> 231 Received an UPDATE event
> 232 The mess

Re: Welcome Zhitao Li as Mesos Committer and PMC Member

[Benjamin Mahler]

Welcome Zhitao! Thanks for your contributions so far

On Mon, Mar 12, 2018 at 2:02 PM, Gilbert Song  wrote:

> Hi,
>
> I am excited to announce that the PMC has voted Zhitao Li as a new
> committer and member of PMC for the Apache Mesos project. Please join me to
> congratulate Zhitao!
>
> Zhitao has been an active contributor to Mesos for one and a half years.
> His main contributions include:
>
>- Designed and implemented Container Image Garbage Collection (
>MESOS-4945 );
>- Designed and implemented part of the HTTP Operator API (MESOS-6007
>);
>- Reported and fixed a lot of bugs
>
> 
>.
>
> Zhitao spares no effort to improve the project quality and to propose
> ideas. Thank you Zhitao for all contributions!
>
> Here is his committer candidate checklist for your perusal:
> https://docs.google.com/document/d/1HGz7iBdo1Q9z9c8fNRgNNLnj0XQ_
> PhDhjXLAfOx139s/
>
> Congrats Zhitao!
>
> Cheers,
> Gilbert
>

Re: Welcome Chun-Hung Hsiao as Mesos Committer and PMC Member

[Benjamin Mahler]

Welcome Chun! It's been great discussing things with you so far and thanks
for the all the hard work!

On Sat, Mar 10, 2018 at 9:14 PM, Jie Yu  wrote:

> Hi,
>
> I am happy to announce that the PMC has voted Chun-Hung Hsiao as a new
> committer and member of PMC for the Apache Mesos project. Please join me
> to congratulate him!
>
> Chun has been an active contributor for the past year. His main
> contributions to the project include:
> * Designed and implemented gRPC client support to libprocess (MESOS-7749)
> * Designed and implemented Storage Local Resource Provider (MESOS-7235,
> MESOS-8374)
> * Implemented part of the CSI support (MESOS-7235, MESOS-8374)
>
> Chun is friendly and humble, but also intelligent, insightful, and
> opinionated. I am confident that he will be a great addition to our
> committer pool. Thanks Chun for all your contributions to the project so
> far!
>
> His committer checklist can be found here:
> https://docs.google.com/document/d/1FjroAvjGa5NdP29zM7-
> 2eg6tLPAzQRMUmCorytdEI_U/edit?usp=sharing
>
> - Jie
>

Re: 答复: Status update: task 1 is in state TASK_ERROR

[Benjamin Mahler]

The message clarifies it, the task+executor have some unreserved resources:
cpus(allocated: controller):6; mem(allocated: controller):8000

But the resources offered were reserved:
cpus(allocated: controller)(reservations: [(STATIC,controller)]):6;
mem(allocated: controller)(reservations: [(STATIC,controller)]):8000; +
disk + ports

The scheduler needs to provide resources that are contained in the offer,
in this case it needs to include the missing reservation metadata.

On Thu, Mar 8, 2018 at 6:57 PM, 罗 辉 <luo...@zetyun.com> wrote:

> yes,I modified my code like below:
>
>   def acknowledgeTaskMessage(taskStatus: TaskStatus): String = {
> taskStatus.getMessage
>   }
> def update(mesos: Mesos, status: TaskStatus) = {
> val message = acknowledgeTaskMessage(status)
> println("The message of current task is :" + message)
> println("Status update: task " + status.getTaskId().getValue() + " is
> in state " + status.getState().getValueDescriptor().getName())
>
> ..
>
> And I got below log as attched file line 231:
> 231 Received an UPDATE event
> 232 The message of current task is :Total resources cpus(allocated:
> controller):6; mem(allocated: controller):8000 required by task and its
> executor is more than available cpus(allocated: controller)(reservations:
> [(STATIC,controller)]):6; mem(allocated: controller)(reservations:
> [(STATIC,controller)]):8000; disk(allocated: controller)(reservations:
> [(STATIC,controller)]):550264; ports(allocated:
> controller):[31000-32000]
> 233 Status update: task 1 is in state TASK_ERROR
>
>
>
> 罗辉
>
> 基础架构
> --
> *发件人:* Benjamin Mahler <bmah...@apache.org>
> *发送时间:* 2018年3月9日 9:24:37
> *收件人:* user
> *主题:* Re: Status update: task 1 is in state TASK_ERROR
>
> Can you log the message provided in the TaskStatus?
>
> https://github.com/apache/mesos/blob/1.5.0/include/mesos/v1/
> mesos.proto#L2424
>
> On Wed, Mar 7, 2018 at 11:23 PM, 罗 辉 <luo...@zetyun.com> wrote:
>
> Hi guys:
>
> I got a mesos test app, mostly likely
>
> https://github.com/apache/mesos/blob/master/src/java/src/org
> /apache/mesos/v1/scheduler/V1Mesos.java
>
> just to run a simple task "free -m". The app can not run the task
> successfully, always got a log info :
>
> Received an UPDATE event
> Status update: task 1 is in state TASK_ERROR
>
>
> I checked the logs , but no Errors  in the mesos-master.ERROR or
> mesos-agent.ERROR, only in mesos-master.INFO shows :
>
> W0307 17:55:28.180716 29438 validation.cpp:1298] Executor 'default' for
> task '1' uses less CPUs (None) than the minimum required (0.01). Please
> update your executor, as this will be mandatory in future releases.
> W0307 17:55:28.180766 29438 validation.cpp:1310] Executor 'default' for
> task '1' uses less memory (None) than the minimum required (32MB). Please
> update your executor, as this will be mandatory in future releases.
>   Following this log, I didn't find a way to set the executor's
> resource or similar code example
>
>   Why my little app always fails? Thanks for any ideas.
>
>
> San
>
>
>

Re: Status update: task 1 is in state TASK_ERROR

[Benjamin Mahler]

Can you log the message provided in the TaskStatus?

https://github.com/apache/mesos/blob/1.5.0/include/
mesos/v1/mesos.proto#L2424

On Wed, Mar 7, 2018 at 11:23 PM, 罗 辉  wrote:

> Hi guys:
>
> I got a mesos test app, mostly likely
>
> https://github.com/apache/mesos/blob/master/src/java/src/
> org/apache/mesos/v1/scheduler/V1Mesos.java
>
> just to run a simple task "free -m". The app can not run the task
> successfully, always got a log info :
>
> Received an UPDATE event
> Status update: task 1 is in state TASK_ERROR
>
>
> I checked the logs , but no Errors  in the mesos-master.ERROR or
> mesos-agent.ERROR, only in mesos-master.INFO shows :
>
> W0307 17:55:28.180716 29438 validation.cpp:1298] Executor 'default' for
> task '1' uses less CPUs (None) than the minimum required (0.01). Please
> update your executor, as this will be mandatory in future releases.
> W0307 17:55:28.180766 29438 validation.cpp:1310] Executor 'default' for
> task '1' uses less memory (None) than the minimum required (32MB). Please
> update your executor, as this will be mandatory in future releases.
>   Following this log, I didn't find a way to set the executor's
> resource or similar code example
>
>   Why my little app always fails? Thanks for any ideas.
>
>
> San
>

Re: Tasks may be explicitly dropped by agent in Mesos 1.5

[Benjamin Mahler]

Put another way, we currently don't guarantee in-order task delivery to the
executor. Due to the changes for MESOS-1720, one special case of task
re-ordering now leads to the re-ordered task being dropped (rather than
delivered out-of-order as before). Technically, this is strictly better.

However, we'd like to start guaranteeing in-order task delivery.

On Thu, Mar 1, 2018 at 2:56 PM, Meng Zhu  wrote:

> Hi all:
>
> TLDR: In Mesos 1.5, tasks may be explicitly dropped by the agent
> if all three conditions are met:
> (1) Several `LAUNCH_TASK` or `LAUNCH_GROUP` calls
>  use the same executor.
> (2) The executor currently does not exist on the agent.
> (3) Due to some race conditions, these tasks are trying to launch
> on the agent in a different order from their original launch order.
>
> In this case, tasks that are trying to launch on the agent
> before the first task in the original order will be explicitly dropped by
> the agent (TASK_DROPPED` or `TASK_LOST` will be sent)).
>
> This bug will be fixed in 1.5.1. It is tracked in
> https://issues.apache.org/jira/browse/MESOS-8624
>
> 
>
> In https://issues.apache.org/jira/browse/MESOS-1720, we introduced an
> ordering dependency between two `LAUNCH`/`LAUNCH_GROUP`
> calls to a new executor. The master would specify that the first call is
> the
> one to launch a new executor through the `launch_executor` field in
> `RunTaskMessage`/`RunTaskGroupMessage`, and the second one should
> use the existing executor launched by the first one.
>
> On the agent side, running a task/task group goes through a series of
> continuations, one is `collect()` on the future that unschedule
> frameworks from
> being GC'ed:
> https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L2158
> another is `collect()` on task authorization:
> https://github.com/apache/mesos/blob/master/src/slave/slave.cpp#L2333
> Since these `collect()` calls run on individual actors, the futures of the
> `collect()` calls for two `LAUNCH`/`LAUNCH_GROUP` calls may return
> out-of-order, even if the futures these two `collect()` wait for are
> satisfied in
> order (which is true in these two cases).
>
> As a result, under some race conditions (probably under some heavy load
> conditions), tasks rely on the previous task to launch executor may
> get processed before the task that is supposed to launch the executor
> first, resulting in the tasks being explicitly dropped by the agent.
>
> -Meng
>
>
>

Re: is there any docs to show how to secure http(s) for masters

[Benjamin Mahler]

+Alexander

On Mon, Feb 19, 2018 at 11:00 AM Mclain, Warren 
wrote:

> I am not finding any documentation that tells you how to actually
> implement  the following on the mesos masters and agents.
>
>
>
> authenticate=true
>
> authenticate_http_readonly=true
>
> authenticate_http_readwrite=true
>
>
>
> there is a ton of “official” mesos docs but nothing tells you how to
> actually make it work.
>
>
>
> We are using the open source version and trying to secure the basic
> infrastructure so looking for anyone who has actually been able to make
> this work and how.
>
>
>
> We are running pretty much the latest versions of everything.
>
>
>
> If anyone has some pointers (other than Mesos.org docs), please contact me
> directly.
>
>
>
> Thanks.
>
>
>
> ___
>
> Warren McLain
>
> Enterprise Engineering Services
>
> IEI Foundation Engineering - Compute, Optum Technology
>
>  warren_mcl...@optum.com Office: 763-744-3107
>
>
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>

Re: http://mesos.apache.org/downloads/ is not up to date

[Benjamin Mahler]

Thanks for pointing this out Adam, I've added mpark who is the release
manager for 1.3.2.

On Tue, Feb 6, 2018 at 6:12 AM, Adam Cecile  wrote:

> Hi guys,
>
>
> Did you notice Mesos 1.3.2 is missing from the official download page ?
>
> http://mesos.apache.org/downloads/
>
>
> Regards, Adam.
>
>

Reminder: Design Doc for Mesos CLI Re-design

[Benjamin Mahler]

I've heard a lot of interest in there being investment in the mesos CLI.
For those that are interested, please take a look at the re-design doc and
share your feedback:

https://docs.google.com/document/d/1r6Iv4Efu8v8IBrcUTjgYkvZ32WVsc
gYqrD07OyIglsA/edit

Feel free to make comments in the doc, suggest commands that you think
would be useful, or share any high level feedback, questions, or concerns
here and Kevin / Armand can answer them.

Ben

Re: Questions about Pods and the Mesos Containerizer

[Benjamin Mahler]

If moving the conversation to slack, it would be great to post back to the
list with a summary!

On Mon, Jan 29, 2018 at 1:38 PM, Vinod Kone  wrote:

> Hi David,
>
> It's probably worth having a synchronous discussion around your proposed
> approach in our slack. I would like to understand if TASK_GROUP is the
> right primitive for your use case.
>
> On Mon, Jan 29, 2018 at 1:32 PM, David Morrison  wrote:
>
>>
>>
>> On Thu, Jan 25, 2018 at 5:49 PM, Gilbert Song 
>> wrote:
>>
>>>
-

Is it possible to allocate a separate IP address per container in a
pod?

 Right now nested containers share the network from their parent
>>> container (pod). Do we have a specific use case that we need containers
>>> inside of a taskgroup have different IP addresses?
>>>
>>
>> For our use case, we need to be able to launch a relatively large number
>> of containers inside a taskgroup that all listen on the same port (and the
>> port is not easily-changeable).  So we need to be able to assign different
>> IPs to the containers so they don't conflict.
>>
>> Cheers,
>> David
>>
>
>

Re: java driver/shutdown call

[Benjamin Mahler]

Can you tell us more about what the use case is? Why do you think it's more
robust?

On Tue, Jan 16, 2018 at 8:41 PM Mohit Jaggi <mohit.ja...@uber.com> wrote:

> I am trying to change Apache Aurora's code to call SHUTDOWN instead of
> KILL. SHUTDOWN seems to offer more robust termination than KILL.
>
> On Tue, Jan 16, 2018 at 6:40 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
>> Mohit, what are you trying to accomplish by going from KILL to SHUTDOWN?
>>
>> On Tue, Jan 16, 2018 at 5:15 PM, Joseph Wu <jos...@mesosphere.io> wrote:
>>
>>> If a framework launches tasks, then it will use an executor.  Mesos
>>> provides a "default" executor if the framework doesn't explicitly specify
>>> an executor.  (And the Shutdown call will work with that default executor.)
>>>
>>> On Tue, Jan 16, 2018 at 4:49 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>> wrote:
>>>
>>>> Gotcha. Another question: if a framework doesn't use executors, can it
>>>> still use the SHUTDOWN call?
>>>>
>>>> On Fri, Jan 12, 2018 at 2:37 PM, Anand Mazumdar <
>>>> mazumdar.an...@gmail.com> wrote:
>>>>
>>>>> Yes; It's a newer interface that still allows you to switch between
>>>>> the v1 (new) and the old API.
>>>>>
>>>>> -anand
>>>>>
>>>>> On Fri, Jan 12, 2018 at 3:28 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>>> wrote:
>>>>>
>>>>>> Are you suggesting
>>>>>>
>>>>>> *send(new Call(METHOD, Param1, ...)) *
>>>>>>
>>>>>> instead of
>>>>>>
>>>>>> *driver.method(Param1, )*
>>>>>>
>>>>>> *?*
>>>>>>
>>>>>> On Fri, Jan 12, 2018 at 10:59 AM, Anand Mazumdar <
>>>>>> mazumdar.an...@gmail.com> wrote:
>>>>>>
>>>>>>> Mohit,
>>>>>>>
>>>>>>> You can use the V1Mesos class that uses the v1 API internally
>>>>>>> allowing you to send the 'SHUTDOWN' call. We also have a V0Mesos class 
>>>>>>> that
>>>>>>> uses the old scheduler driver internally.
>>>>>>>
>>>>>>> -anand
>>>>>>>
>>>>>>> On Wed, Jan 10, 2018 at 2:53 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Vinod. Is there a V1SchedulerDriver.java file? I see
>>>>>>>> https://github.com/apache/mesos/tree/72752fc6deb8ebcbfbd5448dc599ef3774339d31/src/java/src/org/apache/mesos/v1/scheduler
>>>>>>>> but it does not have a V1 driver.
>>>>>>>>
>>>>>>>> On Fri, Jan 5, 2018 at 3:59 PM, Vinod Kone <vinodk...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> That's right. It is only available for v1 schedulers.
>>>>>>>>>
>>>>>>>>> On Fri, Jan 5, 2018 at 3:38 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Folks,
>>>>>>>>>> I am trying to change Apache Aurora's code to call SHUTDOWN
>>>>>>>>>> instead of KILL. However, it seems that the SchedulerDriver class in 
>>>>>>>>>> Mesos
>>>>>>>>>> does not have a shutdownExecutor() call.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/mesos/blob/72752fc6deb8ebcbfbd5448dc599ef3774339d31/src/java/src/org/apache/mesos/SchedulerDriver.java
>>>>>>>>>>
>>>>>>>>>> Mohit.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Anand Mazumdar
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Anand Mazumdar
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Mesos slave ID change after reboot

[Benjamin Mahler]

Yes, the agent used to check for the boot id having changed in order to
decide whether to try to recover.

On Wed, Jan 10, 2018 at 5:53 PM, Srikanth Viswanathan 
wrote:

> I am trying to understand under what cases the mesos slave ID changes in
> response to reboot.  I noticed this note at http://mesos.apache.org/
> documentation/latest/upgrades/#upgrading-from-1-3-x-to-1-4-x:
>
> Agent is now allowed to recover its agent ID post a host reboot. This
>> prevents the unnecessary discarding of agent ID by prior Mesos versions.
>> Notes about backwards compatibility:
>>
>>- In case the agent’s recovery runs into agent info mismatch which
>>may happen due to resource change associated with reboot, it’ll fall back
>>to recovering as a new agent (existing behavior).
>>
>>
>>- In other cases such as checkpointed resources (e.g. persistent
>>volumes) being incompatible with the agent’s resources the recovery will
>>still fail (existing behavior).
>>
>>
> I was wondering if the behavior prior to 1.3 is also similarly
> well-defined. Is the answer "Will always change after a reboot"?
>
> Thanks,
> Srikanth
>

Re: java driver/shutdown call

[Benjamin Mahler]

Mohit, what are you trying to accomplish by going from KILL to SHUTDOWN?

On Tue, Jan 16, 2018 at 5:15 PM, Joseph Wu  wrote:

> If a framework launches tasks, then it will use an executor.  Mesos
> provides a "default" executor if the framework doesn't explicitly specify
> an executor.  (And the Shutdown call will work with that default executor.)
>
> On Tue, Jan 16, 2018 at 4:49 PM, Mohit Jaggi  wrote:
>
>> Gotcha. Another question: if a framework doesn't use executors, can it
>> still use the SHUTDOWN call?
>>
>> On Fri, Jan 12, 2018 at 2:37 PM, Anand Mazumdar > > wrote:
>>
>>> Yes; It's a newer interface that still allows you to switch between the
>>> v1 (new) and the old API.
>>>
>>> -anand
>>>
>>> On Fri, Jan 12, 2018 at 3:28 PM, Mohit Jaggi 
>>> wrote:
>>>
 Are you suggesting

 *send(new Call(METHOD, Param1, ...)) *

 instead of

 *driver.method(Param1, )*

 *?*

 On Fri, Jan 12, 2018 at 10:59 AM, Anand Mazumdar <
 mazumdar.an...@gmail.com> wrote:

> Mohit,
>
> You can use the V1Mesos class that uses the v1 API internally allowing
> you to send the 'SHUTDOWN' call. We also have a V0Mesos class that uses 
> the
> old scheduler driver internally.
>
> -anand
>
> On Wed, Jan 10, 2018 at 2:53 PM, Mohit Jaggi 
> wrote:
>
>> Thanks Vinod. Is there a V1SchedulerDriver.java file? I see
>> https://github.com/apache/mesos/tree/72752fc6deb8ebcbfbd
>> 5448dc599ef3774339d31/src/java/src/org/apache/mesos/v1/scheduler but
>> it does not have a V1 driver.
>>
>> On Fri, Jan 5, 2018 at 3:59 PM, Vinod Kone 
>> wrote:
>>
>>> That's right. It is only available for v1 schedulers.
>>>
>>> On Fri, Jan 5, 2018 at 3:38 PM, Mohit Jaggi 
>>> wrote:
>>>
 Folks,
 I am trying to change Apache Aurora's code to call SHUTDOWN instead
 of KILL. However, it seems that the SchedulerDriver class in Mesos 
 does not
 have a shutdownExecutor() call.

 https://github.com/apache/mesos/blob/72752fc6deb8ebcbfbd5448
 dc599ef3774339d31/src/java/src/org/apache/mesos/SchedulerDri
 ver.java

 Mohit.

>>>
>>>
>>
>
>
> --
> Anand Mazumdar
>


>>>
>>>
>>> --
>>> Anand Mazumdar
>>>
>>
>>
>

Re: Duplicate task ID for same framework on different agents

[Benjamin Mahler]

It's a known issue:
https://issues.apache.org/jira/browse/MESOS-3070

Putting in place a protection mechanism sounds good, but is rather
complicated. See the comment in this ticket:
https://issues.apache.org/jira/browse/MESOS-6785

On Wed, Dec 20, 2017 at 8:26 PM, Zhitao Li  wrote:

> Hi all,
>
> We have seen a mesos master crash loop after a leader failover. After more
> investigation, it seems that a same task ID was managed to be created onto
> multiple Mesos agents in the cluster.
>
> One possible logical sequence which can lead to such problem:
>
> 1. Task T1 was launched to master M1 on agent A1 for framework F;
> 2. Master M1 failed over to M2;
> 3. Before A1 reregistered to M2, the same T1 was launched on to agent A2:
> M2 does not know previous T1 yet so it accepted it and sent to A2;
> 4. A1 reregistered: this probably crashed M2 (because same task cannot be
> added twice);
> 5. When M3 tries to come up after M2, it further crashes because both A1
> and A2 tried to add a T1 to the framework.
>
> (I only have logs to prove the last step right now)
>
> This happened on 1.4.0 masters.
>
> Although this is probably triggered by incorrect retry logic on framework
> side, I wonder whether Mesos master should do extra protection to prevent
> such issue to cause master crash loop. Some possible ideas are to instruct
> one of the agents carrying tasks w/ duplicate ID to terminate corresponding
> tasks, or just refuse to reregister such agents and instruct them to
> shutdown.
>
> I also filed MESOS-8353 
> to track this potential bug. Thanks!
>
>
> --
>
> Cheers,
>
> Zhitao Li
>

Re: Mesos 1.5.0 Release

[Benjamin Mahler]

Meng is working on https://issues.apache.org/jira/browse/MESOS-8352 and we
should land it tonight if not tomorrow. I can cherry pick if it's after
your cut, and worst case it can go in 1.5.1.

Have you guys gone over the unresolved items targeted for 1.5.0? I see a
lot of stuff, might be good to start adjusting / removing their target
versions to give folks a chance to respond on the ticket?

https://issues.apache.org/jira/issues/?jql=project%20%3D%20MESOS%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reviewable%2C%20Accepted)%20AND%20%22Target%20Version%2Fs%22%20%3D%201.5.0

For example, https://issues.apache.org/jira/browse/MESOS-8337 looks pretty
bad to me (master crash).

On Thu, Dec 21, 2017 at 7:00 PM, Jie Yu  wrote:

> Hi,
>
> We're about to cut 1.5.0-rc1 tomorrow. If you have any thing that needs to
> go into 1.5.0 that hasn't landed, please let me or Gilbert know asap.
> Thanks!
>
> - Jie
>
> On Fri, Dec 1, 2017 at 3:58 PM, Gilbert Song  wrote:
>
>> Folks,
>>
>> It is time for Mesos 1.5.0 release. I am the release manager.
>>
>> We plan to cut the rc1 in next couple weeks. Please start to wrap up
>> patches if you are contributing or shepherding any issue. If you expect
>> any
>> particular JIRA for this new release, please set *Target Version* as "
>> *1.5.0"* and mark it as "*Blocker*" priority.
>>
>> The dashboard for Mesos 1.5.0 will be posted in this thread soon.
>>
>> Cheers,
>> Gilbert
>>
>
>

Re: [VOTE] Release Apache Mesos 1.3.2 (rc1)

[Benjamin Mahler]

+1 (binding)

make check passes on macOS 10.13.2 with Apple LLVM version 9.0.0
(clang-900.0.39.2)

On Thu, Dec 7, 2017 at 2:44 PM, Michael Park  wrote:

> Hi all,
>
> Please vote on releasing the following candidate as Apache Mesos 1.3.2.
>
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_
> plain;f=CHANGELOG;hb=1.3.2-rc1
> 
> 
>
> The candidate for Mesos 1.3.2 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.3.2-rc1/mesos-1.3.2.tar.gz
>
> The tag to be voted on is 1.3.2-rc1:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.3.2-rc1
>
> The MD5 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.3.2-rc1/
> mesos-1.3.2.tar.gz.md5
>
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.3.2-rc1/
> mesos-1.3.2.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is up in Maven in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1220
>
> Please vote on releasing this package as Apache Mesos 1.3.2!
>
> The vote is open until Wed, Dec 13 and passes if a majority of at least 3
> +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 1.3.2
> [ ] -1 Do not release this package because ...
>
> Thanks,
>
> MPark
>

December Performance Working Group Report

[Benjamin Mahler]

The December performance working group report is published on the website
here:
http://mesos.apache.org/blog/performance-working-group-progress-report/

This report highlights the progress we've made recently in the performance
of master failover. Special thanks to Dmitry Zhuk, Michael Park and Yan Xu
for the work in this area.

I didn't get time to write up something for the libprocess improvements, so
I decided to split that out into its own blog post.

I tweeted the above link here if you'd like to help us share it:
https://twitter.com/bmahler/status/940325681312940033

Ben

Re: Resource allocation cycle in DRF for multiple frameworks

[Benjamin Mahler]

Q1: we randomly sort the agents, so the pseudo-code I showed is:

- for each agent:
+ for each agent in random_sort(agents):

Q2: It depends on which version you're running. We used to immediately
re-offer, but this was problematic since it kept going back to the same
framework when using a low timeout. Now, the current implementation won't
immediately re-offer it in an attempt to let it go to another framework
during the next allocation "cycle":

https://github.com/apache/mesos/blob/1.4.0/src/master/
allocator/mesos/hierarchical.cpp#L1202-L1213

Q3: We implement best-effort DRF to improve utilization. That is, we let a
role go above its fair share if the lower share roles do not want the
resources, and a role may have to wait for the resources to be released
before it can get its fair share (since we cannot revoke resources). So, we
increase utilization at the cost of no longer providing a guarantee that a
role can get its fair share without waiting! In the future, we will use
revocation to ensure a user is guaranteed to get their fair share without
having to wait.

On Tue, Dec 5, 2017 at 9:04 AM, bigggyan <biggg...@gmail.com> wrote:

> Hi Benjamin,
> Thanks for the clear explanation. This loop structure makes it clear to
> understand how resource allocation is actually happening inside mesos
> master allocation module. However I have few quires. I will try to ask
> questions to clarify them. My goal is to understand how DRF is implemented
> in Apache Mesos based on the DRF paper. I am doing this for an academic
> project to develop a custom framework.
> I am using few in-house frameworks along with Mesosphere Marathon and
> Chronos. I am using default role and no weigh to any frameworks and
> constraint. so  the loop becomes simpler.
>
> I understand that there exists no such cycle, but what I meant was the end
> of the outer loop when all the agents are allocated to frameworks.
>
> Q1: the loop "for each agent" : how one agent is being picked over other
> agents, to be assigned to a framework?
> Q2: now after all the agents are allocated to available frameworks, each
> framework can decide whether to use it or not. So the question is: what if
> a framework rejects a offer with 0 second filter duration, can it be
> offered to the same framework due to its low dominant share again ?  or is
> there any penalty that a rejected offer can not be immediately offered to
> the same framework?
>
> let me explain why this is important to know:
> User A may be using 80% of the share and user B is receiving the rest of
> the offers first, because of its low share, but rejecting offers due to no
> pending tasks to launch. Now according to DRF, master will always pick
>  user B first, and user A will not receive anything even though it has many
> tasks in the waiting queue.
>
> Q3: my observation is once a offers is declined or partially used by a
> framework, it immediately comes to to next available framework even though
> next frameworks share is higher than the previous one. Is that by
> implementation or I am getting something wrong here?
>
> Thanks
>
>
> On Mon, Dec 4, 2017 at 2:37 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
>> I don't think I understood the questions here, but let me add some
>> explanation and we can go from there.
>>
>> Mesos will use DRF to choose an ordering amongst the roles that are
>> actively interested in obtaining resources. Within a role, we currently use
>> DRF again to choose an ordering amongst the frameworks in that role. The
>> simplified pseudo-code looks something like this:
>>
>> for each agent:
>>   for each role in drf_sorted(roles):
>> for each framework subscribed to role in drf_sorted(frameworks):
>>   if framework already filtered these resources:
>> continue
>>   else
>> allocate to framework
>>
>> There is no strong concept of a "cycle" as you were referring to, that
>> is, mesos will not remember which offers were sent out during which time we
>> ran this overall loop. Currently, when resources are offered, as far as the
>> allocator is concerned, they are considered allocated to that role and
>> framework.
>>
>> Mesos provides an --offer_timeout flag on the master after which the
>> offer will be rescinded.
>>
>> If you could share a little more about what you're trying to accomplish
>> in your particular use case we could advise on how best to set things up.
>>
>> On Thu, Nov 30, 2017 at 1:05 PM, bigggyan <biggg...@gmail.com> wrote:
>>
>>> Hello
>>> My understanding is, during a single DRF cycle mesos master will not
>>> offer same framework twice. I belie

Re: Resource allocation cycle in DRF for multiple frameworks

[Benjamin Mahler]

I don't think I understood the questions here, but let me add some
explanation and we can go from there.

Mesos will use DRF to choose an ordering amongst the roles that are
actively interested in obtaining resources. Within a role, we currently use
DRF again to choose an ordering amongst the frameworks in that role. The
simplified pseudo-code looks something like this:

for each agent:
  for each role in drf_sorted(roles):
for each framework subscribed to role in drf_sorted(frameworks):
  if framework already filtered these resources:
continue
  else
allocate to framework

There is no strong concept of a "cycle" as you were referring to, that is,
mesos will not remember which offers were sent out during which time we ran
this overall loop. Currently, when resources are offered, as far as the
allocator is concerned, they are considered allocated to that role and
framework.

Mesos provides an --offer_timeout flag on the master after which the offer
will be rescinded.

If you could share a little more about what you're trying to accomplish in
your particular use case we could advise on how best to set things up.

On Thu, Nov 30, 2017 at 1:05 PM, bigggyan  wrote:

> Hello
> My understanding is, during a single DRF cycle mesos master will not offer
> same framework twice. I believe, if a framework rejects or left over offer
> after partial use will come to next eligible framework.
> Now the question is if one framework takes longer time to make decision,
> will the same DRF allocation cycle will stay alive to allocate rest of the
> resources to other users or master will start a new cycle?
> Is there any allocation cycle expiry period? I am using multiple in-house
> frameworks with same role and same weight with no quota set. Will
> appreciate your help to understand the resource allocation.
>
> Thanks
> Bigggyan
>

Re: Documentation for Mesos On windows

[Benjamin Mahler]

+Andrew

On Tue, Nov 28, 2017 at 5:41 PM, sweta Das  wrote:

> Hi
>
> Is there any other documentation than the one on mesos site
> http://mesos.apache.org/documentation/latest/windows/
>
> I was able to build mesos on AWS on an windows 2016 server. But I am not
> able to find any docs for starting the mesos master on windows?
> I understand that this is not recommended as of now, but for testing can
> anyone tell how can i start a master on windows?
>
> Sent from my iPhone
>

Re: [VOTE] Release Apache Mesos 1.2.3 (rc1)

[Benjamin Mahler]

+1 (binding)

make check on macOS 10.13.1

On Wed, Nov 29, 2017 at 9:17 PM, Adam Bordelon  wrote:

> +1 (binding)
>
> Passed all tests in DC/OS integration CI, with a bump to 1.2.x at f8706e5,
> just one changelog update before 1.2.3-rc1.
> https://github.com/dcos/dcos/pull/2104#pullrequestreview-79333676
>
> On Tue, Nov 21, 2017 at 2:46 PM, Vinod Kone  wrote:
>
>> +1 (binding)
>>
>> Tested on ASF CI. The failures are due to 2 issues 1) perf core dump
>>  which was fixed in
>> 1.5.0 and 2) flaky oversubscription test
>>  also fixed in 1.5.0.
>>
>> *Revision*: 7559c9352c78912526820f6222ed2b17ad3b19cf
>>
>>- refs/tags/1.2.3-rc1
>>
>> Configuration Matrix gcc clang
>> centos:7 --verbose --enable-libevent --enable-ssl autotools
>> [image: Success]
>> 
>> [image: Not run]
>> cmake
>> [image: Success]
>> 
>> [image: Not run]
>> --verbose autotools
>> [image: Success]
>> 
>> [image: Not run]
>> cmake
>> [image: Success]
>> 
>> [image: Not run]
>> ubuntu:14.04 --verbose --enable-libevent --enable-ssl autotools
>> [image: Failed]
>> 
>> [image: Failed]
>> 
>> cmake
>> [image: Failed]
>> 
>> [image: Success]
>> 
>> --verbose autotools
>> [image: Success]
>> 
>> [image: Failed]
>> 
>> cmake
>> [image: Success]
>> 
>> [image: Success]
>> 
>>
>> On Wed, Nov 15, 2017 at 9:57 PM, Adam Bordelon 
>> wrote:
>>
>>> Hi all,
>>>
>>> Please vote on releasing the following candidate as Apache Mesos 1.2.3.
>>> 1.2.3 is our last scheduled bug fix release in the 1.2.x branch.
>>>
>>> The CHANGELOG for the release is available at:
>>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_p
>>> lain;f=CHANGELOG;hb=1.2.3-rc1
>>> 
>>> 
>>>
>>> 

Re: Persistent volumes

[Benjamin Mahler]

+jpeach

The polling mechanism is used by the "disk/du" isolator to handle the case
where we don't have filesystem support for enforcing a quota on a
per-directory basis. I believe the "disk/xfs" isolator will stop writes
with EDQUOT without killing the task:

http://mesos.apache.org/documentation/latest/isolators/disk-xfs/

On Tue, Nov 28, 2017 at 1:19 PM, Gabriel Hartmann 
wrote:

> I agree with pretty much everything Hendrik just said with the exception
> of the use of disk quota.  The polling mechanism employed for enforcing
> disk usage implies that any breach of the disk usage limit by a Task also
> implies loss of access to that data forever.  This is true for ROOT volumes
> at least.  MOUNT volumes can be configured to map to "real" devices which
> can provide normal write failures when exceeding disk limits instead of
> essentially revoking all access to data forever.
>
> On Mon, Nov 27, 2017 at 11:34 PM Hendrik Haddorp 
> wrote:
>
>> As said, I only use persistent volumes with my only scheduler straight
>> on Mesos so do not exactly know how this works in Marathon...
>>
>> The persistent volume is created on a Mesos agent and basically ends up
>> being a folder on that hosts disk. So yes, you can not use the volume on
>> a different agent/slave. For marathon you would need to set a hostname
>> constraint that makes sure the same host is used when restarting the
>> task. You won't be able to use fail over to different agents just have
>> Marathon restart your task once it fails. Also only one task at a time
>> can have the volume bound.
>>
>> Yes, you can achieve persistence in pretty much the same way by using a
>> hostpath but then you are using implicit knowledge about your
>> environment, which is not very clean in my opinion, and thus have a
>> tighter coupling. The nice thing about persistent volumes is that they
>> are managed by Mesos. I do not need to tell the Mesos admin that I need
>> space at some location. I do not need to do something special if I have
>> multiple instances running as they get all their own directory. And I
>> can programatically destroy the volume and then the directory on the
>> host gets deleted again (at least since Mesos 1.0). So in my opinion the
>> usage of persistent volumes is much cleaner. But there are certainly use
>> cases that do not really work with them, like being able to fail over to
>> different host. For that you would wither need a shared network mount or
>> storage like HDFS. Btw, the Mesos containerizer should also enforce disk
>> quotas so your task would not be able to fill the filesystem.
>>
>> On 27.11.2017 16:11, Dino Lokmic wrote:
>> > yes I did. So I don't have to prepare it before task? I can't use
>> > volume created on slave A, from slave B
>> >
>> > Once task fails where will it be restarted? Do I have to specify host?
>> >
>> > If I do, it means I can achieve "persistence" same way I deploy now,
>> > by specifying hostpath for volume and hostname
>> >
>> > 
>> >   "constraints": [
>> > [
>> >   "hostname",
>> >   "CLUSTER",
>> >   "MYHOSTNAME"
>> > ]
>> >   ],
>> >   "container": {
>> > "type": "DOCKER",
>> > "volumes": [
>> >   {
>> > "containerPath": "/opt/storm/storm-local",
>> > "hostPath": "/opt/docker_data/storm/storm-local",
>> > "mode": "RW"
>> >   },
>> >   {
>> > "containerPath": "/opt/storm/logs",
>> > "hostPath": "/opt/docker_logs/storm/logs",
>> > "mode": "RW"
>> >   },
>> >   {
>> > "containerPath": "/home/xx/runtime/storm",
>> > "hostPath": "/home/xx/runtime/storm",
>> > "mode": "RO"
>> >   }
>> > ],
>> > "docker": {
>> >   "image": "xxx/storm-1.1.0",
>> >   "network": "HOST",
>> >   "portMappings": [],
>> >   "privileged": false,
>> >   "parameters": [],
>> >   "forcePullImage": true
>> > }
>> >   },
>> >
>> > 
>> >
>> >
>> >
>> > On Mon, Nov 27, 2017 at 3:05 PM, Hendrik Haddorp
>> > > wrote:
>> >
>> > I have my own scheduler that is performing a create operation. As
>> > you are using Marathon this call would have to be done by Marathon.
>> > Did you read
>> > https://mesosphere.github.io/marathon/docs/persistent-volumes.html
>> > 
>> ?
>> >
>> > On 27.11.2017 14:59, Dino Lokmic wrote:
>> >
>> > @hendrik
>> >
>> > How did you create this
>> > "my-volume-227927c2-3266-412b-8572-92c5c93c051a" volume?
>> >
>> > On Mon, Nov 27, 2017 at 7:59 AM, Hendrik Haddorp
>> > 
>> > > > >> wrote:
>> >
>> > Hi,
>> >
>> > I'm using persistent volumes directly on Mesos, without

Re: Welcome Andrew Schwartzmeyer as a new committer and PMC member!

[Benjamin Mahler]

Welcome and thanks for your contributions so far!

On Mon, Nov 27, 2017 at 11:00 PM, Joseph Wu  wrote:

> Hi devs & users,
>
> I'm happy to announce that Andrew Schwartzmeyer has become a new committer
> and member of the PMC for the Apache Mesos project.  Please join me in
> congratulating him!
>
> Andrew has been an active contributor to Mesos for about a year.  He has
> been the primary contributor behind our efforts to change our default build
> system to CMake and to port Mesos onto Windows.
>
> Here is his committer candidate checklist for your perusal:
> https://docs.google.com/document/d/1MfJRYbxxoX2-A-g8NEeryUdU
> i7FvIoNcdUbDbGguH1c/
>
> Congrats Andy!
> ~Joseph
>

Stripping Offer.AllocationInfo and Resource.AllocationInfo for non-MULTI_ROLE schedulers.

[Benjamin Mahler]

Hi folks,

When we released MULTI_ROLE support, Offers and Resources within them
included additional information, specifically the AllocationInfo which
indicated which role was being allocated to:

https://github.com/apache/mesos/blob/1.3.0/include/
mesos/v1/mesos.proto#L907-L923
https://github.com/apache/mesos/blob/1.3.0/include/
mesos/v1/mesos.proto#L1453-L1459

We included this information even for non-MULTI_ROLE schedulers, because:

(1) Any schedulers with the old protos would continue to work, since they
ignore the new fields their notion of matching resources to offers keeps
working.

(2) Any schedulers that update the protobuf, but leave their resource
matching logic as is, also continue to work since they ignore the
allocation info.

(3) Any schedulers that update the protobuf and upgrade their matching
logic, would need to update their scheduler code to reflect the changes in
the matching logic. This was OK since the scheduler is updating their own
code to line up with their own resource matching logic.

However, this change introduced some difficulty for libraries that exposed
resource matching logic and that support both schedulers that know about
allocation info and schedulers that do not. Such a library would need to do
something to ensure that both old and new schedulers work against it (e.g.
strip the information from incoming offers if the scheduler instantiated
the library without MULTI_ROLE capability).

So, we're thinking of stripping the AllocationInfo for non-MULTI_ROLE
schedulers to simplify this for libraries. Strictly speaking this is a
*breaking change* for any non-MULTI_ROLE schedulers that have already
updated their logic to depend on the AllocationInfo presence in 1.3.x or
1.4.x.

The assumption so far is that this will be an OK change since
non-MULTI_ROLE schedulers are probably ignoring this information. But
please let me know if this is not the case!

More information here: https://issues.apache.org/jira/browse/MESOS-8237

Ben

Re: 1.3.2 Release

[Benjamin Mahler]

Great!

I cherry picked Gaston's fix for https://issues.apache.org/
jira/browse/MESOS-8135.

On Wed, Nov 1, 2017 at 6:57 PM, Michael Park  wrote:

> Please reply to this email if you have pending patches to be backported to
> 1.3.x, I'm aiming to cut a 1.3.2 on Friday.
>
> Thanks,
>
> MPark
>

Re: orphan executor

[Benjamin Mahler]

I filed one: https://issues.apache.org/jira/browse/MESOS-8167

It's a pretty significant effort, and hasn't been requested a lot, so it's
unlikely to be worked on for some time.

On Tue, Oct 31, 2017 at 8:18 PM, Mohit Jaggi <mohit.ja...@uber.com> wrote:

> :-)
> Is there a Jira ticket to track this? Any idea when this will be worked on?
>
> On Tue, Oct 31, 2017 at 5:22 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
>> The question was posed merely to point out that there is no notion of the
>> executor "running away" currently, due to the answer I provided: there
>> isn't a complete lifecycle API for the executor. (This includes
>> healthiness, state updates, reconciliation, ability for scheduler to shut
>> it down, etc).
>>
>> On Tue, Oct 31, 2017 at 4:27 PM, Mohit Jaggi <mohit.ja...@uber.com>
>> wrote:
>>
>>> Good question.
>>> - I don't know what the interaction between mesos agent and executor is.
>>> Is there a health check?
>>> - There is a reconciliation between Mesos and Frameworks: will Mesos
>>> include the "orphan" executor in the list there, so framework can find
>>> runaways and kill them(using Mesos provided API)?
>>>
>>> On Tue, Oct 31, 2017 at 3:49 PM, Benjamin Mahler <bmah...@apache.org>
>>> wrote:
>>>
>>>> What defines a runaway executor?
>>>>
>>>> Mesos does not know that this particular executor should self-terminate
>>>> within some reasonable time after its task terminates. In this case the
>>>> framework (Aurora) knows this expected behavior of Thermos and can clean up
>>>> ones that get stuck after the task terminates. However, we currently don't
>>>> provide a great executor lifecycle API to enable schedulers to do this
>>>> (it's long overdue).
>>>>
>>>> On Tue, Oct 31, 2017 at 2:47 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>> wrote:
>>>>
>>>>> I was asking if this can happen automatically.
>>>>>
>>>>> On Tue, Oct 31, 2017 at 2:41 PM, Benjamin Mahler <bmah...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> You can kill it manually by SIGKILLing the executor process.
>>>>>> Using the agent API, you can launch a nested container session and
>>>>>> kill the executor. +jie,gilbert, is there a CLI command for 'exec'ing 
>>>>>> into
>>>>>> the container?
>>>>>>
>>>>>> On Tue, Oct 31, 2017 at 12:47 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes. There is a fix available now in Aurora/Thermos to try and exit
>>>>>>> in such scenarios. But I am curious to know if Mesos agent has the
>>>>>>> functionality to reap runaway executors.
>>>>>>>
>>>>>>> On Tue, Oct 31, 2017 at 12:08 PM, Benjamin Mahler <
>>>>>>> bmah...@apache.org> wrote:
>>>>>>>
>>>>>>>> Is my understanding correct that the Thermos transitions the task
>>>>>>>> to TASK_FAILED, but Thermos gets stuck and can't terminate itself? The
>>>>>>>> typical workflow for thermos, as a 1:1 task:executor approach, is that 
>>>>>>>> the
>>>>>>>> executor terminates itself after the task is terminal.
>>>>>>>>
>>>>>>>> The full logs of the agent during this window would help, it looks
>>>>>>>> like an agent termination is involved here as well?
>>>>>>>>
>>>>>>>> On Fri, Oct 27, 2017 at 3:09 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Here are some relevant logs. Aurora scheduler logs shows the task
>>>>>>>>> going from:
>>>>>>>>> INIT
>>>>>>>>> ->PENDING
>>>>>>>>> ->ASSIGNED
>>>>>>>>> ->STARTING
>>>>>>>>> ->RUNNING for a long time
>>>>>>>>> ->FAILED due to health check error, OSError: Resource temporarily
>>>>>>>>> unavailable (I think this is referring to running out of PID space, 
>>>>>>>>> see
>>>>>>>>> thermos logs below)
>>>>>>>>

Re: orphan executor

[Benjamin Mahler]

The question was posed merely to point out that there is no notion of the
executor "running away" currently, due to the answer I provided: there
isn't a complete lifecycle API for the executor. (This includes
healthiness, state updates, reconciliation, ability for scheduler to shut
it down, etc).

On Tue, Oct 31, 2017 at 4:27 PM, Mohit Jaggi <mohit.ja...@uber.com> wrote:

> Good question.
> - I don't know what the interaction between mesos agent and executor is.
> Is there a health check?
> - There is a reconciliation between Mesos and Frameworks: will Mesos
> include the "orphan" executor in the list there, so framework can find
> runaways and kill them(using Mesos provided API)?
>
> On Tue, Oct 31, 2017 at 3:49 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
>> What defines a runaway executor?
>>
>> Mesos does not know that this particular executor should self-terminate
>> within some reasonable time after its task terminates. In this case the
>> framework (Aurora) knows this expected behavior of Thermos and can clean up
>> ones that get stuck after the task terminates. However, we currently don't
>> provide a great executor lifecycle API to enable schedulers to do this
>> (it's long overdue).
>>
>> On Tue, Oct 31, 2017 at 2:47 PM, Mohit Jaggi <mohit.ja...@uber.com>
>> wrote:
>>
>>> I was asking if this can happen automatically.
>>>
>>> On Tue, Oct 31, 2017 at 2:41 PM, Benjamin Mahler <bmah...@apache.org>
>>> wrote:
>>>
>>>> You can kill it manually by SIGKILLing the executor process.
>>>> Using the agent API, you can launch a nested container session and kill
>>>> the executor. +jie,gilbert, is there a CLI command for 'exec'ing into the
>>>> container?
>>>>
>>>> On Tue, Oct 31, 2017 at 12:47 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>> wrote:
>>>>
>>>>> Yes. There is a fix available now in Aurora/Thermos to try and exit in
>>>>> such scenarios. But I am curious to know if Mesos agent has the
>>>>> functionality to reap runaway executors.
>>>>>
>>>>> On Tue, Oct 31, 2017 at 12:08 PM, Benjamin Mahler <bmah...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Is my understanding correct that the Thermos transitions the task to
>>>>>> TASK_FAILED, but Thermos gets stuck and can't terminate itself? The 
>>>>>> typical
>>>>>> workflow for thermos, as a 1:1 task:executor approach, is that the 
>>>>>> executor
>>>>>> terminates itself after the task is terminal.
>>>>>>
>>>>>> The full logs of the agent during this window would help, it looks
>>>>>> like an agent termination is involved here as well?
>>>>>>
>>>>>> On Fri, Oct 27, 2017 at 3:09 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Here are some relevant logs. Aurora scheduler logs shows the task
>>>>>>> going from:
>>>>>>> INIT
>>>>>>> ->PENDING
>>>>>>> ->ASSIGNED
>>>>>>> ->STARTING
>>>>>>> ->RUNNING for a long time
>>>>>>> ->FAILED due to health check error, OSError: Resource temporarily
>>>>>>> unavailable (I think this is referring to running out of PID space, see
>>>>>>> thermos logs below)
>>>>>>>
>>>>>>>
>>>>>>> --- mesos agent ---
>>>>>>>
>>>>>>> I1005 22:56:47.902153 127818 fetcher.cpp:285] Fetching directly into 
>>>>>>> the sandbox directory
>>>>>>> I1005 22:56:47.902170 127818 fetcher.cpp:222] Fetching URI 
>>>>>>> '/usr/bin/X'
>>>>>>> I1005 22:56:47.913270 127818 fetcher.cpp:207] Copied resource 
>>>>>>> '/usr/bin/x' to 
>>>>>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9f97-05bd84a32b54/runs/bb904e1d-4c32-4d7a-b1b6-9b3f78ddfe95/xxx'
>>>>>>> I1005 22:56:47.913331 127818 fetcher.cpp:582] Fetched '/usr/bin/xxx' to 
>>>>>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9

Re: orphan executor

[Benjamin Mahler]

What defines a runaway executor?

Mesos does not know that this particular executor should self-terminate
within some reasonable time after its task terminates. In this case the
framework (Aurora) knows this expected behavior of Thermos and can clean up
ones that get stuck after the task terminates. However, we currently don't
provide a great executor lifecycle API to enable schedulers to do this
(it's long overdue).

On Tue, Oct 31, 2017 at 2:47 PM, Mohit Jaggi <mohit.ja...@uber.com> wrote:

> I was asking if this can happen automatically.
>
> On Tue, Oct 31, 2017 at 2:41 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
>> You can kill it manually by SIGKILLing the executor process.
>> Using the agent API, you can launch a nested container session and kill
>> the executor. +jie,gilbert, is there a CLI command for 'exec'ing into the
>> container?
>>
>> On Tue, Oct 31, 2017 at 12:47 PM, Mohit Jaggi <mohit.ja...@uber.com>
>> wrote:
>>
>>> Yes. There is a fix available now in Aurora/Thermos to try and exit in
>>> such scenarios. But I am curious to know if Mesos agent has the
>>> functionality to reap runaway executors.
>>>
>>> On Tue, Oct 31, 2017 at 12:08 PM, Benjamin Mahler <bmah...@apache.org>
>>> wrote:
>>>
>>>> Is my understanding correct that the Thermos transitions the task to
>>>> TASK_FAILED, but Thermos gets stuck and can't terminate itself? The typical
>>>> workflow for thermos, as a 1:1 task:executor approach, is that the executor
>>>> terminates itself after the task is terminal.
>>>>
>>>> The full logs of the agent during this window would help, it looks like
>>>> an agent termination is involved here as well?
>>>>
>>>> On Fri, Oct 27, 2017 at 3:09 PM, Mohit Jaggi <mohit.ja...@uber.com>
>>>> wrote:
>>>>
>>>>> Here are some relevant logs. Aurora scheduler logs shows the task
>>>>> going from:
>>>>> INIT
>>>>> ->PENDING
>>>>> ->ASSIGNED
>>>>> ->STARTING
>>>>> ->RUNNING for a long time
>>>>> ->FAILED due to health check error, OSError: Resource temporarily
>>>>> unavailable (I think this is referring to running out of PID space, see
>>>>> thermos logs below)
>>>>>
>>>>>
>>>>> --- mesos agent ---
>>>>>
>>>>> I1005 22:56:47.902153 127818 fetcher.cpp:285] Fetching directly into the 
>>>>> sandbox directory
>>>>> I1005 22:56:47.902170 127818 fetcher.cpp:222] Fetching URI 
>>>>> '/usr/bin/X'
>>>>> I1005 22:56:47.913270 127818 fetcher.cpp:207] Copied resource 
>>>>> '/usr/bin/x' to 
>>>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9f97-05bd84a32b54/runs/bb904e1d-4c32-4d7a-b1b6-9b3f78ddfe95/xxx'
>>>>> I1005 22:56:47.913331 127818 fetcher.cpp:582] Fetched '/usr/bin/xxx' to 
>>>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9f97-05bd84a32b54/runs/bb904e1d-4c32-4d7a-b1b6-9b3f78ddfe95/xxx'
>>>>> WARNING: Your kernel does not support swap limit capabilities, memory 
>>>>> limited without swap.
>>>>> twitter.common.app debug: Initializing: twitter.common.log (Logging 
>>>>> subsystem.)
>>>>> Writing log files to disk in /mnt/mesos/sandbox
>>>>> I1005 22:58:15.677225 7 exec.cpp:162] Version: 1.1.0
>>>>> I1005 22:58:15.68086714 exec.cpp:237] Executor registered on agent 
>>>>> b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540
>>>>> Writing log files to disk in /mnt/mesos/sandbox
>>>>> I1006 01:13:52.95055239 exec.cpp:487] Agent exited, but framework has 
>>>>> checkpointing enabled. Waiting 365days to reconnect with agent 
>>>>> b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540
>>>>>
>>>>>
>>>>>
>>>>> --- thermos (Aurora) 
>>>>>
>>>>> 1 I1023 19:03:05.765677 52364 fetcher.cpp:582] Fetched '/usr/bin/xxx' to 
>>>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S3295/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-d3c1c4d9-4d74-433a-b26a-8a88bb7687b8/runs/982e7236-fccd-40bc-a2a5-d8a1901cf0bf/fxxx'
>>>

Re: orphan executor

[Benjamin Mahler]

You can kill it manually by SIGKILLing the executor process.
Using the agent API, you can launch a nested container session and kill the
executor. +jie,gilbert, is there a CLI command for 'exec'ing into the
container?

On Tue, Oct 31, 2017 at 12:47 PM, Mohit Jaggi <mohit.ja...@uber.com> wrote:

> Yes. There is a fix available now in Aurora/Thermos to try and exit in
> such scenarios. But I am curious to know if Mesos agent has the
> functionality to reap runaway executors.
>
> On Tue, Oct 31, 2017 at 12:08 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
>> Is my understanding correct that the Thermos transitions the task to
>> TASK_FAILED, but Thermos gets stuck and can't terminate itself? The typical
>> workflow for thermos, as a 1:1 task:executor approach, is that the executor
>> terminates itself after the task is terminal.
>>
>> The full logs of the agent during this window would help, it looks like
>> an agent termination is involved here as well?
>>
>> On Fri, Oct 27, 2017 at 3:09 PM, Mohit Jaggi <mohit.ja...@uber.com>
>> wrote:
>>
>>> Here are some relevant logs. Aurora scheduler logs shows the task going
>>> from:
>>> INIT
>>> ->PENDING
>>> ->ASSIGNED
>>> ->STARTING
>>> ->RUNNING for a long time
>>> ->FAILED due to health check error, OSError: Resource temporarily
>>> unavailable (I think this is referring to running out of PID space, see
>>> thermos logs below)
>>>
>>>
>>> --- mesos agent ---
>>>
>>> I1005 22:56:47.902153 127818 fetcher.cpp:285] Fetching directly into the 
>>> sandbox directory
>>> I1005 22:56:47.902170 127818 fetcher.cpp:222] Fetching URI '/usr/bin/X'
>>> I1005 22:56:47.913270 127818 fetcher.cpp:207] Copied resource 
>>> '/usr/bin/x' to 
>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9f97-05bd84a32b54/runs/bb904e1d-4c32-4d7a-b1b6-9b3f78ddfe95/xxx'
>>> I1005 22:56:47.913331 127818 fetcher.cpp:582] Fetched '/usr/bin/xxx' to 
>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9f97-05bd84a32b54/runs/bb904e1d-4c32-4d7a-b1b6-9b3f78ddfe95/xxx'
>>> WARNING: Your kernel does not support swap limit capabilities, memory 
>>> limited without swap.
>>> twitter.common.app debug: Initializing: twitter.common.log (Logging 
>>> subsystem.)
>>> Writing log files to disk in /mnt/mesos/sandbox
>>> I1005 22:58:15.677225 7 exec.cpp:162] Version: 1.1.0
>>> I1005 22:58:15.68086714 exec.cpp:237] Executor registered on agent 
>>> b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540
>>> Writing log files to disk in /mnt/mesos/sandbox
>>> I1006 01:13:52.95055239 exec.cpp:487] Agent exited, but framework has 
>>> checkpointing enabled. Waiting 365days to reconnect with agent 
>>> b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540
>>>
>>>
>>>
>>> --- thermos (Aurora) 
>>>
>>> 1 I1023 19:03:05.765677 52364 fetcher.cpp:582] Fetched '/usr/bin/xxx' to 
>>> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S3295/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-d3c1c4d9-4d74-433a-b26a-8a88bb7687b8/runs/982e7236-fccd-40bc-a2a5-d8a1901cf0bf/fxxx'
>>>  22 WARNING: Your kernel does not support swap limit capabilities, memory 
>>> limited without swap.
>>>  23 twitter.common.app debug: Initializing: twitter.common.log (Logging 
>>> subsystem.)
>>>  24 Writing log files to disk in /mnt/mesos/sandbox
>>>  25 I1023 19:04:32.261165 7 exec.cpp:162] Version: 1.2.0
>>>  26 I1023 19:04:32.26487042 exec.cpp:237] Executor registered on agent 
>>> b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S3295
>>>  27 Writing log files to disk in /mnt/mesos/sandbox
>>>  28 Traceback (most recent call last):
>>>  29   File 
>>> "/root/.pex/install/twitter.common.exceptions-0.3.7-py2-none-any.whl.f6376bcca9bfda5eba4396de2676af5dfe36237d/twitter.common.exceptions-0.3.7-py2-none-any.whl/twitter/common/exceptions/__init__.py",
>>>  line 126, in _excepting_run
>>>  30 self.__real_run(*args, **kw)
>>>  31   File "apache/thermos/monitoring/resource.py", line 243, in run
>>>  32   File 
>>> "/root/.pex/install/twitter.common.concurrent-0.3.7-py2-none-any.whl.f1ab836a5554c86d07fa3f075905

Re: rotating secrets when authenticating framework

[Benjamin Mahler]

+adam, alexander

On Fri, Oct 20, 2017 at 2:54 PM, Devendra Ayalasomayajula <
devend...@nvidia.com> wrote:

> Corrected the subject
>
>
>
> *From:* Devendra Ayalasomayajula
> *Sent:* Friday, October 20, 2017 2:40 PM
> *To:* user@mesos.apache.org
> *Subject:* rotting secrets when authenticating framework
>
>
>
> Hi,
>
>
>
> The framework I am experimenting with is using MesosSchedulerDriver and I
> am planning to pass Credential. But If the secret is updated how can the
> Credential that’s passed to the driver be updated.
>
> How to handle secrets with expiry ?
>
>
>
> Thank You
>
> Devendra
> --
>
> This email message is for the sole use of the intended recipient(s) and
> may contain confidential information.  Any unauthorized review, use,
> disclosure or distribution is prohibited.  If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> --
>