Re: Master configuration in the registry

[Benjamin Mahler]

+user

I explicitly didn't mention "update without restart" as a criterion because
you could use a watched configuration file to achieve this, like we did for
the whitelist. If the master or agent observe any changes to the
configuration file then they can apply them. If they cannot apply the
changes without restarting (or at all!), they could exit or ignore/warn.

If we expect there to be natural churn in a particular configuration via
tooling, then the watched file approach adds some friction when wanting to
make interactive changes. Accordingly, the only ones I would have excluded
from your list is the ping timeouts and the max completed frameworks/tasks,
as these don't seem like things that we expect to have natural churn over
time via tooling.

Still, some of these I can imagine operators having a preference for
configuration files or flags over the endpoint + registry persistence:

e.g. I may want to use a private credentials file rather than storing
it in the master's registry.
e.g. I may prefer to "deploy" new rate limit rules using my existing
configuration deploy tooling, because hitting an endpoint currently
requires me to find the leader, hit the endpoint, deal with it failing over
while the endpoint is being hit, etc.
e.g. I want to enforce that the master comes up with these ACLs,
without having to do a two phase procedure which leaves the master
vulnerable with no ACLs while I quickly try to set ACLs!

My thought here is that we may need an approach that allows the operator to
choose between flags/files and endpoints for setting some of the "dynamic"
configuration, or provide a means to solve each specific problem that
arises with using endpoints (examples listed above).

Curious if users/operators have any thoughts on this.

Ben

On Sat, Jun 11, 2016 at 4:20 AM, Adam Bordelon  wrote:

> The motivation for "dynamic" quota/weights was that we wanted operators to
> be able to change these without restarting (all) masters. That means you'd
> update these values via operator endpoints, and if you're doing that, then
> the flag values become outdated/irrelevant, and we must persist these
> values somewhere that they can be restored on master failover (i.e.
> registry). However, we have kept the flags around for bootstrapping and
> backwards compatibility.
>
> Other flags that could benefit from dynamic updates (without restarting
> masters):
> - acls
> - credentials
> - firewall_rules
> - rate_limits
> - ping timeouts
> - max completed frameworks/tasks
> - whitelist (actually reloads the file every so often)
>
> On Fri, Jun 10, 2016 at 4:29 AM, Neil Conway 
> wrote:
>
> > Makes sense: arguably you could say that "quota" and "weights" are
> > part of the master's (mutable) "state", not its "configuration", which
> > is largely immutable.
> >
> > Another distinction is that some configuration flags control behavior
> > that doesn't need to be consistent between master replicas (e.g.,
> > "--ip", "--port", "--advertise-ip", "--advertise-port", "--hostname",
> > "--hostname-lookup", "--quiet", "--log_dir", "--modules_dir",
> > "--work_dir", etc).
> >
> > Neil
> >
> > On Fri, Jun 10, 2016 at 3:52 AM, Benjamin Mahler 
> > wrote:
> > > I'm curious to hear thoughts on the distinction between using flags and
> > > persisting in the registry for master configuration. This topic had
> come
> > up
> > > in a discussion and our current choices are intuitive but the criteria
> > were
> > > not immediately obvious.
> > >
> > > Two cases seem interesting to me:
> > >
> > > (1) Quota.
> > > (2) Weights.
> > >
> > > These are configuration, but we persist them in the registry. Why is
> > that?
> > >
> > > My intuition is that they reflect the organizational aspects of the
> > > workloads that are running and so we expect administrators and (most
> > > importantly!!) tooling to be view and modify these over time.
> > >
> > > Timeouts, work directories, etc, on the other hand, are rarely modified
> > and
> > > require initial values. There are also sane defaults for these that
> will
> > > work for most users.
> > >
> > > Thought this might be helpful for others that may wonder about this.
> Let
> > me
> > > know if there are any other important criteria that I've missed.
> > >
> > > Ben
> >
>

Re: Protobuf syntax version for Mesos

[Zhitao Li]

Yes we are indeed using protoc version 3.0.0. I'll try out using protoc v2.
Thanks.

Meanwhile, is there an issue or conversation about upgrading Mesos to
protoc v3? Not sure whether there's actual benefit.

On Mon, Jun 13, 2016 at 2:51 PM, Joseph Wu  wrote:

> Looks like that is a warning in v3, see [1].  The same code in v2.6.1 is
> [2], and does not have that warning.
>
> [1]
>
> https://github.com/google/protobuf/blob/088c5c491e7a1c95c7b8eb55f119a8a999c81dc1/src/google/protobuf/compiler/parser.cc#L547-L550
> [2]
>
> https://github.com/google/protobuf/blob/bba83652e1be610bdb7ee1566ad18346d98b843c/src/google/protobuf/compiler/parser.cc#L436-L444
>
> On Mon, Jun 13, 2016 at 2:46 PM, Vinod Kone  wrote:
>
> > Are you are using a v3 protoc compiler? Looks like the "syntax" keyword
> was
> > introduced in
> > https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-2.
> > Not sure if v2 protoc compilers recognize that keyword.
> >
> > On Mon, Jun 13, 2016 at 5:42 PM, Zhitao Li 
> wrote:
> >
> > > Hi,
> > >
> > > We are experimenting with HTTP API and tried to compile Mesos's .proto
> > > files with our repo (with make commands like "protoc
> > --proto_path=protobuf
> > > --go_out=.gen protobuf/mesos/v1/scheduler/scheduler.proto", but we see
> > > warnings like
> > >
> > > ```
> > > [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> > > specified for the proto file: mesos/v1/scheduler/scheduler.proto.
> > > Please use 'syntax
> > > = "proto2";' or 'syntax = "proto3";' to specify a syntax version.
> > > (Defaulted
> > > to proto2 syntax.)
> > > ```
> > >
> > > Should we have proper `syntax` clauses in various .proto files? If not,
> > > what's the reason?
> > >
> > > Thanks.
> > >
> > > --
> > > Cheers,
> > >
> > > Zhitao Li
> > >
> >
>



-- 
Cheers,

Zhitao Li

Re: Protobuf syntax version for Mesos

[Joseph Wu]

Looks like that is a warning in v3, see [1].  The same code in v2.6.1 is
[2], and does not have that warning.

[1]
https://github.com/google/protobuf/blob/088c5c491e7a1c95c7b8eb55f119a8a999c81dc1/src/google/protobuf/compiler/parser.cc#L547-L550
[2]
https://github.com/google/protobuf/blob/bba83652e1be610bdb7ee1566ad18346d98b843c/src/google/protobuf/compiler/parser.cc#L436-L444

On Mon, Jun 13, 2016 at 2:46 PM, Vinod Kone  wrote:

> Are you are using a v3 protoc compiler? Looks like the "syntax" keyword was
> introduced in
> https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-2.
> Not sure if v2 protoc compilers recognize that keyword.
>
> On Mon, Jun 13, 2016 at 5:42 PM, Zhitao Li  wrote:
>
> > Hi,
> >
> > We are experimenting with HTTP API and tried to compile Mesos's .proto
> > files with our repo (with make commands like "protoc
> --proto_path=protobuf
> > --go_out=.gen protobuf/mesos/v1/scheduler/scheduler.proto", but we see
> > warnings like
> >
> > ```
> > [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> > specified for the proto file: mesos/v1/scheduler/scheduler.proto.
> > Please use 'syntax
> > = "proto2";' or 'syntax = "proto3";' to specify a syntax version.
> > (Defaulted
> > to proto2 syntax.)
> > ```
> >
> > Should we have proper `syntax` clauses in various .proto files? If not,
> > what's the reason?
> >
> > Thanks.
> >
> > --
> > Cheers,
> >
> > Zhitao Li
> >
>

Re: Protobuf syntax version for Mesos

[Vinod Kone]

Are you are using a v3 protoc compiler? Looks like the "syntax" keyword was
introduced in https://github.com/google/protobuf/releases/tag/v3.0.0-alpha-2.
Not sure if v2 protoc compilers recognize that keyword.

On Mon, Jun 13, 2016 at 5:42 PM, Zhitao Li  wrote:

> Hi,
>
> We are experimenting with HTTP API and tried to compile Mesos's .proto
> files with our repo (with make commands like "protoc --proto_path=protobuf
> --go_out=.gen protobuf/mesos/v1/scheduler/scheduler.proto", but we see
> warnings like
>
> ```
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: mesos/v1/scheduler/scheduler.proto.
> Please use 'syntax
> = "proto2";' or 'syntax = "proto3";' to specify a syntax version.
> (Defaulted
> to proto2 syntax.)
> ```
>
> Should we have proper `syntax` clauses in various .proto files? If not,
> what's the reason?
>
> Thanks.
>
> --
> Cheers,
>
> Zhitao Li
>

1.0.0 RC2

[Vinod Kone]

Hi folks,

I'm planning to cut 1.0 RC2 later this week (likely friday). So please make
sure to get any patches targeted for 1.0 (esp. blockers) upstreamed.

The dashboard for the release is here:
https://issues.apache.org/jira/issues/?filter=12335793

Thanks,
Vinod

It's now easy to build & deploy a full search-stack with YBT 0.2.0rc6 !

[Itamar Ostricher]

Hi,

I released YBT 0.2.0rc6 and merged to master some changes that use it.
No urgency to pull master or upgrade ybt - only when you get to it :-) .

*Here's a changelist*, yay!

1. YBT is *less noisy*, hurray! (less internal prints to stdout. run with
`ybt -v` or `--verbose` to get those outputs if you want them. use
`--logtostderr` for even more, btw).
2. *Built-in YBT version checker* - when running `ybt` commands in the
project, it will now tell you if your version is not updated enough + print
the upgrade command if necessary (but will not upgrade for you). Current
minimal version is 0.2.0rc6 (surprisingly).
3. Perr'y feature request - ability to *control number of scons jobs* (e.g.
`SCONS_JOBS=2 ybt build :run-scons` to use 2 jobs just for this run, or
`export SCONS_JOBS=2` to use 2 jobs in the current session).
4. *Grunt builder* allows running grunt using YBT. This means that now `ybt
:search-stack` also builds frontend, and it is *much easier now to build
frontend* even for non-Shai! (but don't try this without pulling
master...)

In fact, this last bullet makes it a one-liner to build and deploy a
complete search-stack from any machine (not just prod-us-central)!
./scripts/deploy.py --logtostderr --norun_scons --skip_grunt --version
 (e.g. your name)

Docker Tip: On the mortality of the container filesystem

[Itamar Ostricher]

As mentioned in the seminar(s) and in the previous tip, "bind-mounting" is
the Docker way to share a directory between the host filesystem (yours),
and a running container.

Again, reminding that you'd want to bind-mount to:
1. Give the container access to files on your host, that are not available
inside the sandbox (e.g. the Docker image).
2. Allow the container to write output to the host, so it can live beyond
the lifetime of the container. (like SCons does to put the built binaries
in your build/release)

That second point can be tricky to digest.
Whenever you use Docker to run something that has any output that lives in
the filesystem, this output can go to one of two places:
1. A path inside the container filesystem itself (e.g. `/tmp`).
2. A bind-mounted volume that is mapped from the host into the container
(e.g. `/project` in most of the commands we commonly use).

Docker containers are ephemeral (e.g., mortal - go away once the
application they're running ends). Once the container ends, the container
filesystem "evaporates" with it.
*This means that anything written to it since it began also goes away with
it!*

You may consider this an annoyance (why this Docker thing throws away my
stuff?!), but it's also a pretty useful feature (hey, look at this easy way
to revert my application to its default good state!).
It is actually used in production systems to quickly recover from hacking
attacks (the application in a container was hacked - throw away hacked
container and start a clean new one).

*For the cases that you actually need to have outputs from the container
outlive the container itself* - bind-mounting is the solution that enables
it, and it is done using (one or more) `-v /foo:/bar` flags on the `docker
run` command-line.

Now, with this understanding, you might want to revisit the previous tip on
faking uid inside a container, in case it was confusing... :-)

Re: [Tech-debt] Introduce regex into Mesos

[Klaus Ma]

As discussed in the RR, we'll enhance current code to handle port ranges as a 
short term solution.


I've logged MESOS-5602 (Introduce expression grammar 
library) to trace the long 
term solution for Mesos defined data format, e.g. resources.




Da (Klaus), Ma (??), PMP®| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me





From: Klaus Ma 
Sent: Saturday, June 11, 2016 2:09 AM
To: dev
Subject: Re: [Tech-debt] Introduce regex into Mesos

As Joseph said, regex works well in llvm 7.3 & gcc 5.3, but does not work in 
gcc 4.8.4; also try "" in gc 4.8.4, but "ld" failed :(.




Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me



From: Joseph Wu 
Sent: Friday, June 10, 2016 8:15:51 PM
To: dev
Subject: Re: [Tech-debt] Introduce regex into Mesos

Same here.

Mesos currently requires GCC 4.8.1+.  Regex support was implemented in GCC
4.9.0, see [1].

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53631

On Fri, Jun 10, 2016 at 11:39 AM, Kevin Klues  wrote:

> By compiler errors, I mean "internal compiler errors"
>
> On Fri, Jun 10, 2016 at 11:38 AM, Kevin Klues  wrote:
> > I've run into compiler errors using simple regex stuff from the
> > standard library on our supported version of gcc.
> >
> > On Thu, Jun 9, 2016 at 7:30 PM, Klaus Ma  wrote:
> >> Hi team,
> >>
> >>
> >> We're discussing to introduce regex into Mesos when investigating
> MESOS-4627; so I'd like
> to ask whether anyone has experience on regex after C++11? for example,
> supported compiler, compatibility, performance and so on :).
> >>
> >>
> >> 
> >>
> >> Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
> >> Platform DCOS Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>
> >> 
> >
> >
> >
> > --
> > ~Kevin
>
>
>
> --
> ~Kevin
>

[Compatibility] More strict parsing of ranges, e.g. port of resources

[Klaus Ma]

Hi team,


In "r/43561/", we'd like to move towards 
more strict parsing of ranges. If any comments, please let me know.


Before the patch, the follow ranges are all valid:

[1-2,3-4]

[1-4]

[[1-4]]

[1-2]\n[3-4]

[1-2],[3-4]


After the patch, only the following ranges are valid:

[1-2,3-4]

[1-4]




Da (Klaus), Ma (??), PMP®| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

[Compatibility] More strict parsing of ranges, e.g. port of resources

[Klaus Ma]

Hi team,


In "r/43561/", we'd like to move towards 
more strict parsing of ranges. If any comments, please let me know.


Before the patch, the follow ranges are all valid:

[1-2,3-4]

[1-4]

[[1-4]]

[1-2]\n[3-4]

[1-2],[3-4]


After the patch, only the following ranges are valid:

[1-2,3-4]

[1-4]




Da (Klaus), Ma (??), PMP®| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: Code Quality Improvements for docker-compose-executor

[Joseph Wu]

I'm not sure what the community adoption of the docker-compose-executor [1]
is, but from a Mesos perspective, the repo will eventually be superceded by
"pod" support in Mesos itself [2] [3].

Also, you should try to contact the developers of docker-compose-executor
itself, as they might not be subscribed to this mailing list.

[1] https://github.com/mesos/docker-compose-executor
[2] https://issues.apache.org/jira/browse/MESOS-2449
[3] https://issues.apache.org/jira/browse/MESOS-2634

On Sun, Jun 12, 2016 at 8:38 AM,  wrote:

> Hello,
>
> I'd like to send you some pull requests to improve the maintainability of
> docker-compose-executor.
>
> My company - DevFactory - is sponsoring me to identify and fix code
> quality issues and improve unit test coverage in open source projects.
> DevFactory is obsessed with code quality and is providing its commercially
> available code quality improvement service for free to qualified
> open-source projects.
>
> If you are interested, please let me know and we will add it to our
> pipeline. Our first step will be to utilize tools like PMD, FindBugs and
> Sonar to identify the most important issues to fix. Once we fix them, we'll
> follow up with some pull requests.
>
> Thanks,
> M.Ezzat
>
>

Re: [GPU] [Allocation] "Scarce" Resource Allocation

[Yong Feng]

Thanks Ben.M for initiative.

You already define the scarce resource and describe the problem clearly.
Just recap the requirements we want to resolved as follows

1) Scarce resource should not be treated as dominated resource during
allocating so that user could continue using non-scarce resource no matter
how many scarce resource he/she used.
2) Host with scarce resource should be firstly (or only) allocated to
frameworks who need scarce resource so that those frameworks will not be
starved

However it also introduce new problems for example

How to enforce "fairness" among frameworks when allocating scarce
resources?
How to improve resource utilization when cluster is partitioned by scarce
resource?

Go back to the use case of GPU, I agree on the short term solutions

a) Exclude GPU from dominated resource
b) Allow framework to specify scarce resource for example GPU as capability
when register, and then Mesos only send offer with scarce resource to those
framework. The capability should not be hard coded to only handle GPU.

For long term, since scarce resource and even host attribute already
partition cluster into resource pools, we should have solution to resolve
the new introduced problems mentioned above.

1) Mesos will need enforce fairness during allocating scarce resource (or
host with special attribute) as well
2) Mesos should allow frameworks who do not ask for scarce resource (or
host with special attribute) to use host with scarce resource (or special
attribute), if there is no framework ask for scarce resource (or host with
special attribute).
3) Mesos should allow frameworks who ask for scarce resource preempt scarce
resource back.

YARN's feature of node label in future release of 2.8 (
https://wangda.live/2016/04/16/suggestions-about-how-to-better-use-yarn-node-label/)
try to resolve the similar use cases (GPU and so on). Even though YARN's
allocation model is request based, we still are able to gain experience
from it.

Thanks,

Yong

On Fri, Jun 10, 2016 at 10:50 PM, Benjamin Mahler 
wrote:

> I wanted to start a discussion about the allocation of "scarce" resources.
> "Scarce" in this context means resources that are not present on every
> machine. GPUs are the first example of a scarce resource that we support as
> a known resource type.
>
> Consider the behavior when there are the following agents in a cluster:
>
> 999 agents with (cpus:4,mem:1024,disk:1024)
> 1 agent with (gpus:1,cpus:4,mem:1024,disk:1024)
>
> Here there are 1000 machines but only 1 has GPUs. We call GPUs a "scarce"
> resource here because they are only present on a small percentage of the
> machines.
>
> We end up with some problematic behavior here with our current allocation
> model:
>
> (1) If a role wishes to use both GPU and non-GPU resources for tasks,
> consuming 1 GPU will lead DRF to consider the role to have a 100% share of
> the cluster, since it consumes 100% of the GPUs in the cluster. This
> framework will then not receive any other offers.
>
> (2) Because we do not have revocation yet, if a framework decides to
> consume the non-GPU resources on a GPU machine, it will prevent the GPU
> workloads from running!
>
> 
>
> I filed an epic [1] to track this. The plan for the short-term is to
> introduce two mechanisms to mitigate these issues:
>
> -Introduce a resource fairness exclusion list. This allows the shares
> of resources like "gpus" to be excluded from the dominant share.
>
> -Introduce a GPU_AWARE framework capability. This indicates that the
> scheduler is aware of GPUs and will schedule tasks accordingly. Old
> schedulers will not have the capability and will not receive any offers for
> GPU machines. If a scheduler has the capability, we'll advise that they
> avoid placing their additional non-GPU workloads on the GPU machines.
>
> 
>
> Longer term, we'll want a more robust way to manage scarce resources. The
> first thought we had was to have sub-pools of resources based on machine
> profile and perform fair sharing / quota within each pool. This addresses
> (1) cleanly, and for (2) the operator needs to explicitly disallow non-GPU
> frameworks from participating in the GPU pool.
>
> Unfortunately, by excluding non-GPU frameworks from the GPU pool we may
> have a lower level of utilization. In the even longer term, as we add
> revocation it will be possible to allow a scheduler desiring GPUs to revoke
> the resources allocated to the non-GPU workloads running on the GPU
> machines. There are a number of things we need to put in place to support
> revocation ([2], [3], [4], etc), so I'm glossing over the details here.
>
> If anyone has any thoughts or insight in this area, please share!
>
> Ben
>
> [1] https://issues.apache.org/jira/browse/MESOS-5377
> [2] https://issues.apache.org/jira/browse/MESOS-5524
> [3] https://issues.apache.org/jira/browse/MESOS-5527
> [4] https://issues.apache.org/jira/browse/MESOS-4392
>

Re: Apply to be added to Mesos contributor list

[Alex Rukletsov]

Done. Welcome!

On Sun, Jun 12, 2016 at 5:42 AM, Yan Yan YY Hu  wrote:

>
> Dear Mesos team,
>
> I'm Yanyan Hu from IBM China Research. I hope to propose a fix to Mesos for
> jira MESOS-5425. Very appreciated if anyone can help to add me to Mesos
> contributor list. Thank you so much!
>
>
> Best regards!
> **
> Yanyan Hu(胡彦彦) Ph.D.
> Cloud Infrastructure & Technology Team
> Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, Haidian
> District, Beijing,P.R.C.100094
> E-mail: yanya...@cn.ibm.com
> Tel: 8610-58748025
> ***
>