Re: MESOS-5731 Add metric types to GetMetrics v1 api

[Tuan-Anh Hoang-Vu]

I've pushed 4 review requests for this issue:

https://reviews.apache.org/r/49936/
https://reviews.apache.org/r/49937/
https://reviews.apache.org/r/49938/
https://reviews.apache.org/r/49939/

@Benjamin: I'm not sure I understand what you mean. Some of current Timers
have `statistics` to calculate count, min, max, p50, p95, ... so
technically they are already Histograms?


On Mon, Jul 11, 2016 at 1:44 PM, Benjamin Mahler  wrote:

> Have you considered introducing 'histograms' rather than 'timers'? Our
> Timers are just a special case of a histogram. For example, you could have
> a histogram of response sizes.
>
> On Fri, Jul 8, 2016 at 3:32 PM, Tuan-Anh Hoang-Vu 
> wrote:
>
> > Hi all,
> >
> > I've finished implementing and testing this issue
> > https://issues.apache.org/jira/browse/MESOS-5731 and would like to find
> a
> > shepherd for my patch.
> >
> > Should I go ahead and submit the patch?
> >
> > Following Vinod's suggestion, I've changed Response::GetMetrics in
> > master.proto and v1/master.proto from
> >
> >   message GetMetrics {
> > repeated Metric metrics = 1;
> >   }
> >
> > to
> >
> >   message GetMetrics {
> > repeated Metric counters = 1;
> > repeated Metric gauges = 2;
> > repeated Metric timers = 3;
> >   }
> >
>

Re: MESOS: Improve the performance of Resources.

[Guangya Liu]

Hi Joris,

For `2x` number, can you please show more detail for how did you do and
evaluate the test? How did you compare the result? Did you have any test
code to share?

Thanks,

Guangya

On Mon, Jul 11, 2016 at 6:05 PM, Klaus Ma  wrote:

> Hi Joris,
>
> For `Scalars`, yes, it's also dynamic allocated in
> `Resources::mutable_scalar()`.
>
> For `2x` number, do you have any patch to share? Tested cases this
> afternoon: add resources including 100 roles (1 CPU for each); the
> performance is downgrade a lot; so I agree with you to improve some
> algorithms in Resources/Sorter.
>
> For 'basic benchmarks', temporary tracking in my personal github (
> https://github.com/k82cn/mesos/blob/resources_benchmark/src/tests/resources_tests.cpp).
> The following cases are in my mind to add:
> 1. simple resources, e.g. 1 cpu
> 2. resources with port, e.g. [1-2], [3-4], ... [101-102]
> 3. resources with reservation, cpus(r1):1;cpus(r2):1;  cpus(r10):1
> 4. resources with diskInfo
> 5. resources with revocableInfo
>
> The operators will be +, +=, -, -=, cpus(), contains.
>
> I booked a weekly call to discuss for allocaiton performance and sent the
> invitation to the dev@.
>
> If any comments, please let me know.
>
> 
> Da (Klaus), Ma (马达) | PMP® | Software Architect
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Mon, Jul 11, 2016 at 3:28 PM, Joris Van Remoortere  > wrote:
>
>> +Dev
>>
>> Hey Klaus,
>>
>> Using Stout's `Optional` to represent the `optional` concept of a message
>> in protobuf is definitely a step in the right direction.
>> Regarding your comment in slack yesterday: From my version of the
>> protobuf generated code there definitely is dynamic allocation even for
>> scalars.
>>
>> It looks like in our case there is a minimum of 3 dynamic allocations per
>> Resource object:
>>
>>> void Resource::SharedDtor() {
>>>   if (name_ != &::google::protobuf::internal::kEmptyString) {
>>> delete name_;
>>>   }
>>>   if (role_ != _default_role_) {
>>> delete role_;
>>>   }
>>>   if (this != default_instance_) {
>>> delete scalar_;
>>> delete ranges_;
>>> delete set_;
>>> delete reservation_;
>>> delete disk_;
>>> delete revocable_;
>>>   }
>>> }
>>
>>
>>  The 2x number I mentioned came from running some of the existing
>> benchmarks. I didn't explore further because it didn't have as big an
>> impact as I had hoped. The first battle is simplifying some of the
>> algorithms in the Sorter / Resources. Once that is done then the resource
>> arithmetic will be more of a bounding factor.
>>
>> I agree with Ben that we should focus on writing some basic benchmarks
>> that represent the common uses of Resources in the allocator. We should
>> scale these benchmarks to represent some of the more stressful environments
>> that could occur. For example, had we had such a benchmark, we would have
>> realized much earlier on that we needed to aggregate only quantities in the
>> Sorter, and that using the existing form of Resources would have led to a
>> grinding halt if a reservation were made on every machine.
>>
>> Is there a regular call that is scheduled to discuss this? I think there
>> are some other folks also working on benchmarks and interested in the
>> discussion.
>>
>> —
>> *Joris Van Remoortere*
>> Mesosphere
>>
>> On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma  wrote:
>>
>>> + more devs :).
>>>
>>> 
>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>> Platform OpenSource Technology, STG, IBM GCG
>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>>
>>> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma 
>>> wrote:
>>>
 Hi Joris,

 I think `Option` is helpful to the performance improvement, it used
 `placement new` to avoid dynamic allocation. Suppose you're using Option
 for optional member in protobuf, and using class instance directly
 (operator=).

 I'm adding some benchmark for `Resources`, especially for the
 `Resources` with Rang, DiskInfo and ReservationInfo

 Draft PR for Benchmark of Resources:
 https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d68a8cf2249eb641c


 
 Da (Klaus), Ma (马达) | PMP® | Software Architect
 Platform OpenSource Technology, STG, IBM GCG
 +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

>>>
>>>
>>
>

Fwd: [jira] [Created] (MESOS-5817) Port libprocess process_tests.cpp

[Benjamin Mahler]

This doesn't look like a "Bug" (there are a bunch of other tickets that are
misclassified). Please be sure to set the appropriate "issue type". :)

Can you go through your tickets and update the types?

-- Forwarded message --
From: Alex Clemmer (JIRA) 
Date: Fri, Jul 8, 2016 at 1:45 PM
Subject: [jira] [Created] (MESOS-5817) Port libprocess process_tests.cpp
To: iss...@mesos.apache.org


Alex Clemmer created MESOS-5817:
---

 Summary: Port libprocess process_tests.cpp
 Key: MESOS-5817
 URL: https://issues.apache.org/jira/browse/MESOS-5817
 Project: Mesos
  Issue Type: Bug
  Components: libprocess
Reporter: Alex Clemmer
Assignee: Alex Clemmer






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Patch for website

[Benjamin Mahler]

I just ran a website update, you should see it now if you force a refresh.

On Mon, Jul 11, 2016 at 2:26 PM, Tomek Janiszewski 
wrote:

> Page need to be manually generated and published.
>
> pon., 11.07.2016, 22:59 użytkownik Rich Bowen  napisał:
>
> >
> >
> > On 2016-07-07 12:36 (-0400), "Rich Bowen" wrote:
> > >
> > >
> > > On 2016-07-07 11:58 (-0400), Vinod Kone  wrote:
> > > > Thanks Rich! The easiest is to just send a PR to our github repo:
> > > > https://github.com/apache/mesos
> > >
> > >
> > > Excellent. Will do.
> > >
> > >
> >
> > I see that the PR was merged, but the site still has the old links. Is
> > there something that still needs to be done?
> >
>

Re: Patch for website

[Tomek Janiszewski]

Page need to be manually generated and published.

pon., 11.07.2016, 22:59 użytkownik Rich Bowen  napisał:

>
>
> On 2016-07-07 12:36 (-0400), "Rich Bowen" wrote:
> >
> >
> > On 2016-07-07 11:58 (-0400), Vinod Kone  wrote:
> > > Thanks Rich! The easiest is to just send a PR to our github repo:
> > > https://github.com/apache/mesos
> >
> >
> > Excellent. Will do.
> >
> >
>
> I see that the PR was merged, but the site still has the old links. Is
> there something that still needs to be done?
>

Re: Patch for website

[Rich Bowen]

On 2016-07-07 12:36 (-0400), "Rich Bowen" wrote: 
> 
> 
> On 2016-07-07 11:58 (-0400), Vinod Kone  wrote: 
> > Thanks Rich! The easiest is to just send a PR to our github repo:
> > https://github.com/apache/mesos
> 
> 
> Excellent. Will do.
> 
> 

I see that the PR was merged, but the site still has the old links. Is there 
something that still needs to be done?

Re: MESOS-5731 Add metric types to GetMetrics v1 api

[Benjamin Mahler]

Have you considered introducing 'histograms' rather than 'timers'? Our
Timers are just a special case of a histogram. For example, you could have
a histogram of response sizes.

On Fri, Jul 8, 2016 at 3:32 PM, Tuan-Anh Hoang-Vu 
wrote:

> Hi all,
>
> I've finished implementing and testing this issue
> https://issues.apache.org/jira/browse/MESOS-5731 and would like to find a
> shepherd for my patch.
>
> Should I go ahead and submit the patch?
>
> Following Vinod's suggestion, I've changed Response::GetMetrics in
> master.proto and v1/master.proto from
>
>   message GetMetrics {
> repeated Metric metrics = 1;
>   }
>
> to
>
>   message GetMetrics {
> repeated Metric counters = 1;
> repeated Metric gauges = 2;
> repeated Metric timers = 3;
>   }
>

[GitHub] mesos pull request #138: Added Yong Tang to contributors list.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/mesos/pull/138


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Replicated Log] Enable Mesos to use etcd for replicated_log

[Joseph Wu]

Jay, I'll shepherd the two detector/contender/network module changes.

On Mon, Jul 11, 2016 at 2:44 AM, Jay JN Guo  wrote:

>
> Shuai,
>
> I think you are right. AFAIK, replicated_log interacts with zk in only
> following two ways:
>
> 1) replicated_log creates znode and maintain 'membership' via
> zookeeper::Group. It actually stores pid under zk_url/log_replicas.
> 2) replicated_log detects other replicas using pid created in previous step
> and store them in set pids [1]. This is done via ZookeeperNetwork.
>
> Other than these, replicas inter-communicate with others through protobuf
> processes. Since there's only one active master at a time, that master will
> be the proposer (coordinator) and others being acceptor. So there is no
> need for leader election in replicated_log.
>
> Please correct me if I'm wrong.
>
> Thanks,
> /J
>
> [1] https://github.com/apache/mesos/blob/master/src/log/network.hpp#L316
>
> Shuai Lin  wrote on 07/11/2016 00:13:12:
>
> > From: Shuai Lin 
> > To: dev 
> > Cc: Jie Yu , Kapil Arya 
> > Date: 07/11/2016 00:13
> > Subject: Re: [Replicated Log] Enable Mesos to use etcd for replicated_log
> >
> > >
> > > i.e. The MasterContender is the piece that decides the "coordinator" of
> the
> > > replicated log.
> >
> >
> > IINM master contender/detector is not related to replicated logs. The
> only
> > thing they have in common (when using zookeeper) is they both get the
> > zookeeper servers list from the `--zk` flag.
> >
> >
> > On Sat, Jul 9, 2016 at 1:54 AM, Joseph Wu  wrote:
> >
> > > Jay,
> > >
> > > (1) Looks like we missed this when we modularized the
> > > MasterDetector/Contender [1].  We need to expand on src/master/main.cpp
> a
> > > bit.
> > > Can you file a bug?  (cc: Kapil)  I can shepherd if Kapil doesn't have
> the
> > > cycles.
> > >
> > > (2) The bit of the replicated log which relies on ZK is a small portion
> > > called the ZookeeperNetwork [2].  The job of this component is to watch
> the
> > > ZK group for membership changes.  Log replication messages are
> broadcasted
> > > to all members in this "network abstraction".
> > > This is also a piece that needs to be modularized.  (Can you file
> another
> > > bug? :)
> > >
> > > (3) The replicated log is something stored locally on the master (i.e.
> > > LevelDB).  The network abstraction has some similarity with the
> > > MasterDetector, but those pieces are otherwise unrelated.
> > > i.e. The MasterContender is the piece that decides the "coordinator" of
> the
> > > replicated log.  But the replicated log uses it's own implementation of
> > > Paxos after the coordinator is chosen.
> > >
> > > [1] https://issues.apache.org/jira/browse/MESOS-4610
> > > [2]
> https://github.com/apache/mesos/blob/master/src/log/network.hpp#L107
> > >
> > > On Fri, Jul 8, 2016 at 9:25 AM, Avinash Sridharan
> 
> > > wrote:
> > >
> > > > +Jie
> > > >
> > > > I think replicated log uses ZK only for leader election. Hence,
> without
> > > ZK
> > > > the quorum is hard-coded to 1.
> > > >
> > > > For (#2), trying to understand what you mean by replicated log being
> > > > pluggable? You mean turning of replicated log on the Master for
> storing
> > > > Registrar information?
> > > >
> > > > On Fri, Jul 8, 2016 at 2:26 AM, Jay JN Guo 
> > > wrote:
> > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > We are working on a Mesos module to substitute Zookeeper with Etcd.
> > > > > Contender and detector are done through modulerized interfaces,
> > > however,
> > > > > replicated_log is still coupled with ZK. Here are my questions:
> > > > >
> > > > > #1 What's the difference between replicated_log with/without ZK?
> > > Without
> > > > > flag --zk, Log is constructed with hardcoded quorum of 1. Does it
> > > assume
> > > > > master to be running in non-HA mode? Otherwise, we observed that
> znodes
> > > > are
> > > > > created in ZK to store log_replica information, does it help Paxos
> > > > > coordination in some way?
> > > > > #2 We hope to make replicated_log pluggable. Some code change need
> to
> > > > > happen in Mesos upstream (interface modulerization, extra flags,
> etc).
> > > So
> > > > > we wonder if someone could shepherd them? Also, it would be great
> if we
> > > > > could get some help on better understanding replicated_log
> internals.
> > > > > #3 Is there a plan to use replicated_log to do master
> contend/detect
> > > > > instead of ZK? If yes, what's the status?
> > > > >
> > > > > Your help and suggestions are highly appreciated!!
> > > > >
> > > > > Thanks,
> > > > > /Jay
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Avinash Sridharan, Mesosphere
> > > > +1 (323) 702 5245
> > > >
> > >
>

Re: Coverity Scan: Analysis completed for Mesos

[Jörg Schad]

ASF CI: Yes that would be the goal.
Frequency: There is a limit to the number of submissions for the free OSS
Coverity scan: https://scan.coverity.com/faq#frequency. Happy to run that
daily, though...

On Mon, Jul 11, 2016 at 7:28 PM, Vinod Kone  wrote:

> On Mon, Jul 11, 2016 at 6:35 AM, Jörg Schad  wrote:
>
> > b) Furthermore, we will set up a weekly Coverity build for HEAD (and
> maybe
> > we should also consider runs for RCs and releases).
> >
>
> I'm guessing this will be on ASF CI? Also, why weekly instead of
> continuously?
>

[GitHub] mesos pull request #138: Added Yong Tang to contributors list.

[yongtang]

GitHub user yongtang opened a pull request:

https://github.com/apache/mesos/pull/138

Added Yong Tang to contributors list.

This PR added Yong Tang to contributors list.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yongtang/mesos add-yong-tang-contributor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/mesos/pull/138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #138


commit 61db91144b47661d55c9bcd91b3594005a4f4869
Author: Yong Tang 
Date:   2016-07-11T18:07:47Z

Added Yong Tang to contributors list.

This PR added Yong Tang to contributors list.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: getting added to contributors

[Lawrence Wu]

Thanks Vinod! Done.

On Mon, Jul 11, 2016 at 10:31 AM, Vinod Kone  wrote:

> Welcome to the community!
>
> Mind sending a PR to add yourself to
> https://github.com/apache/mesos/blob/master/docs/contributors.yaml ?
>
> On Mon, Jul 11, 2016 at 10:28 AM, Lawrence Wu
>  > wrote:
>
> > Hi, I will be working on
> https://issues.apache.org/jira/browse/MESOS-5376.
> > idownes already added me as a contributor but I'm sending this email just
> > for reference.
> >
>

Re: getting added to contributors

[Vinod Kone]

Welcome to the community!

Mind sending a PR to add yourself to
https://github.com/apache/mesos/blob/master/docs/contributors.yaml ?

On Mon, Jul 11, 2016 at 10:28 AM, Lawrence Wu  wrote:

> Hi, I will be working on https://issues.apache.org/jira/browse/MESOS-5376.
> idownes already added me as a contributor but I'm sending this email just
> for reference.
>

Re: Coverity Scan: Analysis completed for Mesos

[Vinod Kone]

On Mon, Jul 11, 2016 at 6:35 AM, Jörg Schad  wrote:

> b) Furthermore, we will set up a weekly Coverity build for HEAD (and maybe
> we should also consider runs for RCs and releases).
>

I'm guessing this will be on ASF CI? Also, why weekly instead of
continuously?

getting added to contributors

[Lawrence Wu]

Hi, I will be working on https://issues.apache.org/jira/browse/MESOS-5376.
idownes already added me as a contributor but I'm sending this email just
for reference.

Re: [VOTE] Release Apache Mesos 1.0.0 (rc2)

[Kapil Arya]

None of the stable builds have SSL yet. The first SSL-enabled stable build
will be 1.0.0. Sorry for the confusion.

Kapil

On Mon, Jul 11, 2016 at 1:03 PM, Zhitao Li  wrote:

> Hi Kapil,
>
> Do you mean that the stable builds from
> http://open.mesosphere.com/downloads/mesos is using the new configuration?
>
> On Sun, Jul 10, 2016 at 10:07 AM, Kapil Arya  wrote:
>
>> The binary rpm/deb packages can be found here:
>> http://open.mesosphere.com/downloads/mesos-rc/#apache-mesos-1.0.0-rc2
>> .
>>
>> Please note that starting with the 1.0.0 release (including RCs and
>> recent nightly builds), Mesos is configured with SSL and 3rdparty
>> module dependency installation. Here is the configure command line:
>> ./configure --enable-libevent --enable-ssl
>> --enable-install-module-dependencies
>>
>> As always, the stable builds are available at:
>> http://open.mesosphere.com/downloads/mesos
>>
>> The instructions for nightly builds are available at:
>> http://open.mesosphere.com/downloads/mesos-nightly/
>>
>> Best,
>> Kapil
>>
>>
>> On Thu, Jul 7, 2016 at 9:35 PM, Vinod Kone  wrote:
>> >
>> > Hi all,
>> >
>> >
>> > Please vote on releasing the following candidate as Apache Mesos 1.0.0.
>> >
>> >
>> > 1.0.0 includes the following:
>> >
>> >
>> 
>> >
>> >   * Scheduler and Executor v1 HTTP APIs are now considered stable.
>> >
>> >
>> >
>> >
>> >
>> >   * [MESOS-4791] - **Experimental** support for v1 Master and Agent
>> APIs.
>> > These
>> >
>> > APIs let operators and services (monitoring, load balancers) send
>> HTTP
>> >
>> >
>> > requests to '/api/v1' endpoint on master or agent. See
>> >
>> >
>> > `docs/operator-http-api.md` for details.
>> >
>> >
>> >
>> >
>> >
>> >   * [MESOS-4828] - **Experimental** support for a new `disk/xfs'
>> isolator
>> >
>> >
>> > has been added to isolate disk resources more efficiently. Please
>> refer
>> > to
>> >
>> > docs/mesos-containerizer.md for more details.
>> >
>> >
>> >
>> >
>> >
>> >   * [MESOS-4355] - **Experimental** support for Docker volume plugin. We
>> > added a
>> >
>> > new isolator 'docker/volume' which allows users to use external
>> volumes
>> > in
>> >
>> > Mesos containerizer. Currently, the isolator interacts with the
>> Docker
>> >
>> >
>> > volume plugins using a tool called 'dvdcli'. By speaking the Docker
>> > volume
>> >
>> > plugin API, most of the Docker volume plugins are supported.
>> >
>> >
>> >
>> >
>> >
>> >   * [MESOS-4641] - **Experimental** A new network isolator, the
>> >
>> >
>> > `network/cni` isolator, has been introduced in the
>> > `MesosContainerizer`. The
>> >
>> > `network/cni` isolator implements the Container Network Interface
>> (CNI)
>> >
>> >
>> > specification proposed by CoreOS.  With CNI the `network/cni`
>> isolator
>> > is
>> >
>> > able to allocate a network namespace to Mesos containers and attach
>> the
>> >
>> >
>> > container to different types of IP networks by invoking network
>> drivers
>> >
>> >
>> > called CNI plugins.
>> >
>> >
>> >
>> >
>> >
>> >   * [MESOS-2948, MESOS-5403] - The authorizer interface has been
>> refactored
>> > in
>> >
>> > order to decouple the ACLs definition language from the interface.
>> >
>> >
>> > It additionally includes the option of retrieving `ObjectApprover`.
>> An
>> >
>> >
>> > `ObjectApprover` can be used to synchronously check authorizations
>> for
>> > a
>> >
>> > given object and is hence useful when authorizing a large number of
>> > objects
>> >
>> > and/or large objects (which need to be copied using request based
>> >
>> >
>> > authorization). NOTE: This is a **breaking change** for authorizer
>> > modules.
>> >
>> >
>> >
>> >
>> >   * [MESOS-5405] - The `subject` and `object` fields in
>> > authorization::Request
>> >
>> > have been changed from required to optional. If either of these
>> fields
>> > is
>> >
>> > not set, the request should only be authorized if any subject/object
>> > should
>> >
>> > be allowed.
>> >
>> > NOTE: This is a semantic change for authorizer modules.
>> >
>> >
>> >
>> >
>> >
>> >   * [MESOS-4931, MESOS-5709, MESOS-5704] - Authorization based HTTP
>> > endpoint
>> >
>> > filtering enables operators to restrict what part of the cluster
>> state
>> > a
>> >
>> > user is authorized to see.
>> >
>> >
>> > Consider for example the `/state` master endpoint: an operator can
>> now
>> >
>> >
>> > authorize users to only see a subset of the running frameworks,
>> tasks,
>> > or
>> >
>> > executors. The following endpoints support HTTP endpoint filtering:
>> >
>> >
>> > '/state', '/state-summary', '/tasks', '/frameworks','/weights',
>> >
>> >
>> > and '/roles'. Additonally the following v1 API calls support
>> filtering:
>> >
>> >
>> > 

Re: [VOTE] Release Apache Mesos 1.0.0 (rc2)

[Zhitao Li]

Hi Kapil,

Do you mean that the stable builds from
http://open.mesosphere.com/downloads/mesos is using the new configuration?

On Sun, Jul 10, 2016 at 10:07 AM, Kapil Arya  wrote:

> The binary rpm/deb packages can be found here:
> http://open.mesosphere.com/downloads/mesos-rc/#apache-mesos-1.0.0-rc2.
>
> Please note that starting with the 1.0.0 release (including RCs and
> recent nightly builds), Mesos is configured with SSL and 3rdparty
> module dependency installation. Here is the configure command line:
> ./configure --enable-libevent --enable-ssl
> --enable-install-module-dependencies
>
> As always, the stable builds are available at:
> http://open.mesosphere.com/downloads/mesos
>
> The instructions for nightly builds are available at:
> http://open.mesosphere.com/downloads/mesos-nightly/
>
> Best,
> Kapil
>
>
> On Thu, Jul 7, 2016 at 9:35 PM, Vinod Kone  wrote:
> >
> > Hi all,
> >
> >
> > Please vote on releasing the following candidate as Apache Mesos 1.0.0.
> >
> >
> > 1.0.0 includes the following:
> >
> >
> 
> >
> >   * Scheduler and Executor v1 HTTP APIs are now considered stable.
> >
> >
> >
> >
> >
> >   * [MESOS-4791] - **Experimental** support for v1 Master and Agent APIs.
> > These
> >
> > APIs let operators and services (monitoring, load balancers) send
> HTTP
> >
> >
> > requests to '/api/v1' endpoint on master or agent. See
> >
> >
> > `docs/operator-http-api.md` for details.
> >
> >
> >
> >
> >
> >   * [MESOS-4828] - **Experimental** support for a new `disk/xfs' isolator
> >
> >
> > has been added to isolate disk resources more efficiently. Please
> refer
> > to
> >
> > docs/mesos-containerizer.md for more details.
> >
> >
> >
> >
> >
> >   * [MESOS-4355] - **Experimental** support for Docker volume plugin. We
> > added a
> >
> > new isolator 'docker/volume' which allows users to use external
> volumes
> > in
> >
> > Mesos containerizer. Currently, the isolator interacts with the
> Docker
> >
> >
> > volume plugins using a tool called 'dvdcli'. By speaking the Docker
> > volume
> >
> > plugin API, most of the Docker volume plugins are supported.
> >
> >
> >
> >
> >
> >   * [MESOS-4641] - **Experimental** A new network isolator, the
> >
> >
> > `network/cni` isolator, has been introduced in the
> > `MesosContainerizer`. The
> >
> > `network/cni` isolator implements the Container Network Interface
> (CNI)
> >
> >
> > specification proposed by CoreOS.  With CNI the `network/cni`
> isolator
> > is
> >
> > able to allocate a network namespace to Mesos containers and attach
> the
> >
> >
> > container to different types of IP networks by invoking network
> drivers
> >
> >
> > called CNI plugins.
> >
> >
> >
> >
> >
> >   * [MESOS-2948, MESOS-5403] - The authorizer interface has been
> refactored
> > in
> >
> > order to decouple the ACLs definition language from the interface.
> >
> >
> > It additionally includes the option of retrieving `ObjectApprover`.
> An
> >
> >
> > `ObjectApprover` can be used to synchronously check authorizations
> for
> > a
> >
> > given object and is hence useful when authorizing a large number of
> > objects
> >
> > and/or large objects (which need to be copied using request based
> >
> >
> > authorization). NOTE: This is a **breaking change** for authorizer
> > modules.
> >
> >
> >
> >
> >   * [MESOS-5405] - The `subject` and `object` fields in
> > authorization::Request
> >
> > have been changed from required to optional. If either of these
> fields
> > is
> >
> > not set, the request should only be authorized if any subject/object
> > should
> >
> > be allowed.
> >
> > NOTE: This is a semantic change for authorizer modules.
> >
> >
> >
> >
> >
> >   * [MESOS-4931, MESOS-5709, MESOS-5704] - Authorization based HTTP
> > endpoint
> >
> > filtering enables operators to restrict what part of the cluster
> state
> > a
> >
> > user is authorized to see.
> >
> >
> > Consider for example the `/state` master endpoint: an operator can
> now
> >
> >
> > authorize users to only see a subset of the running frameworks,
> tasks,
> > or
> >
> > executors. The following endpoints support HTTP endpoint filtering:
> >
> >
> > '/state', '/state-summary', '/tasks', '/frameworks','/weights',
> >
> >
> > and '/roles'. Additonally the following v1 API calls support
> filtering:
> >
> >
> > 'GET_ROLES','GET_WEIGHTS','GET_FRAMEWORKS', 'GET_STATE', and
> > 'GET_TASKS'.
> >
> >
> >
> >
> >   * [MESOS-4909] - Tasks can now specify a kill policy. They are
> > best-effort,
> >
> > because machine failures or forcible terminations may occur.
> Currently,
> > the
> >
> > only available kill policy is how long to wait between graceful and
> > forcible
> >
> > task kill. In the future, more policies may be available 

Accepted: Invitation: Community Meeting @ Fri Jul 15, 2016 1:30am - 2:30am (IST) (machsagel2...@gmail.com)

[Deepak Vij (A)]

BEGIN:VCALENDAR
METHOD:REPLY
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ATTENDEE;PARTSTAT=ACCEPTED;CN=Deepak Vij (A):MAILTO:deepak@huawei.com
SUMMARY;LANGUAGE=en-US:Accepted: Invitation: Community Meeting @ Fri Jul 15
 \, 2016 1:30am - 2:30am (IST) (machsagel2...@gmail.com)
DTSTART;TZID=Pacific Standard Time:20160714T13
DTEND;TZID=Pacific Standard Time:20160714T14
UID:cli3ic1i6gp62b9m64ojcb9k6cp3ebb1cphm4b9n65ij8chk68ojep1...@google.com
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20160711T170042Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:0
LOCATION;LANGUAGE=en-US:
X-MICROSOFT-CDO-APPT-SEQUENCE:0
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:0
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
END:VCALENDAR

Accepted: Invitation: Community Meeting @ Thu Jul 14, 2016 10pm - 11pm (CEST) (Apache Mesos)

[Deepak Vij (A)]

BEGIN:VCALENDAR
METHOD:REPLY
PRODID:Microsoft Exchange Server 2010
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Pacific Standard Time
BEGIN:STANDARD
DTSTART:16010101T02
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=1SU;BYMONTH=11
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:16010101T02
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
RRULE:FREQ=YEARLY;INTERVAL=1;BYDAY=2SU;BYMONTH=3
END:DAYLIGHT
END:VTIMEZONE
BEGIN:VEVENT
ATTENDEE;PARTSTAT=ACCEPTED;CN=Deepak Vij (A):MAILTO:deepak@huawei.com
SUMMARY;LANGUAGE=en-US:Accepted: Invitation: Community Meeting @ Thu Jul 14
 \, 2016 10pm - 11pm (CEST) (Apache Mesos)
DTSTART;TZID=Pacific Standard Time:20160714T13
DTEND;TZID=Pacific Standard Time:20160714T14
UID:vavklkdmf4hlonjarsofg8l...@google.com
RECURRENCE-ID;TZID=Pacific Standard Time:20160714T00
CLASS:PUBLIC
PRIORITY:5
DTSTAMP:20160711T165556Z
TRANSP:OPAQUE
STATUS:CONFIRMED
SEQUENCE:4
LOCATION;LANGUAGE=en-US:
X-MICROSOFT-CDO-APPT-SEQUENCE:4
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY
X-MICROSOFT-CDO-ALLDAYEVENT:FALSE
X-MICROSOFT-CDO-IMPORTANCE:1
X-MICROSOFT-CDO-INSTTYPE:0
X-MICROSOFT-DISALLOW-COUNTER:FALSE
END:VEVENT
END:VCALENDAR

RFC: partitioned tasks and the strict registry

[Neil Conway]

Folks,

We're working on some Mesos features that will allow frameworks to
control how partitioned tasks are handled [1]. As part of designing
how this will work, I'd love to hear from users and framework
developers about they handle partitioned tasks/agents. Specifically:

(a) Have you enabled the strict registry? ('--registry_strict' master flag)

(b) If so, do any of your frameworks _depend_ on the semantics
provided by the strict registry? [2]

(c) Does your framework handle LOST tasks? For example, does your
framework account for the fact that LOST tasks might transition back
to RUNNING in certain circumstances?

(d) Suppose we changed the semantics of LOST in the following way: (1)
strict registry is no longer supported, and (2) LOST tasks will
*always* be allowed to reregister with the master and resume running
(even if the master has not failed over). Would this change cause
problems for any of your frameworks?

Answering "I don't know" to any of these questions is fine :) Feel
free to respond to me privately if you'd prefer.

If you have any other feedback or questions, please contact me.

Thanks!

Neil

[1] More information on the proposed changes can be found here:
https://goo.gl/7dRw4Q

[2] e.g., your framework assumes that LOST tasks will never go back to RUNNING.

Fwd: Coverity Scan: Analysis completed for Mesos

[Jörg Schad]

After Coverity upgraded their build system this weekend (
https://twitter.com/CoverityScan/status/751106512097914880)  we can finally
use Coverity again for static code analyses!
a) There are many non-relevant issues right now (but Coverity is able to
learn and ignore them for future builds), Benjamin Bannier and I will go
through the list, open Jira issues (label: coverity) for the relevant ones
and discard the non-relevant ones. Please be patient and wait for us to do
so, otherwise, we might be duplicating work.
b) Furthermore, we will set up a weekly Coverity build for HEAD (and maybe
we should also consider runs for RCs and releases).

Joerg
P.S. In order to login into the Coverity project, you need to request
access. Feel free to ping me if you experience issues.


-- Forwarded message --
From: 
Date: Mon, Jul 11, 2016 at 3:09 PM
Subject: Coverity Scan: Analysis completed for Mesos
To: jo...@mesosphere.io


Your request for analysis of Mesos has been completed successfully.
The results are available at https://scan.coverity.com/projects/mesos

Analysis Summary:
   New defects found: 288
   Defects eliminated: 0

 If you have difficulty understading any defects, email us at
scan-ad...@coverity.com,
 or post your question to the Coverity Community
 at https://communities.coverity.com/community/scan-(open-source)/content

[GitHub] mesos pull request #136: Add Jacob Janco to contributors and remove whitespa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/mesos/pull/136


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] mesos pull request #137: Add Haosdent Huang to contributors.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/mesos/pull/137


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Request to be added to mesos contributors

[Alex Rukletsov]

Done. Welcome!

On Sat, Jul 9, 2016 at 1:02 AM, Tim Harper  wrote:

> I've submitted and worked on this issue:
> https://issues.apache.org/jira/browse/MESOS-5824 <
> https://issues.apache.org/jira/browse/MESOS-5824>
>
> Per the instructions <
> http://mesos.apache.org/documentation/latest/submitting-a-patch/> I'd
> like to be added to the contributors list so I can assign myself to the
> issue.
>
> Thanks
>
> Tim

Re: MESOS: Improve the performance of Resources.

[Klaus Ma]

Hi Joris,

For `Scalars`, yes, it's also dynamic allocated in
`Resources::mutable_scalar()`.

For `2x` number, do you have any patch to share? Tested cases this
afternoon: add resources including 100 roles (1 CPU for each); the
performance is downgrade a lot; so I agree with you to improve some
algorithms in Resources/Sorter.

For 'basic benchmarks', temporary tracking in my personal github (
https://github.com/k82cn/mesos/blob/resources_benchmark/src/tests/resources_tests.cpp).
The following cases are in my mind to add:
1. simple resources, e.g. 1 cpu
2. resources with port, e.g. [1-2], [3-4], ... [101-102]
3. resources with reservation, cpus(r1):1;cpus(r2):1;  cpus(r10):1
4. resources with diskInfo
5. resources with revocableInfo

The operators will be +, +=, -, -=, cpus(), contains.

I booked a weekly call to discuss for allocaiton performance and sent the
invitation to the dev@.

If any comments, please let me know.


Da (Klaus), Ma (马达) | PMP® | Software Architect
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Mon, Jul 11, 2016 at 3:28 PM, Joris Van Remoortere 
wrote:

> +Dev
>
> Hey Klaus,
>
> Using Stout's `Optional` to represent the `optional` concept of a message
> in protobuf is definitely a step in the right direction.
> Regarding your comment in slack yesterday: From my version of the protobuf
> generated code there definitely is dynamic allocation even for scalars.
>
> It looks like in our case there is a minimum of 3 dynamic allocations per
> Resource object:
>
>> void Resource::SharedDtor() {
>>   if (name_ != &::google::protobuf::internal::kEmptyString) {
>> delete name_;
>>   }
>>   if (role_ != _default_role_) {
>> delete role_;
>>   }
>>   if (this != default_instance_) {
>> delete scalar_;
>> delete ranges_;
>> delete set_;
>> delete reservation_;
>> delete disk_;
>> delete revocable_;
>>   }
>> }
>
>
>  The 2x number I mentioned came from running some of the existing
> benchmarks. I didn't explore further because it didn't have as big an
> impact as I had hoped. The first battle is simplifying some of the
> algorithms in the Sorter / Resources. Once that is done then the resource
> arithmetic will be more of a bounding factor.
>
> I agree with Ben that we should focus on writing some basic benchmarks
> that represent the common uses of Resources in the allocator. We should
> scale these benchmarks to represent some of the more stressful environments
> that could occur. For example, had we had such a benchmark, we would have
> realized much earlier on that we needed to aggregate only quantities in the
> Sorter, and that using the existing form of Resources would have led to a
> grinding halt if a reservation were made on every machine.
>
> Is there a regular call that is scheduled to discuss this? I think there
> are some other folks also working on benchmarks and interested in the
> discussion.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma  wrote:
>
>> + more devs :).
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma  wrote:
>>
>>> Hi Joris,
>>>
>>> I think `Option` is helpful to the performance improvement, it used
>>> `placement new` to avoid dynamic allocation. Suppose you're using Option
>>> for optional member in protobuf, and using class instance directly
>>> (operator=).
>>>
>>> I'm adding some benchmark for `Resources`, especially for the
>>> `Resources` with Rang, DiskInfo and ReservationInfo
>>>
>>> Draft PR for Benchmark of Resources:
>>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d68a8cf2249eb641c
>>>
>>>
>>> 
>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>> Platform OpenSource Technology, STG, IBM GCG
>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>>
>>
>>
>

[GitHub] mesos issue #127: added DevOps category and Vamp

[jmlvanre]

Github user jmlvanre commented on the issue:

https://github.com/apache/mesos/pull/127
  
It will get added to the website the next time it gets published.
We do that pretty regularly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Invitation: Community Meeting @ Fri Jul 15, 2016 1:30am - 2:30am (IST) (machsagel2...@gmail.com)

[mach sagel]

BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REQUEST
BEGIN:VEVENT
DTSTART:20160714T20Z
DTEND:20160714T21Z
DTSTAMP:20160711T011909Z
ORGANIZER;CN=mach sagel:mailto:machsagel2...@gmail.com
UID:cli3ic1i6gp62b9m64ojcb9k6cp3ebb1cphm4b9n65ij8chk68ojep1...@google.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE
 ;CN=mach sagel;X-NUM-GUESTS=0:mailto:machsagel2...@gmail.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP=
 TRUE;CN=dev@mesos.apache.org;X-NUM-GUESTS=0:mailto:dev@mesos.apache.org
CREATED:20160711T011909Z
DESCRIPTION:View your event at https://www.google.com/calendar/event?action
 =VIEW=Y2xpM2ljMWk2Z3A2MmI5bTY0b2pjYjlrNmNwM2ViYjFjcGhtNGI5bjY1aWo4Y2hrN
 jhvamVwMWc2MCBkZXZAbWVzb3MuYXBhY2hlLm9yZw=MjMjbWFjaHNhZ2VsMjgxMUBnbWFpb
 C5jb21jNmI1NTQ2NTNkN2JhNWRkN2VjZWU4ZmFkNzMyMGEyNGE1NDNlMzYz=Asia/Calcut
 ta=en.
LAST-MODIFIED:20160711T011909Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Community Meeting
TRANSP:OPAQUE
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:This is an event reminder
TRIGGER:-P0DT0H30M0S
END:VALARM
END:VEVENT
END:VCALENDAR


invite.ics
Description: application/ics

[GitHub] mesos issue #127: added DevOps category and Vamp

[gggina]

Github user gggina commented on the issue:

https://github.com/apache/mesos/pull/127
  
Thanks.
How does this get added to 
http://mesos.apache.org/documentation/latest/frameworks/ ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: MESOS: Improve the performance of Resources.

[Joris Van Remoortere]

+Dev

Hey Klaus,

Using Stout's `Optional` to represent the `optional` concept of a message
in protobuf is definitely a step in the right direction.
Regarding your comment in slack yesterday: From my version of the protobuf
generated code there definitely is dynamic allocation even for scalars.

It looks like in our case there is a minimum of 3 dynamic allocations per
Resource object:

> void Resource::SharedDtor() {
>   if (name_ != &::google::protobuf::internal::kEmptyString) {
> delete name_;
>   }
>   if (role_ != _default_role_) {
> delete role_;
>   }
>   if (this != default_instance_) {
> delete scalar_;
> delete ranges_;
> delete set_;
> delete reservation_;
> delete disk_;
> delete revocable_;
>   }
> }


 The 2x number I mentioned came from running some of the existing
benchmarks. I didn't explore further because it didn't have as big an
impact as I had hoped. The first battle is simplifying some of the
algorithms in the Sorter / Resources. Once that is done then the resource
arithmetic will be more of a bounding factor.

I agree with Ben that we should focus on writing some basic benchmarks that
represent the common uses of Resources in the allocator. We should scale
these benchmarks to represent some of the more stressful environments that
could occur. For example, had we had such a benchmark, we would have
realized much earlier on that we needed to aggregate only quantities in the
Sorter, and that using the existing form of Resources would have led to a
grinding halt if a reservation were made on every machine.

Is there a regular call that is scheduled to discuss this? I think there
are some other folks also working on benchmarks and interested in the
discussion.

—
*Joris Van Remoortere*
Mesosphere

On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma  wrote:

> + more devs :).
>
> 
> Da (Klaus), Ma (马达) | PMP® | Software Architect
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma  wrote:
>
>> Hi Joris,
>>
>> I think `Option` is helpful to the performance improvement, it used
>> `placement new` to avoid dynamic allocation. Suppose you're using Option
>> for optional member in protobuf, and using class instance directly
>> (operator=).
>>
>> I'm adding some benchmark for `Resources`, especially for the `Resources`
>> with Rang, DiskInfo and ReservationInfo
>>
>> Draft PR for Benchmark of Resources:
>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d68a8cf2249eb641c
>>
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>
>