Re: Feature requests for Mesos

[Klaus Ma]

> - willingness of remaining active committers to be active on a regular
> basis in engagements with the community, both on the user and contributor
> side (in PRs, review requests, on mailing lists),
> - transparent and active discussions in the community, among committers
and
> contributors, and among committers, in applicable form, beyond roll calls,
> - timely and consistent process to address user issues, and
> - consistent ownership of the bug and feature backlog.

How can we get more contributors or committers to help on those items?



On Tue, Mar 2, 2021 at 2:13 AM Charles-François Natali 
wrote:

> I couldn't agree more.
>
>
>
> On Mon, 1 Mar 2021, 15:08 Benjamin Bannier,  wrote:
>
> > Hi Charles-François,
> >
> > thanks for your detailed message, you captured important points, and I
> > think I agree with your sentiment here. Mesos might still have a place,
> and
> > before thinking about what new features to add, the project first needs
> to
> > solve more fundamental issues.
> >
> > My previous pessimistic assessment on this list came from a similar angle
> > but I think with wider scope: a healthy project requires a healthy
> > community where users can find help, but also can have some hope that
> > important issues will get fixed. I have not been able to spend much time
> on
> > Mesos in the last year, but was following Slack and the mailing lists
> (the
> > ones with humans and the ones with bots). On the mailing lists I see
> users
> > ask for help with issues they run into or questions, but only rarely will
> > get a response from committers or other community members. Few new JIRA
> > issues were filed in the since fall 2020, but hardly any of them have
> been
> > triaged let alone fixed (this is on top of the existing bug backlog). I
> do
> > not think one needs to be a committer to improve on that situation if one
> > can get help getting patches discussed, reviewed and ultimately merged.
> It
> > looks like Andrei and Qian have committed to help on the latter, but I
> have
> > only rarely seen community members volunteer for the former.
> >
> > When I wrote that I thought starting a new project on top of Apache Mesos
> > today might be not a good idea, I mainly came from that angle. While the
> > software does work for many use cases it seems to be unmaintained with
> > hardly any folks active in taking it further globally, beyond their own
> > immediate needs, and willing to take on the needed work. Being a
> top-level
> > Apache project with a strong history, Apache Mesos still has a brand,
> but I
> > don't think it has lived up to the associated expectations. Similarly,
> big
> > ownership gaps (technical and project-wise) have developed which neither
> > active committers nor community members have filled. Again, one would not
> > need to be a committer to develop expertise and contribute, and actually
> > the natural and historic process was for folks to do exactly that with
> > committership being a thing only after getting involved (see
> > https://community.apache.org/newcommitter.html for Apache's high-level
> > view
> > on that). This is the issue of continued trust Renan mentioned in their
> > message to the user mailing list which I also believe is critical so the
> > project can live up to its promise (this is integral to being an Apache
> > project, see e.g., https://www.apache.org/theapacheway).
> >
> > As a non-user with emotional attachment to the historic Apache Mesos
> brand,
> > my list of areas in need of improvement to resurrect this project would
> be:
> >
> > - willingness of remaining active committers to be active on a regular
> > basis in engagements with the community, both on the user and contributor
> > side (in PRs, review requests, on mailing lists),
> > - transparent and active discussions in the community, among committers
> and
> > contributors, and among committers, in applicable form, beyond roll
> calls,
> > - timely and consistent process to address user issues, and
> > - consistent ownership of the bug and feature backlog.
> >
> > Note that work on new feature requests is absent from my list. That folks
> > want to discuss that here and now seems to me to be another sign that the
> > Mesos community is not in a good place given all its existing
> non-technical
> > issues.
> >
> >
> > Best,
> >
> > Benjamin
> >
>

Re: Feature requests for Mesos

[Klaus Ma]

Mesos is really a great project!

In addition to new features, is it possible to make Mesos focus on some 
specific area, e.g. HPC?

-- Klaus


From: Damien GERARD 
Sent: Monday, March 1, 2021 1:05 AM
To: u...@mesos.apache.org 
Cc: mesos 
Subject: Re: Feature requests for Mesos

On 2021-02-28 05:38 PM, Qian Zhang wrote:
> Hi Folks,
>
> To reboot this awesome project, I'd like to collect feature requests
> for Mesos. Please let us know your requirements for Mesos and whether
> you or your organization would like to contribute to the
> implementation of the requirements. Thanks!

We can already summarize what has been said in previous discussions,
mainly
making mesos easier to deploy / maintain, and my personal one, with more
batteries included by default.
There are also several ways of improvements, especially regarding GPU
support,
numa, ZFS integration, volume management...

But first, reducing the effort to have a fully functional mesos cluster
would be to me, one of the priorities (and also targeting local
development
like minimesos was trying to achieve)


--
Damien GERARD

Re: Welcome Andrei Sekretenko as a new committer and PMC member!

[Klaus Ma]

Congratulations!

On Wed, Jan 22, 2020 at 5:42 AM Benjamin Mahler  wrote:

> Please join me in welcoming Andrei Sekretenko as the newest committer and
> PMC member!
>
> Andrei has been active in the project for almost a year at this point and
> has been a productive and collaborative member of the community.
>
> He has helped out a lot with allocator work, both with code and
> investigations of issues. He made improvements to multi-role framework
> scalability (which includes the addition of the UPDATE_FRAMEWORK call), and
> exposed metrics for per-role quota consumption.
>
> He has also investigated, identified, and followed up on important bugs.
> One such example is the message re-ordering issue he is currently working
> on: https://issues.apache.org/jira/browse/MESOS-10023
>
> Thanks for all your work so far Andrei, I'm looking forward to more of your
> contributions in the project.
>
> Ben
>

Re: Welcome Benno Evers as committer and PMC member!

[Klaus Ma]

Congratulations!

-- Klaus


On Thu, Jan 31, 2019 at 7:31 PM Andrei Budnik  wrote:

> Congratulations!
>
> On Thu, Jan 31, 2019 at 2:41 AM Benjamin Mahler 
> wrote:
>
>> Welcome Benno! Thanks for all the great contributions
>>
>> On Wed, Jan 30, 2019 at 6:21 PM Alex R  wrote:
>>
>> > Folks,
>> >
>> > Please welcome Benno Evers as an Apache committer and PMC member of the
>> > Apache Mesos!
>> >
>> > Benno has been active in the project for more than a year now and has
>> made
>> > significant contributions, including:
>> >   * Agent reconfiguration, MESOS-1739
>> >   * Memory profiling, MESOS-7944
>> >   * "/state" performance improvements, MESOS-8345
>> >
>> > I have been working closely with Benno, paired up on, and shepherded
>> some
>> > of his work. Benno has very strong technical knowledge in several areas
>> and
>> > he is willing to share it with others and help his peers.
>> >
>> > Benno, thanks for all your contributions so far and looking forward to
>> > continuing to work with you on the project!
>> >
>> > Alex.
>> >
>>
>

Re: Welcome Andrew Schwartzmeyer as a new committer and PMC member!

[Klaus Ma]

Congratulations

On Tue, Nov 28, 2017 at 3:08 PM Andrew Schwartzmeyer <
and...@schwartzmeyer.com> wrote:

> Thank you everyone for the welcome!
>
> It's been great working with you this past year, and I'm glad to
> continue making this great project even better.
>
> Thanks again,
>
> Andy
>
> On 11/27/2017 3:00 pm, Joseph Wu wrote:
> > Hi devs & users,
> >
> > I'm happy to announce that Andrew Schwartzmeyer has become a new
> > committer
> > and member of the PMC for the Apache Mesos project.  Please join me in
> > congratulating him!
> >
> > Andrew has been an active contributor to Mesos for about a year.  He
> > has
> > been the primary contributor behind our efforts to change our default
> > build
> > system to CMake and to port Mesos onto Windows.
> >
> > Here is his committer candidate checklist for your perusal:
> > https://docs.google.com/document/d/1MfJRYbxxoX2-A-
> > g8NEeryUdUi7FvIoNcdUbDbGguH1c/
> >
> > Congrats Andy!
> > ~Joseph
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: Welcome Greg Mann as a new committer and PMC member!

[Klaus Ma]

Congrats!


> On 14 Jun 2017, at 06:29, Ben Lin  wrote:
> 
> Congrats Greg, well deserved!
> 
> From: Jie Yu 
> Sent: Wednesday, June 14, 2017 5:54:48 AM
> To: user
> Cc: dev
> Subject: Re: Welcome Greg Mann as a new committer and PMC member!
>  
> Congrats Greg!
> 
> On Tue, Jun 13, 2017 at 2:42 PM, Vinod Kone  > wrote:
> Hi folks,
> 
> Please welcome Greg Mann as the newest committer and PMC member of the Apache 
> Mesos project.
> 
> Greg has been an active contributor to the Mesos project for close to 2 years 
> now and has made many solid contributions. His biggest source code 
> contribution to the project has been around adding authentication support for 
> default executor. This was a major new feature that involved quite a few 
> moving parts. Additionally, he also worked on improving the scheduler and 
> executor APIs.
> 
> Here is his more formal checklist for your perusal.
> 
> https://docs.google.com/document/d/1S6U5OFVrl7ySmpJsfD4fJ3_R8JYRRc5spV0yKrpsGBw/edit
>  
> 
> 
> Thanks,
> Vinod
> 
> 

Re: Welcome Gilbert Song as a new committer and PMC member!

[Klaus Ma]

Congratulations Gilbert!

On Thu, May 25, 2017 at 3:39 AM Greg Mann <g...@mesosphere.io> wrote:

> Congratulations Gilbert!! :D
>
> On Wed, May 24, 2017 at 12:01 PM, Avinash Sridharan <avin...@mesosphere.io
> > wrote:
>
>> Congrats Gilbert !! Very well deserved !!
>>
>> On Wed, May 24, 2017 at 11:56 AM, Timothy Chen <tnac...@gmail.com> wrote:
>>
>> > Congrats! Rocking the containerizer world!
>> >
>> > Tim
>> >
>> > On Wed, May 24, 2017 at 11:23 AM, Zhitao Li <zhitaoli...@gmail.com>
>> wrote:
>> > > Congrats Gilbert!
>> > >
>> > > On Wed, May 24, 2017 at 11:08 AM, Yan Xu <y...@jxu.me> wrote:
>> > >
>> > >> Congrats! Well deserved!
>> > >>
>> > >> ---
>> > >> Jiang Yan Xu <y...@jxu.me> | @xujyan <https://twitter.com/xujyan>
>> > >>
>> > >> On Wed, May 24, 2017 at 10:54 AM, Vinod Kone <vinodk...@apache.org>
>> > wrote:
>> > >>
>> > >>> Congrats Gilbert!
>> > >>>
>> > >>> On Wed, May 24, 2017 at 1:32 PM, Neil Conway <neil.con...@gmail.com
>> >
>> > >>> wrote:
>> > >>>
>> > >>> > Congratulations Gilbert! Well-deserved!
>> > >>> >
>> > >>> > Neil
>> > >>> >
>> > >>> > On Wed, May 24, 2017 at 10:32 AM, Jie Yu <yujie@gmail.com>
>> > wrote:
>> > >>> > > Hi folks,
>> > >>> > >
>> > >>> > > I' happy to announce that the PMC has voted Gilbert Song as a
>> new
>> > >>> > committer
>> > >>> > > and member of PMC for the Apache Mesos project. Please join me
>> to
>> > >>> > > congratulate him!
>> > >>> > >
>> > >>> > > Gilbert has been working on Mesos project for 1.5 years now. His
>> > main
>> > >>> > > contribution is his work on unified containerizer, nested
>> container
>> > >>> (aka
>> > >>> > > Pod) support. He also helped a lot of folks in the community
>> > regarding
>> > >>> > their
>> > >>> > > patches, questions and etc. He also played an important role
>> > >>> organizing
>> > >>> > > MesosCon Asia last year and this year!
>> > >>> > >
>> > >>> > > His formal committer checklist can be found here:
>> > >>> > > https://docs.google.com/document/d/1iSiqmtdX_0CU-YgpViA6r6PU_
>> > >>> > aMCVuxuNUZ458FR7Qw/edit?usp=sharing
>> > >>> > >
>> > >>> > > Welcome, Gilbert!
>> > >>> > >
>> > >>> > > - Jie
>> > >>> >
>> > >>>
>> > >>
>> > >>
>> > >
>> > >
>> > > --
>> > > Cheers,
>> > >
>> > > Zhitao Li
>> >
>>
>>
>>
>> --
>> Avinash Sridharan, Mesosphere
>> +1 (323) 702 5245
>>
>
> --

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: Welcome Guangya Liu as Mesos Committer and PMC member!

[Klaus Ma]

Congratulations!!

On Sat, Dec 17, 2016 at 1:23 PM Dharmesh Kakadia <dhkaka...@gmail.com>
wrote:

> Congrats Guangya !
>
> Thanks,
> Dharmesh
>
> On Fri, Dec 16, 2016 at 5:03 PM, Dario Rexin <dre...@apple.com> wrote:
>
> Congrats!
>
> > On Dec 16, 2016, at 4:27 PM, Vinod Kone <vinodk...@apache.org> wrote:
> >
> > Congrats Guangya! Welcome to the PMC!
> >
> >> On Fri, Dec 16, 2016 at 7:03 PM, Sam <usultra...@gmail.com> wrote:
> >> congratulations Guangya
> >>
> >> Sent from my iPhone
> >>
> >>> On 17 Dec 2016, at 3:23 AM, Avinash Sridharan <avin...@mesosphere.io>
> wrote:
> >>>
> >>> Congrats Guangya !!
> >>>
> >>>> On Fri, Dec 16, 2016 at 11:20 AM, Greg Mann <g...@mesosphere.io>
> wrote:
> >>>> Congratulations Guangya!!! :D
> >>>>
> >>>>> On Fri, Dec 16, 2016 at 11:10 AM, Jie Yu <yujie@gmail.com>
> wrote:
> >>>>> Hi folks,
> >>>>>
> >>>>> Please join me in formally welcoming Guangya Liu as Mesos Committer
> and PMC
> >>>>> member.
> >>>>>
> >>>>> Guangya has worked on the project for more than a year now and has
> been a
> >>>>> very active contributor to the project. I think one of the most
> important
> >>>>> contribution he has for the community is that he helped grow the
> Mesos
> >>>>> community in China. He initiated the Xian-Mesos-User-Group and
> successfully
> >>>>> organized two meetups which attracted more than 100 people from Xi’an
> >>>>> China. He wrote a handful of blogs and articles in Chinese tech
> media which
> >>>>> attracted a lot of interests in Mesos. He had given several talks
> about
> >>>>> Mesos at conferences in China.
> >>>>>
> >>>>> His major coding contribution to the project was the docker volume
> driver
> >>>>> isolator. He has also been involved in allocator performance
> improvement,
> >>>>> gpu support for docker containerizer, Mesos Tiers/Optimistic Offer
> design,
> >>>>> scarce resources discussion, and many others.
> >>>>>
> >>>>> His formal checklist is here:
> >>>>> https://docs.google.com/document/d/1tot79kyJCTTgJHBhzStFKrVkDK4pX
> >>>>> qfl-LHCLOovNtI/edit?usp=sharing
> >>>>>
> >>>>> Thanks,
> >>>>> - Jie
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Avinash Sridharan, Mesosphere
> >>> +1 (323) 702 5245
> >
>
>
> --

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: How to shutdown mesos-agent gracefully?

[Klaus Ma]

I'd like to notify framework to kill its tasks and then terminate the
mesos-agent. To the Maintenance feature, can not remember whether the slave
info will be clearup if that slave will not re-register back.

On Wed, Oct 12, 2016 at 10:13 PM Alex Rukletsov <a...@mesosphere.com> wrote:

> To make sure: you are aware of SIGUSR1?
>
> On Tue, Oct 11, 2016 at 5:37 PM, tommy xiao <xia...@gmail.com> wrote:
>
> > Hi Ma,
> >
> > could you please input more background, why Maintenance feature  is not
> > best option for your request?
> >
> > 2016-10-11 14:47 GMT+08:00 haosdent <haosd...@gmail.com>:
> >
> > > gracefully means not affect running tasks?
> > >
> > > On Tue, Oct 11, 2016 at 2:36 PM, Klaus Ma <klaus1982...@gmail.com>
> > wrote:
> > >
> > >> It seems there's not a way to shutdown mesos-agent gracefully.
> > >> Maintenance feature expect the agents re-register back in the future.
> > >>
> > >> Thanks
> > >> Klaus
> > >> --
> > >>
> > >> Regards,
> > >> 
> > >> Da (Klaus), Ma (马达), PMP® | Software Architect
> > >> IBM Platform Development & Support, STG, IBM GCG
> > >> +86-10-8245 4084 <+86%2010%208245%204084> | mad...@cn.ibm.com |
> http://k82.me
> > >>
> > >
> > >
> > >
> > > --
> > > Best Regards,
> > > Haosdent Huang
> > >
> >
> >
> >
> > --
> > Deshi Xiao
> > Twitter: xds2000
> > E-mail: xiaods(AT)gmail.com
> >
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

How to shutdown mesos-agent gracefully?

[Klaus Ma]

It seems there's not a way to shutdown mesos-agent gracefully.
Maintenance feature expect the agents re-register back in the future.

Thanks
Klaus
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: 中国的Mesos爱好者们，关于今年在杭州的MesosCon大会

[Klaus Ma]

https://www.surveymonkey.com should be ok to China contributors :).


Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Spectrum Conductor Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com<mailto:mad...@cn.ibm.com> | http://k82.me

On Sep 20, 2016, at 11:20, Yan Xu <y...@jxu.me<mailto:y...@jxu.me>> wrote:

Would surveymonkey.com<http://surveymonkey.com/> be more accessible?

On Monday, September 19, 2016, Yan Yan YY Hu 
<yanya...@cn.ibm.com<mailto:yanya...@cn.ibm.com>> wrote:

We need proxy to access Google in China mainland. But I guess this won't be a 
problem for us since there are many tools can help to cross the GreatFireWall :)

Best regards!
**
Yanyan Hu(胡彦彦) Ph.D.
Cloud Infrastructure & Technology Team
Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, Haidian
District, Beijing,P.R.C.100094
E-mail: yanya...@cn.ibm.com<javascript:_e(%7B%7D,'cvml','yanya...@cn.ibm.com');>
Tel: 8610-58748025
***

[Inactive hide details for David Greenberg ---2016-09-20 上午 
10:48:43---Specifically, we want to make sure that everyone will]David 
Greenberg ---2016-09-20 上午 10:48:43---Specifically, we want to make sure that 
everyone will be able to access the forms if we put them on

From: David Greenberg 
<dsg123456...@gmail.com<javascript:_e(%7B%7D,'cvml','dsg123456...@gmail.com');>>
To: user 
<u...@mesos.apache.org<javascript:_e(%7B%7D,'cvml','u...@mesos.apache.org');>>
Cc: 
"dev@mesos.apache.org<javascript:_e(%7B%7D,'cvml','dev@mesos.apache.org');>" 
<dev@mesos.apache.org<javascript:_e(%7B%7D,'cvml','dev@mesos.apache.org');>>
Date: 2016-09-20 上午 10:48
Subject: Re: 中国的Mesos爱好者们，关于今年在杭州的MesosCon大会





Specifically, we want to make sure that everyone will be able to access the 
forms if we put them on Google forms/docs.

On Mon, Sep 19, 2016 at 7:04 PM Hechen Gao 
<hechen@autodesk.com<javascript:_e(%7B%7D,'cvml','hechen@autodesk.com');>>
 wrote:

Hey David,

I would love to contribute to your survey about the MesosCon, please count me 
in.

Best regards,
Hechen Gao
Senior Software Engineer, Cloud Platforms - Engineering Core Services

Autodesk, Inc.
The Landmark @ One Market, Suite 500
San Francisco, CA  94105
www.autodesk.com<http://www.autodesk.com/>


On Sep 19, 2016, at 5:57 PM, tommy xiao 
<xia...@gmail.com<javascript:_e(%7B%7D,'cvml','xia...@gmail.com');>> wrote:

+1

在 2016年9月20日 上午8:22，David Greenberg 
<dsg123456...@gmail.com<javascript:_e(%7B%7D,'cvml','dsg123456...@gmail.com');>>写道：
作为此次MesosCon大会的主席，我希望你们能够在今年杭州的MesosCon大会中听到你们喜欢的演讲和分享。所以，我们正在准备发出一个Google
Forms的调查，这个调查将会帮助我们更好的决定演讲和分享的内容。希望你们能够积极参与这个调查。你们的意见对我们很重要。


David Greenberg, co-chair of MesosCon 敬上



--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com<http://gmail.com/>






--
Sent from mobile

Re: How many roles are we supported?

[Klaus Ma]

@Zhitao, thanks; that's helpful :).

On Thu, Sep 8, 2016 at 10:28 PM Zhitao Li <zhitaoli...@gmail.com> wrote:

> I'll share some of our targets which we aim to support per Mesos cluster,
> which may not be representative:
> - up to about 100 roles;
> - up to low hundreds of frameworks;
> - up to low tens of thousands of agents.
>
> On Thu, Sep 8, 2016 at 12:42 AM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > any suggestion?
> >
> > On Wed, Sep 7, 2016 at 11:35 AM Klaus Ma <klaus1982...@gmail.com> wrote:
> >
> >> + user@
> >>
> >>
> >> On Wed, Sep 7, 2016 at 11:31 AM Klaus Ma <klaus1982...@gmail.com>
> wrote:
> >>
> >>> IMO, it does not make sense to let user to try it :). It's better for
> us
> >>> (Mesos Dev) to provide suggestion :).
> >>>
> >>> On Wed, Sep 7, 2016 at 11:27 AM Zhitao Li <zhitaoli...@gmail.com>
> wrote:
> >>>
> >>>> I think polling user group for how people uses or plan to use Mesos
> will
> >>>> help.
> >>>>
> >>>> I personally already know at least two different ways of modeling
> >>>> multiple
> >>>> workloads to roles and frameworks in Mesos, which results in quite
> >>>> different numbers for roles and frameworks even for similar sized
> >>>> cluster.
> >>>>
> >>>> On Tue, Sep 6, 2016 at 7:54 PM, Klaus Ma <klaus1982...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > Question on Mesos's scalability of 1.0: how many roles are we going
> to
> >>>> > support? how many nodes are we going to support? how many frameworks
> >>>> are we
> >>>> > going to support? ...
> >>>> >
> >>>> > When using Mesos as resource manager, those info is important to us
> >>>> when
> >>>> > proposing solution.
> >>>> >
> >>>> > And in community, it's better for us to have a target for
> performance
> >>>> > related project; it takes time to keeping improving the performance
> >>>> :).
> >>>> >
> >>>> > Thanks
> >>>> > Klaus
> >>>> > --
> >>>> >
> >>>> > Regards,
> >>>> > 
> >>>> > Da (Klaus), Ma (马达), PMP® | Software Architect
> >>>> > IBM Platform Development & Support, STG, IBM GCG
> >>>> > +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Cheers,
> >>>>
> >>>> Zhitao Li
> >>>>
> >>> --
> >>>
> >>> Regards,
> >>> 
> >>> Da (Klaus), Ma (马达), PMP® | Software Architect
> >>> IBM Platform Development & Support, STG, IBM GCG
> >>> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>>
> >> --
> >>
> >> Regards,
> >> 
> >> Da (Klaus), Ma (马达), PMP® | Software Architect
> >> IBM Platform Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>
> > --
> >
> > Regards,
> > 
> > Da (Klaus), Ma (马达), PMP® | Software Architect
> > IBM Platform Development & Support, STG, IBM GCG
> > +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >
>
>
>
> --
> Cheers,
>
> Zhitao Li
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: How many roles are we supported?

[Klaus Ma]

any suggestion?

On Wed, Sep 7, 2016 at 11:35 AM Klaus Ma <klaus1982...@gmail.com> wrote:

> + user@
>
>
> On Wed, Sep 7, 2016 at 11:31 AM Klaus Ma <klaus1982...@gmail.com> wrote:
>
>> IMO, it does not make sense to let user to try it :). It's better for us
>> (Mesos Dev) to provide suggestion :).
>>
>> On Wed, Sep 7, 2016 at 11:27 AM Zhitao Li <zhitaoli...@gmail.com> wrote:
>>
>>> I think polling user group for how people uses or plan to use Mesos will
>>> help.
>>>
>>> I personally already know at least two different ways of modeling
>>> multiple
>>> workloads to roles and frameworks in Mesos, which results in quite
>>> different numbers for roles and frameworks even for similar sized
>>> cluster.
>>>
>>> On Tue, Sep 6, 2016 at 7:54 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>>>
>>> > Question on Mesos's scalability of 1.0: how many roles are we going to
>>> > support? how many nodes are we going to support? how many frameworks
>>> are we
>>> > going to support? ...
>>> >
>>> > When using Mesos as resource manager, those info is important to us
>>> when
>>> > proposing solution.
>>> >
>>> > And in community, it's better for us to have a target for performance
>>> > related project; it takes time to keeping improving the performance :).
>>> >
>>> > Thanks
>>> > Klaus
>>> > --
>>> >
>>> > Regards,
>>> > 
>>> > Da (Klaus), Ma (马达), PMP® | Software Architect
>>> > IBM Platform Development & Support, STG, IBM GCG
>>> > +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
>>> >
>>>
>>>
>>>
>>> --
>>> Cheers,
>>>
>>> Zhitao Li
>>>
>> --
>>
>> Regards,
>> 
>> Da (Klaus), Ma (马达), PMP® | Software Architect
>> IBM Platform Development & Support, STG, IBM GCG
>> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
>>
> --
>
> Regards,
> 
> Da (Klaus), Ma (马达), PMP® | Software Architect
> IBM Platform Development & Support, STG, IBM GCG
> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: How many roles are we supported?

[Klaus Ma]

+ user@

On Wed, Sep 7, 2016 at 11:31 AM Klaus Ma <klaus1982...@gmail.com> wrote:

> IMO, it does not make sense to let user to try it :). It's better for us
> (Mesos Dev) to provide suggestion :).
>
> On Wed, Sep 7, 2016 at 11:27 AM Zhitao Li <zhitaoli...@gmail.com> wrote:
>
>> I think polling user group for how people uses or plan to use Mesos will
>> help.
>>
>> I personally already know at least two different ways of modeling multiple
>> workloads to roles and frameworks in Mesos, which results in quite
>> different numbers for roles and frameworks even for similar sized cluster.
>>
>> On Tue, Sep 6, 2016 at 7:54 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>>
>> > Question on Mesos's scalability of 1.0: how many roles are we going to
>> > support? how many nodes are we going to support? how many frameworks
>> are we
>> > going to support? ...
>> >
>> > When using Mesos as resource manager, those info is important to us when
>> > proposing solution.
>> >
>> > And in community, it's better for us to have a target for performance
>> > related project; it takes time to keeping improving the performance :).
>> >
>> > Thanks
>> > Klaus
>> > --
>> >
>> > Regards,
>> > 
>> > Da (Klaus), Ma (马达), PMP® | Software Architect
>> > IBM Platform Development & Support, STG, IBM GCG
>> > +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
>> >
>>
>>
>>
>> --
>> Cheers,
>>
>> Zhitao Li
>>
> --
>
> Regards,
> 
> Da (Klaus), Ma (马达), PMP® | Software Architect
> IBM Platform Development & Support, STG, IBM GCG
> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: How many roles are we supported?

[Klaus Ma]

IMO, it does not make sense to let user to try it :). It's better for us
(Mesos Dev) to provide suggestion :).

On Wed, Sep 7, 2016 at 11:27 AM Zhitao Li <zhitaoli...@gmail.com> wrote:

> I think polling user group for how people uses or plan to use Mesos will
> help.
>
> I personally already know at least two different ways of modeling multiple
> workloads to roles and frameworks in Mesos, which results in quite
> different numbers for roles and frameworks even for similar sized cluster.
>
> On Tue, Sep 6, 2016 at 7:54 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > Question on Mesos's scalability of 1.0: how many roles are we going to
> > support? how many nodes are we going to support? how many frameworks are
> we
> > going to support? ...
> >
> > When using Mesos as resource manager, those info is important to us when
> > proposing solution.
> >
> > And in community, it's better for us to have a target for performance
> > related project; it takes time to keeping improving the performance :).
> >
> > Thanks
> > Klaus
> > --
> >
> > Regards,
> > 
> > Da (Klaus), Ma (马达), PMP® | Software Architect
> > IBM Platform Development & Support, STG, IBM GCG
> > +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >
>
>
>
> --
> Cheers,
>
> Zhitao Li
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

How many roles are we supported?

[Klaus Ma]

Question on Mesos's scalability of 1.0: how many roles are we going to
support? how many nodes are we going to support? how many frameworks are we
going to support? ...

When using Mesos as resource manager, those info is important to us when
proposing solution.

And in community, it's better for us to have a target for performance
related project; it takes time to keeping improving the performance :).

Thanks
Klaus
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: Validate lamba function in `Flags::add` for required parameters

[Klaus Ma]

Agree with you clean up `FlagBase::add` :).

alexR/bbannier/neil is also working on enhancement to the `FlagBase` 
(MESOS-3335), I think
we can work together to make `FlagBase::add` more clear.

Thanks
Klaus

On Jul 14, 2016, at 02:08, Greg Mann 
<g...@mesosphere.io<mailto:g...@mesosphere.io>> wrote:

Thanks for bringing this up, Klaus - in this case, I think that extra
argument appears simply to match the desired function overload. Over time,
the overloads for `FlagsBase::add` have multiplied considerably; It looks
like we have about 20 now :-) I think it would be really nice to clean
these up somehow. I didn't see a JIRA issue for this improvement so I
created one: MESOS-5841 <https://issues.apache.org/jira/browse/MESOS-5841>

One option would be to get rid of all the overloads except for
`FlagsBase::add(const Flag& flag)`, add a couple helper functions for
modifying `Flag` objects, and construct flag objects in the 'flags.cpp'
files:

 Flag flag;
 flag.name = "work_dir";
 flag.help = help_string;
 flag.set_storage(::work_dir); // New helper
 flag.set_validation(lambda_function); // New helper
 add(flag);

I think this would make the 'flags.cpp' files more readable, and it would
clean up `FlagsBase` by getting rid of all those overloads.

Cheers,
Greg


On Tue, Jul 12, 2016 at 11:14 PM, Klaus Ma 
<klaus1982...@gmail.com<mailto:klaus1982...@gmail.com>> wrote:

Hi team,


When I updating the patch for MESOS-5123<
https://issues.apache.org/jira/browse/MESOS-5123>, I found the validate
lamba function for in `Flags::add` for required parameters is different
with optional parameters. Does any know why? The coding style is
inconsistent, it took times to find the suitable function  :).


Flags::add for optional parameters:

```

 add(::executor_environment_variables,

 "executor_environment_variables",

 "JSON object representing the environment variables that should be\n"

 "passed to the executor, and thus subsequently task(s). By default
this\n"

 "flag is none. Users have to define executor environment
explicitly.\n"

 "Example:\n"

 "{\n"

 "  \"PATH\": \"/bin:/usr/bin\",\n"

 "  \"LD_LIBRARY_PATH\": \"/usr/local/lib\"\n"

 "}",

 [](const Option& object) -> Option {

   if (object.isSome()) {

 foreachvalue (const JSON::Value& value, object.get().values) {

   if (!value.is<http://value.is>()) {

 return Error("`executor_environment_variables` must "

  "only contain string values");

   }

 }

   }

   return None();

 });

```


Flags::add for required parameters:


```

 add(::work_dir,

 "work_dir",

 None(),   // <= Additional parameters to Flags::add

 "Absolute directory path of the agent work directory. This is
where\n"

 "executor sandboxes will be placed, as well as the agent's
checkpointed\n"

 "state in case of failover. Note that locations like `/tmp` which
are\n"

 "cleaned automatically are not suitable for the work directory
when\n"

 "running in production, since long-running agents could lose data
when\n"

 "cleanup occurs; if launching docker tasks, the path must not
include\n"

 "any disallowed symbols for docker volumes.\n"

 "(Example: `/var/lib/mesos/agent`)",

 static_cast(0),

 [](const string& workDir) -> Option {

   if (!strings::startsWith(workDir, "/")) {

 return Error(

 "The required option `--work_dir` must be absolute path.");

   }

   return None();

 });

```




Da (Klaus), Ma (??), PMP®| Software Architect
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com<mailto:mad...@cn.ibm.com> | http://k82.me

<http://k82.me/>

Validate lamba function in `Flags::add` for required parameters

[Klaus Ma]

Hi team,


When I updating the patch for 
MESOS-5123<https://issues.apache.org/jira/browse/MESOS-5123>, I found the 
validate lamba function for in `Flags::add` for required parameters is 
different with optional parameters. Does any know why? The coding style is 
inconsistent, it took times to find the suitable function  :).


Flags::add for optional parameters:

```

  add(::executor_environment_variables,

  "executor_environment_variables",

  "JSON object representing the environment variables that should be\n"

  "passed to the executor, and thus subsequently task(s). By default this\n"

  "flag is none. Users have to define executor environment explicitly.\n"

  "Example:\n"

  "{\n"

  "  \"PATH\": \"/bin:/usr/bin\",\n"

  "  \"LD_LIBRARY_PATH\": \"/usr/local/lib\"\n"

  "}",

  [](const Option& object) -> Option {

if (object.isSome()) {

  foreachvalue (const JSON::Value& value, object.get().values) {

if (!value.is()) {

  return Error("`executor_environment_variables` must "

   "only contain string values");

}

  }

}

return None();

  });

```


Flags::add for required parameters:


```

  add(::work_dir,

  "work_dir",

  None(),   // <= Additional parameters to Flags::add

  "Absolute directory path of the agent work directory. This is where\n"

  "executor sandboxes will be placed, as well as the agent's checkpointed\n"

  "state in case of failover. Note that locations like `/tmp` which are\n"

  "cleaned automatically are not suitable for the work directory when\n"

  "running in production, since long-running agents could lose data when\n"

  "cleanup occurs; if launching docker tasks, the path must not include\n"

  "any disallowed symbols for docker volumes.\n"

  "(Example: `/var/lib/mesos/agent`)",

  static_cast(0),

      [](const string& workDir) -> Option {

if (!strings::startsWith(workDir, "/")) {

  return Error(

  "The required option `--work_dir` must be absolute path.");

}

return None();

  });

```




Da (Klaus), Ma (??), PMP®| Software Architect
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

<http://k82.me/>

Re: [Tech-debt] Introduce regex into Mesos

[Klaus Ma]

C++11 support regex in `std` which we'd like to use; but it seems only gcc
4.9+ support it well. For the binaries, do you mean c++11's regex also need
libraries?


Da (Klaus), Ma (马达) | PMP® | Software Architect
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Jul 12, 2016 at 5:19 PM, Alexander Rojas <alexan...@mesosphere.io>
wrote:

> That was discussed in the past. The main issue is Mesos only bundles header
> only libraries and regex is a binary one.
>
> > On 05 Jul 2016, at 19:53, haosdent <haosd...@gmail.com> wrote:
> >
> >> As Joseph said, regex works well in llvm 7.3 & gcc 5.3, but does not
> work
> > in gcc 4.8.4; also try "" in gc 4.8.4, but "ld" failed :(.
> >
> > I think we just need to update the bundled boost and use the regex
> headers
> > from boost as what I do in
> > https://reviews.apache.org/r/40053/diff/3#index_header
> >
> > On Mon, Jun 13, 2016 at 9:58 PM, Klaus Ma <klaus1982...@hotmail.com>
> wrote:
> >
> >> As discussed in the RR, we'll enhance current code to handle port ranges
> >> as a short term solution.
> >>
> >>
> >> I've logged MESOS-5602 (Introduce expression grammar library)<
> >> https://issues.apache.org/jira/browse/MESOS-5602> to trace the long
> term
> >> solution for Mesos defined data format, e.g. resources.
> >>
> >>
> >> 
> >>
> >> Da (Klaus), Ma (??), PMP®| Advisory Software Engineer
> >> Platform DCOS Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>
> >> <http://k82.me/>
> >>
> >>
> >> 
> >> From: Klaus Ma <klaus1982...@hotmail.com>
> >> Sent: Saturday, June 11, 2016 2:09 AM
> >> To: dev
> >> Subject: Re: [Tech-debt] Introduce regex into Mesos
> >>
> >> As Joseph said, regex works well in llvm 7.3 & gcc 5.3, but does not
> work
> >> in gcc 4.8.4; also try "" in gc 4.8.4, but "ld" failed :(.
> >>
> >>
> >> 
> >>
> >> Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
> >> Platform DCOS Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>
> >> <http://k82.me/>
> >> 
> >> From: Joseph Wu <jos...@mesosphere.io>
> >> Sent: Friday, June 10, 2016 8:15:51 PM
> >> To: dev
> >> Subject: Re: [Tech-debt] Introduce regex into Mesos
> >>
> >> Same here.
> >>
> >> Mesos currently requires GCC 4.8.1+.  Regex support was implemented in
> GCC
> >> 4.9.0, see [1].
> >>
> >> [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53631
> >>
> >> On Fri, Jun 10, 2016 at 11:39 AM, Kevin Klues <klue...@gmail.com>
> wrote:
> >>
> >>> By compiler errors, I mean "internal compiler errors"
> >>>
> >>> On Fri, Jun 10, 2016 at 11:38 AM, Kevin Klues <klue...@gmail.com>
> wrote:
> >>>> I've run into compiler errors using simple regex stuff from the
> >>>> standard library on our supported version of gcc.
> >>>>
> >>>> On Thu, Jun 9, 2016 at 7:30 PM, Klaus Ma <klaus1982...@gmail.com>
> >> wrote:
> >>>>> Hi team,
> >>>>>
> >>>>>
> >>>>> We're discussing to introduce regex into Mesos when investigating
> >>> MESOS-4627<https://issues.apache.org/jira/browse/MESOS-4627>; so I'd
> >> like
> >>> to ask whether anyone has experience on regex after C++11? for example,
> >>> supported compiler, compatibility, performance and so on :).
> >>>>>
> >>>>>
> >>>>> 
> >>>>>
> >>>>> Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
> >>>>> Platform DCOS Development & Support, STG, IBM GCG
> >>>>> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>>>>
> >>>>> <http://k82.me/>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> ~Kevin
> >>>
> >>>
> >>>
> >>> --
> >>> ~Kevin
> >>>
> >>
> >
> >
> >
> > --
> > Best Regards,
> > Haosdent Huang
>
>

Re: MESOS: Improve the performance of Resources.

[Klaus Ma]

Hi Joris,

For `Scalars`, yes, it's also dynamic allocated in
`Resources::mutable_scalar()`.

For `2x` number, do you have any patch to share? Tested cases this
afternoon: add resources including 100 roles (1 CPU for each); the
performance is downgrade a lot; so I agree with you to improve some
algorithms in Resources/Sorter.

For 'basic benchmarks', temporary tracking in my personal github (
https://github.com/k82cn/mesos/blob/resources_benchmark/src/tests/resources_tests.cpp).
The following cases are in my mind to add:
1. simple resources, e.g. 1 cpu
2. resources with port, e.g. [1-2], [3-4], ... [101-102]
3. resources with reservation, cpus(r1):1;cpus(r2):1;  cpus(r10):1
4. resources with diskInfo
5. resources with revocableInfo

The operators will be +, +=, -, -=, cpus(), contains.

I booked a weekly call to discuss for allocaiton performance and sent the
invitation to the dev@.

If any comments, please let me know.


Da (Klaus), Ma (马达) | PMP® | Software Architect
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Mon, Jul 11, 2016 at 3:28 PM, Joris Van Remoortere <jo...@mesosphere.io>
wrote:

> +Dev
>
> Hey Klaus,
>
> Using Stout's `Optional` to represent the `optional` concept of a message
> in protobuf is definitely a step in the right direction.
> Regarding your comment in slack yesterday: From my version of the protobuf
> generated code there definitely is dynamic allocation even for scalars.
>
> It looks like in our case there is a minimum of 3 dynamic allocations per
> Resource object:
>
>> void Resource::SharedDtor() {
>>   if (name_ != &::google::protobuf::internal::kEmptyString) {
>> delete name_;
>>   }
>>   if (role_ != _default_role_) {
>> delete role_;
>>   }
>>   if (this != default_instance_) {
>> delete scalar_;
>> delete ranges_;
>> delete set_;
>> delete reservation_;
>> delete disk_;
>> delete revocable_;
>>   }
>> }
>
>
>  The 2x number I mentioned came from running some of the existing
> benchmarks. I didn't explore further because it didn't have as big an
> impact as I had hoped. The first battle is simplifying some of the
> algorithms in the Sorter / Resources. Once that is done then the resource
> arithmetic will be more of a bounding factor.
>
> I agree with Ben that we should focus on writing some basic benchmarks
> that represent the common uses of Resources in the allocator. We should
> scale these benchmarks to represent some of the more stressful environments
> that could occur. For example, had we had such a benchmark, we would have
> realized much earlier on that we needed to aggregate only quantities in the
> Sorter, and that using the existing form of Resources would have led to a
> grinding halt if a reservation were made on every machine.
>
> Is there a regular call that is scheduled to discuss this? I think there
> are some other folks also working on benchmarks and interested in the
> discussion.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
>> + more devs :).
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma <klaus1982...@gmail.com> wrote:
>>
>>> Hi Joris,
>>>
>>> I think `Option` is helpful to the performance improvement, it used
>>> `placement new` to avoid dynamic allocation. Suppose you're using Option
>>> for optional member in protobuf, and using class instance directly
>>> (operator=).
>>>
>>> I'm adding some benchmark for `Resources`, especially for the
>>> `Resources` with Rang, DiskInfo and ReservationInfo
>>>
>>> Draft PR for Benchmark of Resources:
>>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d68a8cf2249eb641c
>>>
>>>
>>> 
>>> Da (Klaus), Ma (马达) | PMP® | Software Architect
>>> Platform OpenSource Technology, STG, IBM GCG
>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>>
>>
>>
>

Re: [Tech-debt] Introduce regex into Mesos

[Klaus Ma]

As discussed in the RR, we'll enhance current code to handle port ranges as a 
short term solution.


I've logged MESOS-5602 (Introduce expression grammar 
library)<https://issues.apache.org/jira/browse/MESOS-5602> to trace the long 
term solution for Mesos defined data format, e.g. resources.




Da (Klaus), Ma (??), PMP®| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

<http://k82.me/>


________
From: Klaus Ma <klaus1982...@hotmail.com>
Sent: Saturday, June 11, 2016 2:09 AM
To: dev
Subject: Re: [Tech-debt] Introduce regex into Mesos

As Joseph said, regex works well in llvm 7.3 & gcc 5.3, but does not work in 
gcc 4.8.4; also try "" in gc 4.8.4, but "ld" failed :(.




Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

<http://k82.me/>

From: Joseph Wu <jos...@mesosphere.io>
Sent: Friday, June 10, 2016 8:15:51 PM
To: dev
Subject: Re: [Tech-debt] Introduce regex into Mesos

Same here.

Mesos currently requires GCC 4.8.1+.  Regex support was implemented in GCC
4.9.0, see [1].

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53631

On Fri, Jun 10, 2016 at 11:39 AM, Kevin Klues <klue...@gmail.com> wrote:

> By compiler errors, I mean "internal compiler errors"
>
> On Fri, Jun 10, 2016 at 11:38 AM, Kevin Klues <klue...@gmail.com> wrote:
> > I've run into compiler errors using simple regex stuff from the
> > standard library on our supported version of gcc.
> >
> > On Thu, Jun 9, 2016 at 7:30 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
> >> Hi team,
> >>
> >>
> >> We're discussing to introduce regex into Mesos when investigating
> MESOS-4627<https://issues.apache.org/jira/browse/MESOS-4627>; so I'd like
> to ask whether anyone has experience on regex after C++11? for example,
> supported compiler, compatibility, performance and so on :).
> >>
> >>
> >> 
> >>
> >> Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
> >> Platform DCOS Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>
> >> <http://k82.me/>
> >
> >
> >
> > --
> > ~Kevin
>
>
>
> --
> ~Kevin
>

[Compatibility] More strict parsing of ranges, e.g. port of resources

[Klaus Ma]

Hi team,


In "r/43561/<https://reviews.apache.org/r/43561/>", we'd like to move towards 
more strict parsing of ranges. If any comments, please let me know.


Before the patch, the follow ranges are all valid:

[1-2,3-4]

[1-4]

[[1-4]]

[1-2]\n[3-4]

[1-2],[3-4]


After the patch, only the following ranges are valid:

[1-2,3-4]

[1-4]


----

Da (Klaus), Ma (??), PMP®| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

<http://k82.me/>

[Compatibility] More strict parsing of ranges, e.g. port of resources

[Klaus Ma]

Hi team,


In "r/43561/<https://reviews.apache.org/r/43561/>", we'd like to move towards 
more strict parsing of ranges. If any comments, please let me know.


Before the patch, the follow ranges are all valid:

[1-2,3-4]

[1-4]

[[1-4]]

[1-2]\n[3-4]

[1-2],[3-4]


After the patch, only the following ranges are valid:

[1-2,3-4]

[1-4]


----

Da (Klaus), Ma (??), PMP®| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

<http://k82.me/>

Re: [Tech-debt] Introduce regex into Mesos

[Klaus Ma]

As Joseph said, regex works well in llvm 7.3 & gcc 5.3, but does not work in 
gcc 4.8.4; also try "" in gc 4.8.4, but "ld" failed :(.




Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

<http://k82.me/>

From: Joseph Wu <jos...@mesosphere.io>
Sent: Friday, June 10, 2016 8:15:51 PM
To: dev
Subject: Re: [Tech-debt] Introduce regex into Mesos

Same here.

Mesos currently requires GCC 4.8.1+.  Regex support was implemented in GCC
4.9.0, see [1].

[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53631

On Fri, Jun 10, 2016 at 11:39 AM, Kevin Klues <klue...@gmail.com> wrote:

> By compiler errors, I mean "internal compiler errors"
>
> On Fri, Jun 10, 2016 at 11:38 AM, Kevin Klues <klue...@gmail.com> wrote:
> > I've run into compiler errors using simple regex stuff from the
> > standard library on our supported version of gcc.
> >
> > On Thu, Jun 9, 2016 at 7:30 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
> >> Hi team,
> >>
> >>
> >> We're discussing to introduce regex into Mesos when investigating
> MESOS-4627<https://issues.apache.org/jira/browse/MESOS-4627>; so I'd like
> to ask whether anyone has experience on regex after C++11? for example,
> supported compiler, compatibility, performance and so on :).
> >>
> >>
> >> 
> >>
> >> Da (Klaus), Ma (??), PMP(r)| Advisory Software Engineer
> >> Platform DCOS Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me
> >>
> >> <http://k82.me/>
> >
> >
> >
> > --
> > ~Kevin
>
>
>
> --
> ~Kevin
>

Re: Welcome Anand and Joseph as new committers!

[Klaus Ma]

Congratulations :).


Da (Klaus), Ma (马达), PMP®| Advisory Software Engineer
Platform DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

From: Guangya Liu <gyliu...@gmail.com>
Sent: Thursday, June 9, 2016 11:55:03 PM
To: dev
Subject: Re: Welcome Anand and Joseph as new committers!

Congrats to both, well deserved!!

On Fri, Jun 10, 2016 at 5:00 AM, Vinod Kone <vinodk...@apache.org> wrote:

> Hi folks,
>
> I'm happy to announce that the PMC has voted in *Anand Mazumdar *and
> *Joseph
> Wu* as committers and members of PMC for the Apache Mesos project.
>
> A little about the new committers.
>
> Anand has been working on the Apache Mesos project for about an year now
> and has shown incredible commitment to the project and the community. His
> significant contributions include implementing scheduler HTTP API,
> designing and implementing executor HTTP API and helping out with the
> operator HTTP API. His formal committer checklist is here
> <
> https://docs.google.com/document/d/1DGRO-z-0JKS1dIxcrzXD8QznEv-auSqmIP3-aiwhtFI/edit?usp=sharing
> >
> .
>
> Joseph's passion and dedication to the community is phenomenal. His
> significant contributions include Maintenance Primitives and Container
> Logger Modules. He has also been a valuable contributor and reviewer to our
> testing infrastructure and the Windows work. His formal committer checklist
> is here
> <
> https://docs.google.com/document/d/1o7qLQJQ7TZCaf49gSNc6SSl29qAFagYH2STDfhHDDPw/edit?usp=sharing
> >
> .
>
> Please join me in congratulating them on their new roles and especially
> responsibilities :)
>
> On behalf of the PMC,
> Vinod
>

RE: [REVIEW PROCESS] Proposal for new review process working group

[Klaus Ma]

Please add me :).

Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer Platform OpenSource 
Technology, STG, IBM GCG +86-10-8245 4084 | klaus1982...@gmail.com | 
http://k82.me

> Date: Sat, 21 May 2016 09:22:32 +0800
> Subject: Re: [REVIEW PROCESS] Proposal for new review process working group
> From: zhq527...@gmail.com
> To: dev@mesos.apache.org
> 
> Please add me as well, thanks!
> 
> 
> Thanks,
> Qian Zhang
> 
> On Sat, May 21, 2016 at 9:10 AM, Guangya Liu <gyliu...@gmail.com> wrote:
> 
> > +1, please count me in as well.
> >
> > Thanks,
> >
> > Guangya
> >
> > On Sat, May 21, 2016 at 2:11 AM, Shivam Pathak <shivam.patha...@gmail.com>
> > wrote:
> >
> > > Great! please add me to the group
> > >
> > > On Fri, May 20, 2016 at 11:07 AM, haosdent <haosd...@gmail.com> wrote:
> > >
> > > > This sounds great, add me to the group please.
> > > >
> > > > On Sat, May 21, 2016 at 1:59 AM, Kevin Klues <klue...@gmail.com>
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to propose starting a dedicated "review process" working
> > > > > group.  The goals of this working group will be to:
> > > > >
> > > > > 1) Discuss issues around the current review process
> > > > > 2) Propose improvements to the current review process
> > > > > 3) Implement / Monitor / Enforce the new process we come up with
> > going
> > > > > forward
> > > > >
> > > > > Anyone who'd like to be involved, please respond to this thread so I
> > > > > can add you to the working group.  We will likely start actively
> > > > > discussing things after MesosCon.
> > > > >
> > > > > --
> > > > > ~Kevin
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards,
> > > > Haosdent Huang
> > > >
> > >
> > >
> > >
> > > --
> > > *Shivam Pathak (Mr)*
> > > Software Engineer and Systems Architect
> > > Novatap Private Ltd.
> > > HP: +65 8543 2297
> > >
> >
  

Re: [1/6] mesos git commit: Fixed a memory leak in the scheduler driver.

[Klaus Ma]

+1

Refer to this doc for the detail of deleting null: 
http://www.cplusplus.com/reference/new/operator%20delete/ 


Thanks
Klaus

> On Mar 30, 2016, at 07:24, Neil Conway  wrote:
> 
> On Tue, Mar 29, 2016 at 7:19 PM,   wrote:
>> --- a/src/sched/sched.cpp
>> +++ b/src/sched/sched.cpp
>> @@ -1808,6 +1808,10 @@ MesosSchedulerDriver::~MesosSchedulerDriver()
>> delete process;
>>   }
>> 
>> +  if (credential != NULL) {
>> +delete credential;
>> +  }
> 
> `delete` of a NULL pointer is safe, so I would vote for removing the `if`.
> 
> Neil

Re: Ordering guarantee of future.onAny callbacks

[Klaus Ma]

@Jie, if the target processes in “.onAny(defer())” are the same, is it still 
un-ordered?

> On Mar 29, 2016, at 09:19, Qian Zhang  wrote:
> 
> So the point is, if the state of the future is set to READY between
> .onAny(callback_A) and .onAny(callback_B) and before callback_A is executed
> in this thread, another thread comes in and does onAny(callback_B), then
> callback_B will be executed immediately in this thread, and callback_A will
> only be executed when the previous thread comes in again, right?
> 
> 
> Thanks,
> Qian Zhang
> 
> On Tue, Mar 29, 2016 at 8:50 AM, Jie Yu  wrote:
> 
>> Hi,
>> 
>> While digging a bug reported in, I realized an assumption we shouldn't make
>> in our code.
>> https://issues.apache.org/jira/browse/MESOS-5023
>> 
>> Say you have the following code:
>> 
>> void some_func()
>> {
>>  future
>>.onAny(callback_A)
>>.onAny(callback_B);
>> }
>> 
>> Question: will callback_A already be executed before callback_B?
>> 
>> The answer is NO. We should never assume that. Under the following
>> interleaving, callback_B can be invoked first:
>> 
>> Thread-1   Thread-2
>> 
>> onAny(callback_A) {
>>  onAnyCallbacks.push_back(callback_A);
>> }
>>  set() {
>>lock()
>>if (state ==
>> PENDING) {
>>  state = READY;
>>  result = true;
>>}
>>unlock();
>> 
>> onAny(callback_B) {
>>  lock()
>>  if (state != PENDING) {
>>run = true
>>  }
>>  unlock()
>> 
>>  if (run) {
>>callback_B()
>>  }
>> 
>> if (result) {
>> 
>> internal::run(data->onAnyCallbacks,
>> *this);
>> }
>> 
>> - Jie
>> 

Re: RFC: RevocableInfo Changes

[Klaus Ma]

Answer inline :).

> On Mar 29, 2016, at 05:25, Niklas Nielsen <n...@qni.dk> wrote:
> 
> Echoing Ben Mahler's comment. I still don't find the ThrottleInfo very 
> intuitive.
[Klaus]: Not sure which one is better: "ScavengeInfo"/ 
"BestEffortInfo"/“ThrottleableInfo” :).

> Did you discuss the general notion of resource quality further?
[Klaus]: Yes, I’m thinking how user/developer will use this feature and its 
behaviour; that’ll be helpful to the design.

> 
> On Mon, Mar 21, 2016 at 11:50 PM, Klaus Ma <klaus1982...@gmail.com 
> <mailto:klaus1982...@gmail.com>> wrote:
> @benm/joris,
> 
> here's the user scenario in my mind:
> 
> 1. master offers resources to the framework, e.g. 2 cpu
> 2. framework launch a task (2 cpu) and mark the task/executors as throttleable
> 3. in ResourceEstimator, it should only consider the throttleable 
> task/executors:
>   - keep enough resources for the tasks/executors without throttleable 
> flag/attribute
>   - report allocated but not used resources by task/executor with 
> throttleable flag/attribute; for example, report 1 cpu as 
> "Revocable.Throttleable" resources to framework in this case
> 4. it's up to framework to use which resources; "Revocable.Throttleable" 
> means it'll share compress resources with resources owner, "Revocable" 
> (without ThrottleableInfo) means it'll be evicted when the resources owner 
> reclaimed it back
> 5. QoS Controller makes sure:
>   - enough resources for the tasks/executors without throttleable 
> flag/attribute
>   - if used resources exceed allocated resources with throttleable 
> flag/attribute, evict the task/executor on revocable resource
> 
> So to @connor's question, maybe a flag/attribute to task/executor when 
> launching it. Regarding the name, both "ScavengeInfo"/ 
> "BestEffortInfo"/"ThrottleableInfo" are OK for me, maybe "ScavengeInfo" is 
> better.
> 
> Any comments?
> 
> For this scenario, I think there're still open questions:
> 1. Can framework launch task with throttleable flag/attribute on revocable 
> resources?
> 2. For ResourceEstimator/QoS Controllor, should Agent double check it report?
> 3. What's the behaviour between the two container: the container on original 
> resouces & the container on revocable resource?
> 4. Who handle compressible/in-compressible resources? Maybe 
> ResourceEstimator/QoSController, it should not report in-compressible 
> resources as Revocable.Throttleable.
> 
> Thanks
> Klaus
> 
> On Tuesday, March 22, 2016 at 4:13:10 AM UTC+8, Benjamin Mahler wrote:
> Yeah that's definitely a question I've been asking myself, and we synced on 
> that with Niklas during the last meeting. The thought currently is that we 
> should choose a better name than ThrottleInfo. ThrottleInfo seems to carry 
> too strong of an implication about what the resources will experience. 
> Rather, we could pick a name like "ScavengeInfo" / "BestEffortInfo" / etc 
> that indicates that these resources are running within the un-utilized 
> portion of the machine and _may_ experience degradation.
> 
> On Mon, Mar 21, 2016 at 1:26 AM, Joris Van Remoortere <jo...@mesosphere.io 
> <mailto:jo...@mesosphere.io>> wrote:
> @klaus:
> I think @connor's question is whether we are absolutely sure we never want to 
> support throttle-able but non-revocable resources.
> It's clear from the protos that this is not supported, the question is 
> whether we are sure that is what we want. If so, can you elaborate as to 
> *why* we would never want that concept in Mesos.
> 
> — 
> Joris Van Remoortere
> Mesosphere
> 
> On Sun, Mar 20, 2016 at 8:33 PM, Klaus Ma <klaus1982...@gmail.com 
> <mailto:klaus1982...@gmail.com>> wrote:
> Here's some input :).
> 
> If throttling is tolerable but preemption is not, how would that be 
> expressed? (Is that supported?)
> [Klaus]: It's not supported; only revocable resources has this attribute: 
> non-throttleable or throttleable. The throttleable revocable resources is 
> reported by ResourceEstimator which means the resources maybe throttled by 
> its original owner.
> 
> How does this work with the QoS controller? Will there be a new correction 
> type to indicate throttling, or does throttling happen "behind the agent's 
> back"?
> [Klaus]: The QoSController/ResourceEstimator only manages throttleable 
> revocable resources; the others resources (regular resources and 
> non-throttleable revocable resources) are managed by allocator. The "manage" 
> means generation and destroy/eviction. Regarding "throttling happen", good 
> question. I 

Re: [Proposal] Use dev mailing list for working groups

[Klaus Ma]

+1, that's helpful to filter feature/question out :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Fri, Mar 25, 2016 at 11:04 AM, Du, Fan <fan...@intel.com> wrote:

> +1
>
> This will definitely make the new developer easily get to know where each
> component
> is heading, and which component is most of his interest and then
> contribute.
>
> Thanks for the proposal!
>
>
> On 2016/3/25 6:55, Jie Yu wrote:
>
>> Hi,
>>
>> This came up during today's community sync.
>>
>> Mesos currently has a few working groups for various features:
>>
>> https://cwiki.apache.org/confluence/display/MESOS/Apache+Mesos+Working+Groups
>>
>> Some of those working groups are using separate mailing lists. That limits
>> the visibility of some discussions. Also, some people in the community are
>> not aware of those mailing lists (and the wiki page).
>>
>> Therefore, I am proposing that we consolidate all working groups mailing
>> lists to the dev mailing list. To distinguish discussions from different
>> working groups, please use a special subject format. For instance, if you
>> want to send an email to "Mesos GPU" working group, please use the
>> subject:
>>
>> "[Mesos GPU WG] YOUR SUBJECT HERE"
>>
>> Let me know if you have any comments/thoughts on this!
>>
>> - Jie
>>
>>

Re: RFC: RevocableInfo Changes

[Klaus Ma]

@benm/joris,

here's the user scenario in my mind:

1. master offers resources to the framework, e.g. 2 cpu
2. framework launch a task (2 cpu) and *mark the task/executors as 
throttleable*
3. in ResourceEstimator, it should only consider the throttleable 
task/executors:
  - keep enough resources for the tasks/executors *without* throttleable 
flag/attribute
  - report allocated but not used resources by task/executor *with* 
throttleable flag/attribute; for example, report 1 cpu as "
*Revocable.Throttleable"* resources to framework in this case
4. it's up to framework to use which resources; "*Revocable.Throttleable*" 
means it'll share compress resources with resources owner, "*Revocable*" 
(without ThrottleableInfo) means it'll be evicted when the resources owner 
reclaimed it back
5. QoS Controller makes sure:
  - enough resources for the tasks/executors *without* throttleable 
flag/attribute
  - if used resources exceed allocated resources *with* throttleable 
flag/attribute, evict the task/executor on revocable resource

So to @connor's question, maybe a flag/attribute to task/executor when 
launching it. Regarding the name, both "ScavengeInfo"/ 
"BestEffortInfo"/"ThrottleableInfo" are OK for me, maybe "ScavengeInfo" is 
better.

Any comments?

For this scenario, I think there're still open questions:
1. Can framework launch task with throttleable flag/attribute on revocable 
resources?
2. For ResourceEstimator/QoS Controllor, should Agent double check it 
report?
3. What's the behaviour between the two container: the container on 
original resouces & the container on revocable resource?
4. Who handle compressible/in-compressible resources? Maybe 
ResourceEstimator/QoSController, it should not report in-compressible 
resources as Revocable.Throttleable.

Thanks
Klaus

On Tuesday, March 22, 2016 at 4:13:10 AM UTC+8, Benjamin Mahler wrote:
>
> Yeah that's definitely a question I've been asking myself, and we synced 
> on that with Niklas during the last meeting. The thought currently is that 
> we should choose a better name than ThrottleInfo. ThrottleInfo seems to 
> carry too strong of an implication about what the resources will 
> experience. Rather, we could pick a name like "ScavengeInfo" / 
> "BestEffortInfo" / etc that indicates that these resources are running 
> within the un-utilized portion of the machine and _may_ experience 
> degradation.
>
> On Mon, Mar 21, 2016 at 1:26 AM, Joris Van Remoortere <jo...@mesosphere.io
> > wrote:
>
>> @klaus:
>> I think @connor's question is whether we are absolutely sure we never 
>> want to support throttle-able but non-revocable resources.
>> It's clear from the protos that this is not supported, the question is 
>> whether we are sure that is what we want. If so, can you elaborate as to 
>> *why* we would never want that concept in Mesos.
>>
>> — 
>> *Joris Van Remoortere*
>> Mesosphere
>>
>> On Sun, Mar 20, 2016 at 8:33 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>>
>>> Here's some input :).
>>>
>>> If throttling is tolerable but preemption is not, how would that be 
>>> expressed? (Is that supported?)
>>> [Klaus]: It's not supported; only revocable resources has this 
>>> attribute: non-throttleable or throttleable. The throttleable revocable 
>>> resources is reported by ResourceEstimator which means the resources maybe 
>>> throttled by its original owner.
>>>
>>> How does this work with the QoS controller? Will there be a new 
>>> correction type to indicate throttling, or does throttling happen "behind 
>>> the agent's back"?
>>> [Klaus]: The QoSController/ResourceEstimator only manages throttleable 
>>> revocable resources; the others resources (regular resources and 
>>> non-throttleable revocable resources) are managed by allocator. The 
>>> "manage" means generation and destroy/eviction. Regarding "throttling 
>>> happen", good question. I think the throttling will dependent on 
>>> containers, let me double check it :).
>>>
>>> If any comments, please let me know.
>>>
>>> 
>>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer 
>>> Platform OpenSource Technology, STG, IBM GCG 
>>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>>
>>> On Sat, Mar 19, 2016 at 11:15 PM, <connor@gmail.com> wrote:
>>>
>>>> Thanks for the good explanations so far Ben and Klaus.  Apologies if 
>>>> you guys already covered these questions in the meeting:
>>>>
>>>> If throttling is tole

Re: RFC: RevocableInfo Changes

[Klaus Ma]

Here's some input :).

If throttling is tolerable but preemption is not, how would that be
expressed? (Is that supported?)
[Klaus]: It's not supported; only revocable resources has this attribute:
non-throttleable or throttleable. The throttleable revocable resources is
reported by ResourceEstimator which means the resources maybe throttled by
its original owner.

How does this work with the QoS controller? Will there be a new correction
type to indicate throttling, or does throttling happen "behind the agent's
back"?
[Klaus]: The QoSController/ResourceEstimator only manages throttleable
revocable resources; the others resources (regular resources and
non-throttleable revocable resources) are managed by allocator. The
"manage" means generation and destroy/eviction. Regarding "throttling
happen", good question. I think the throttling will dependent on
containers, let me double check it :).

If any comments, please let me know.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Sat, Mar 19, 2016 at 11:15 PM, <connor@gmail.com> wrote:

> Thanks for the good explanations so far Ben and Klaus.  Apologies if you
> guys already covered these questions in the meeting:
>
> If throttling is tolerable but preemption is not, how would that be
> expressed? (Is that supported?)
>
> How does this work with the QoS controller? Will there be a new correction
> type to indicate throttling, or does throttling happen "behind the agent's
> back"?
>
> Thanks,
> --
> Connor
>
> > On Mar 19, 2016, at 04:01, Klaus Ma <klaus1982...@gmail.com> wrote:
> >
> > @team, in the latest meeting, we agree to keep current name ThrottleInfo.
> >
> > If any more comments, please let me know.
> >
> >> On Wednesday, March 16, 2016 at 9:32:37 PM UTC+8, Guangya Liu wrote:
> >> Also please show your comments if any for the name here, the current
> name is ThrottleInfo, in Kubernetes resources qos design document, they are
> using scavenging as the key work for such behaviour, so a possible name
> here could be ScavengeInfo , please show your comments if any for those two
> names or even if you want to propose a new name here.
> >>
> >> message RevocableInfo {
> >> message ThrottleInfo {}
> >>
> >> // If set, indicates that the resources may be throttled at
> >> // any time. Throttle-able resoruces can be used for tasks
> >> // that do not have strict performance requirements and are
> >> // capable of handling being throttled.
> >> optional ThrottleInfo throttle_info = 1;
> >>   }
> >>
> >> 在 2016年3月16日星期三 UTC+8上午10:24:14，Klaus Ma写道：
> >>>
> >>> The patches are updated accordingly; JIRA: MESOS-3888 , RR:
> https://reviews.apache.org/r/40375/ .
> >>>
> >>> Thanks
> >>> klaus
> >>>
> >>>> On Saturday, March 12, 2016 at 11:09:46 AM UTC+8, Benjamin Mahler
> wrote:
> >>>> Hey folks,
> >>>>
> >>>> In the resource allocation working group we've been looking into a
> few projects that will make the allocator able to offer out resources as
> revocable. For example:
> >>>>
> >>>> -We'll want to eventually allocate resources as revocable _by
> default_, only allowing non-revocable when there are guarantees put in
> place (static reservations or quota).
> >>>>
> >>>> -On the path to revocable by default, we can incrementally start to
> offer certain resources as revocable. Consider when quota is set but the
> role isn't using all of the quota. The unallocated quota can be offered to
> other roles, but it should be revocable because we may revoke them should
> the quota'ed role want to use the resources. Unused reservations fall into
> a similar category.
> >>>>
> >>>> -Going revocable by default also allows us to enforce fairness in a
> dynamically changing cluster by revoking resources as weights are changed,
> frameworks are added or removed, etc.
> >>>>
> >>>> In this context, "revocable" means that the resources may be taken
> away and the container will be destroyed. The meaning of "revocable" in the
> context of usage oversubscription includes this, but also the container may
> experience a throttling (e.g. lower cpu shares, less network priority, etc).
> >>>>
> >>>> For this reason, and because we internally need to distinguish
> revocable resources between the those that are generated by u

Re: RFC: RevocableInfo Changes

[Klaus Ma]

@team, in the latest meeting, we agree to keep current name *ThrottleInfo.*

If any more comments, please let me know.

On Wednesday, March 16, 2016 at 9:32:37 PM UTC+8, Guangya Liu wrote:
>
> Also please show your comments if any for the name here, the current name 
> is *ThrottleInfo*, in Kubernetes resources qos design document, they are 
> using scavenging as the key work for such behaviour, so a possible name 
> here could be *ScavengeInfo , *please show your comments if any for those 
> two names or even if you want to propose a new name here.
>
> message RevocableInfo {
> *message ThrottleInfo {}*
>
> *// If set, indicates that the resources may be throttled at*
> *// any time. Throttle-able resoruces can be used for tasks*
> *// that do not have strict performance requirements and are*
> *// capable of handling being throttled.*
> *optional ThrottleInfo throttle_info = 1;*
>   }
>
> 在 2016年3月16日星期三 UTC+8上午10:24:14，Klaus Ma写道：
>>
>> The patches are updated accordingly; JIRA: MESOS-3888 
>> <https://issues.apache.org/jira/browse/MESOS-3888> , RR: 
>> https://reviews.apache.org/r/40375/ .
>>
>> Thanks
>> klaus
>>
>> On Saturday, March 12, 2016 at 11:09:46 AM UTC+8, Benjamin Mahler wrote:
>>>
>>> Hey folks,
>>>
>>> In the resource allocation working group we've been looking into a few 
>>> projects that will make the allocator able to offer out resources as 
>>> revocable. For example:
>>>
>>> -We'll want to eventually allocate resources as revocable _by default_, 
>>> only allowing non-revocable when there are guarantees put in place (static 
>>> reservations or quota).
>>>
>>> -On the path to revocable by default, we can incrementally start to 
>>> offer certain resources as revocable. Consider when quota is set but the 
>>> role isn't using all of the quota. The unallocated quota can be offered to 
>>> other roles, but it should be revocable because we may revoke them should 
>>> the quota'ed role want to use the resources. Unused reservations fall into 
>>> a similar category.
>>>
>>> -Going revocable by default also allows us to enforce fairness in a 
>>> dynamically changing cluster by revoking resources as weights are changed, 
>>> frameworks are added or removed, etc.
>>>
>>> In this context, "revocable" means that the resources may be taken away 
>>> and the container will be destroyed. The meaning of "revocable" in the 
>>> context of usage oversubscription includes this, but also the container may 
>>> experience a throttling (e.g. lower cpu shares, less network priority, etc).
>>>
>>> For this reason, and because we internally need to distinguish revocable 
>>> resources between the those that are generated by usage oversubscription 
>>> and those that are generated by the allocator, we're thinking of the 
>>> following change to the API:
>>>
>>>
>>>
>>> -  message RevocableInfo {}
>>> +  message RevocableInfo {
>>> +message ThrottleInfo {}
>>> +
>>> +// If set, indicates that the resources may be throttled at
>>> +// any time. Throttle-able resoruces can be used for tasks
>>> +// that do not have strict performance requirements and are
>>> +// capable of handling being throttled.
>>> +optional ThrottleInfo throttle_info;
>>> +  }
>>>
>>>// If this is set, the resources are revocable, i.e., any tasks or
>>> -  // executors launched using these resources could get preempted or
>>> -  // throttled at any time. This could be used by frameworks to run
>>> -  // best effort tasks that do not need strict uptime or performance
>>> +  // executors launched using these resources could be terminated at
>>> +  // any time. This could be used by frameworks to run
>>> +  // best effort tasks that do not need strict uptime
>>>// guarantees. Note that if this is set, 'disk' or 'reservation'
>>>// cannot be set.
>>>optional RevocableInfo revocable = 9;
>>>
>>>
>>>
>>> Essentially we want to distinguish between revocable and revocable + 
>>> throttle-able. This is because usage-oversubscription generates 
>>> throttle-able revocable resources, whereas the allocator does not. This 
>>> also solves our problem of distinguishing between these two kinds of 
>>> revocable resources internally.
>>>
>>> Feedback welcome!
>>>
>>> Ben
>>>
>>>

Re: [RESULT][VOTE] Release Apache Mesos 0.27.2 (rc1)

[Klaus Ma]

@team, are we going to provide LTS version? If so, we definitely need a
branch to back merge issues; if not, we release a new version monthly (??),
we'd suggest user to upgrade to next release.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Sat, Mar 19, 2016 at 7:59 AM, Kevin Klues <klue...@gmail.com> wrote:

> I respectfully disagree.
>
> The whole purpose of tags is to mark permanent things like releases,
> whereas branches are designed as temporary lines of development that
> come and go (and grow and shrink) dynamically all the time.
>
> On Fri, Mar 18, 2016 at 4:04 PM, Jie Yu <yujie@gmail.com> wrote:
> > I like the idea of using branches to manage releases.
> >
> > We can use that to manage point releases and backports as well.
> >
> > Say we want to cut 0.29.0 now, we fork a branch 0.29.0 and tag RCs in
> that
> > branch. Once the RC is accepted, the head of that branch will become the
> > release.
> >
> > Then, we immediate fork that branch and create 0.29.1 branch.
> >
> > When a new bug fix is committed on the trunk, the committer will decide
> > whether it'll affect the old releases (a bounded number, we can decide
> that
> > later). If it does, the committer of that patch should also cherry-pick
> > that patch to the point releases (e.g., 0.29.1 in this case). We can do a
> > timely based point releases.
> >
> > - Jie
> >
> > On Fri, Mar 18, 2016 at 1:35 PM, Cong Wang <cw...@twopensource.com>
> wrote:
> >
> >> On Wed, Mar 16, 2016 at 11:56 AM, Joseph Wu <jos...@mesosphere.io>
> wrote:
> >> > Cong Wang,
> >> >
> >> > The tags are sync'd.  See: https://github.com/apache/mesos/releases
> >> >
> >> > You might not have done: git pull --tags
> >>
> >>
> >> Yeah, I figured it out by myself too. This is why I hate tags
> personally,
> >> branches are better since they are fetched without additional
> parameters.
> >>
> >> Any reason why Mesos maintainers picked tags over branches to manage
> >> releases? Just curious...
> >>
>
>
>
> --
> ~Kevin
>

Re: RFC: RevocableInfo Changes

[Klaus Ma]

The patches are updated accordingly; JIRA: MESOS-3888 
 , RR: 
https://reviews.apache.org/r/40375/ .

Thanks
klaus

On Saturday, March 12, 2016 at 11:09:46 AM UTC+8, Benjamin Mahler wrote:
>
> Hey folks,
>
> In the resource allocation working group we've been looking into a few 
> projects that will make the allocator able to offer out resources as 
> revocable. For example:
>
> -We'll want to eventually allocate resources as revocable _by default_, 
> only allowing non-revocable when there are guarantees put in place (static 
> reservations or quota).
>
> -On the path to revocable by default, we can incrementally start to offer 
> certain resources as revocable. Consider when quota is set but the role 
> isn't using all of the quota. The unallocated quota can be offered to other 
> roles, but it should be revocable because we may revoke them should the 
> quota'ed role want to use the resources. Unused reservations fall into a 
> similar category.
>
> -Going revocable by default also allows us to enforce fairness in a 
> dynamically changing cluster by revoking resources as weights are changed, 
> frameworks are added or removed, etc.
>
> In this context, "revocable" means that the resources may be taken away 
> and the container will be destroyed. The meaning of "revocable" in the 
> context of usage oversubscription includes this, but also the container may 
> experience a throttling (e.g. lower cpu shares, less network priority, etc).
>
> For this reason, and because we internally need to distinguish revocable 
> resources between the those that are generated by usage oversubscription 
> and those that are generated by the allocator, we're thinking of the 
> following change to the API:
>
>
>
> -  message RevocableInfo {}
> +  message RevocableInfo {
> +message ThrottleInfo {}
> +
> +// If set, indicates that the resources may be throttled at
> +// any time. Throttle-able resoruces can be used for tasks
> +// that do not have strict performance requirements and are
> +// capable of handling being throttled.
> +optional ThrottleInfo throttle_info;
> +  }
>
>// If this is set, the resources are revocable, i.e., any tasks or
> -  // executors launched using these resources could get preempted or
> -  // throttled at any time. This could be used by frameworks to run
> -  // best effort tasks that do not need strict uptime or performance
> +  // executors launched using these resources could be terminated at
> +  // any time. This could be used by frameworks to run
> +  // best effort tasks that do not need strict uptime
>// guarantees. Note that if this is set, 'disk' or 'reservation'
>// cannot be set.
>optional RevocableInfo revocable = 9;
>
>
>
> Essentially we want to distinguish between revocable and revocable + 
> throttle-able. This is because usage-oversubscription generates 
> throttle-able revocable resources, whereas the allocator does not. This 
> also solves our problem of distinguishing between these two kinds of 
> revocable resources internally.
>
> Feedback welcome!
>
> Ben
>
>

Re: RFC: RevocableInfo Changes

[Klaus Ma]

@ Niklas/Jie, just sent the invitation to you :)


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Mar 15, 2016 at 10:06 AM, Guangya Liu <gyliu...@gmail.com> wrote:

> This is the google doc that we used to trace all discussion topics:
>
> https://docs.google.com/document/d/1B_v52zCOFcwCpqCPhgYi9h630a0NE-QM9Br0nCOZUR4/edit?usp=sharing
>
> This is the link for google hang out
> video call Join video call
> <
> https://plus.google.com/hangouts/_/calendar/a2xhdXMxOTgyLmNuQGdtYWlsLmNvbQ.rgasirkilgmn1bm69kdcqjsb2s
> >The
> time is March 15th at 5pm PST
>
> Thanks,
>
> Guangya
>
> On Tue, Mar 15, 2016 at 9:47 AM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
> > Sounds good, the next one is tomorrow March 15th at 5pm PST (they are at
> > 5pm PST to accommodate China time zone).
> >
> > Will that work?
> >
> > On Mon, Mar 14, 2016 at 10:53 AM, Niklas Nielsen <n...@qni.dk> wrote:
> >
> >> Ben, when do you have your next mesos allocator sync? We don't have our
> >> next performance isolation sync lined up yet, so we could piggy back on
> >> yours if you have it scheduled already.
> >>
> >> Niklas
> >>
> >> On Mon, Mar 14, 2016 at 9:32 AM, Jie Yu <yujie@gmail.com> wrote:
> >>
> >> > >
> >> > > Just a quick note: Ian D. and the performance isolation working
> group
> >> are
> >> > > discussing similar annotations and we should meet and talk about the
> >> > > options.
> >> >
> >> >
> >> > +1
> >> >
> >> > Would love to understand the relationship between this and the
> >> > task/executor level annotations.
> >> >
> >> > - Jie
> >> >
> >> > On Mon, Mar 14, 2016 at 9:29 AM, Niklas Nielsen <n...@qni.dk> wrote:
> >> >
> >> > > Hi Ben,
> >> > >
> >> > > Just a quick note: Ian D. and the performance isolation working
> group
> >> are
> >> > > discussing similar annotations and we should meet and talk about the
> >> > > options.
> >> > >
> >> > > Niklas
> >> > >
> >> > > On Sat, Mar 12, 2016 at 12:05 AM, Klaus Ma <klaus1982...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > Yes, I think that's true for now; so we define `ThrottleInfo` as
> >> > message
> >> > > to
> >> > > > be more flexible. In Optimistic Offer Phase 1, we only use it to
> >> > > > distinguish usage oversubscriptions and allocation
> oversubscription,
> >> > > > similar to bool :).
> >> > > >
> >> > > > Regarding the resources type, two questions after the discussion:
> >> > > >
> >> > > > 1. should we send different offer to the framework, so when
> >> > > > usage/allocation oversubscription updated, only one type of offer
> >> will
> >> > be
> >> > > > rescinded?
> >> > > > 2. should we define framework's capability against `ThrottleInfo`?
> >> > > >
> >> > > > 
> >> > > > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> >> > > > Platform OpenSource Technology, STG, IBM GCG
> >> > > > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >> > > >
> >> > > > On Sat, Mar 12, 2016 at 12:03 PM, Guangya Liu <gyliu...@gmail.com
> >
> >> > > wrote:
> >> > > >
> >> > > > >
> >> > > > > Hi Ben,
> >> > > > >
> >> > > > > I think that currently and even in the near future, the
> >> > > __ThrottleInfo__
> >> > > > > will only be used by the usage oversubscriptions and the
> >> > > oversubscription
> >> > > > > for allocator (Both quota and reservations) will not use this
> >> value
> >> > but
> >> > > > > only using __RevocableInfo__ is enough.
> >> > > > >
> >> > > > > I can even think that the __ThrottleInfo__ as a boolean value in
> >> > > > > optimistic offer phase 1 as it is mainly used to distinguish
> >> > resources
> >> > > > > 

Re: RFC: RevocableInfo Changes

[Klaus Ma]

Yes, I think that's true for now; so we define `ThrottleInfo` as message to
be more flexible. In Optimistic Offer Phase 1, we only use it to
distinguish usage oversubscriptions and allocation oversubscription,
similar to bool :).

Regarding the resources type, two questions after the discussion:

1. should we send different offer to the framework, so when
usage/allocation oversubscription updated, only one type of offer will be
rescinded?
2. should we define framework's capability against `ThrottleInfo`?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Sat, Mar 12, 2016 at 12:03 PM, Guangya Liu <gyliu...@gmail.com> wrote:

>
> Hi Ben,
>
> I think that currently and even in the near future, the __ThrottleInfo__
> will only be used by the usage oversubscriptions and the oversubscription
> for allocator (Both quota and reservations) will not use this value but
> only using __RevocableInfo__ is enough.
>
> I can even think that the __ThrottleInfo__ as a boolean value in
> optimistic offer phase 1 as it is mainly used to distinguish resources
> between usage oversubscriptions and allocation oversubscription (Quota and
> Reservations), comments?
>
> Thanks,
>
> Guangya
>
> 在 2016年3月12日星期六 UTC+8上午11:09:46，Benjamin Mahler写道：
>
>> Hey folks,
>>
>> In the resource allocation working group we've been looking into a few
>> projects that will make the allocator able to offer out resources as
>> revocable. For example:
>>
>> -We'll want to eventually allocate resources as revocable _by default_,
>> only allowing non-revocable when there are guarantees put in place (static
>> reservations or quota).
>>
>> -On the path to revocable by default, we can incrementally start to offer
>> certain resources as revocable. Consider when quota is set but the role
>> isn't using all of the quota. The unallocated quota can be offered to other
>> roles, but it should be revocable because we may revoke them should the
>> quota'ed role want to use the resources. Unused reservations fall into a
>> similar category.
>>
>> -Going revocable by default also allows us to enforce fairness in a
>> dynamically changing cluster by revoking resources as weights are changed,
>> frameworks are added or removed, etc.
>>
>> In this context, "revocable" means that the resources may be taken away
>> and the container will be destroyed. The meaning of "revocable" in the
>> context of usage oversubscription includes this, but also the container may
>> experience a throttling (e.g. lower cpu shares, less network priority, etc).
>>
>> For this reason, and because we internally need to distinguish revocable
>> resources between the those that are generated by usage oversubscription
>> and those that are generated by the allocator, we're thinking of the
>> following change to the API:
>>
>>
>>
>> -  message RevocableInfo {}
>> +  message RevocableInfo {
>> +message ThrottleInfo {}
>> +
>> +// If set, indicates that the resources may be throttled at
>> +// any time. Throttle-able resoruces can be used for tasks
>> +// that do not have strict performance requirements and are
>> +// capable of handling being throttled.
>> +optional ThrottleInfo throttle_info;
>> +  }
>>
>>// If this is set, the resources are revocable, i.e., any tasks or
>> -  // executors launched using these resources could get preempted or
>> -  // throttled at any time. This could be used by frameworks to run
>> -  // best effort tasks that do not need strict uptime or performance
>> +  // executors launched using these resources could be terminated at
>> +  // any time. This could be used by frameworks to run
>> +  // best effort tasks that do not need strict uptime
>>// guarantees. Note that if this is set, 'disk' or 'reservation'
>>// cannot be set.
>>optional RevocableInfo revocable = 9;
>>
>>
>>
>> Essentially we want to distinguish between revocable and revocable +
>> throttle-able. This is because usage-oversubscription generates
>> throttle-able revocable resources, whereas the allocator does not. This
>> also solves our problem of distinguishing between these two kinds of
>> revocable resources internally.
>>
>> Feedback welcome!
>>
>> Ben
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Mesos Resource Allocation Working Group" group.
> To unsubscribe from this group and stop receivi

Re: Negative durations

[Klaus Ma]

One case I can image is to use negative value for forever duration?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Mar 9, 2016 at 8:21 PM, Alex Rukletsov <a...@mesosphere.com> wrote:

> Folks,
>
> I've recently realized that durations we use in mesos (stout's `Duration`
> and `DurationInfo` protobuf) are based on signed integers. Negative
> duration concept is a bit strange to me, so I googled around a bit and
> found an interesting thread [1].
>
> Was it an explicit intention to allow `Duration`s to be negative? Do we use
> this feature? If yes, maybe we can introduce a class representing time
> delta (can be negative) and base `Duration` on top of it guaranteeing it is
> always non-negative?
>
> My ultimate intention is to avoid boilerplate code that validates every
> single instance of `Duration` in the codebase. I'd rather have a class with
> guarantees certain invariants.
>
> [1] https://internals.rust-lang.org/t/unsigned-version-of-duration/893/2
>

Re: Mesos Maintenance Enhancement (post-MVP)

[Klaus Ma]

Done.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Mar 8, 2016 at 10:17 AM, Benjamin Mahler <bmah...@apache.org> wrote:

> Can you send an invite to bmah...@apache.org?
>
> On Sun, Mar 6, 2016 at 6:40 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > Hi team,
> >
> > I started the following working group to enhance Mesos Maintenance for
> some
> > cases. Please join the working group if you're interesting in it :).
> >
> > *Working Group*:
> > https://groups.google.com/d/forum/mesos-maintenance-working-group
> >
> > If any comments, please let me know.
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
>

Re: Mesos supports to get the available total resource per-roles ?

[Klaus Ma]

So I think you need to calculate it from `/master/state`; `/master/state`
has provided the reserved resources per-slave per-role.




Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Feb 23, 2016 at 9:28 AM, 陈强 <chenqi...@qiyi.com> wrote:

> @Klaus, The total resources reserved by this role I think.
>
> .
>
>
> On 2016年02月22日 19:33, Klaus Ma wrote:
>
>> @陈强，How do you define the "total resources"? All resources of frameworks
>> in
>> the same role, the deserved resources of role it should get, or  the total
>> resources reserved by this role?
>>
>> 
>> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> Platform OpenSource Technology, STG, IBM GCG
>> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>>
>> On Mon, Feb 22, 2016 at 6:07 PM, Guangya Liu <gyliu...@gmail.com> wrote:
>>
>> Does /master/state endpoint help? This can get all resources per-role in
>>> the cluster, including total, used, reserved etc per role.
>>>
>>> *"reserved_resources"*: {
>>>
>>>  *"r2"*: {
>>>
>>>*"mem"*: 0,
>>>
>>>*"disk"*: 0,
>>>
>>>*"cpus"*: 8
>>>
>>>  },
>>>
>>>  *"r1"*: {
>>>
>>>*"mem"*: 8000,
>>>
>>>*"disk"*: 0,
>>>
>>>*"cpus"*: 0
>>>
>>>  }
>>>
>>>}
>>>
>>> Thanks,
>>>
>>> Guangya
>>>
>>> On Mon, Feb 22, 2016 at 5:11 PM, 陈强 <chenqi...@qiyi.com> wrote:
>>>
>>> Hi all,
>>>>
>>>> Does Mesos support to get the available total resource for every roles
>>>> now? if don't, who are focusing on this? thanks.
>>>>
>>>> --
>>>>
>>>> Best Regards,
>>>> ChenQiang
>>>>
>>>>
>>>>
> --
> 陈强
>
> 技术产品中心 计算云 弹性计算
> 手机：＋86 15900964316
> 分机：8377
>
>

Re: Mesos supports to get the available total resource per-roles ?

[Klaus Ma]

@陈强，How do you define the "total resources"? All resources of frameworks in
the same role, the deserved resources of role it should get, or  the total
resources reserved by this role?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Mon, Feb 22, 2016 at 6:07 PM, Guangya Liu <gyliu...@gmail.com> wrote:

> Does /master/state endpoint help? This can get all resources per-role in
> the cluster, including total, used, reserved etc per role.
>
> *"reserved_resources"*: {
>
> *"r2"*: {
>
>   *"mem"*: 0,
>
>   *"disk"*: 0,
>
>   *"cpus"*: 8
>
> },
>
> *"r1"*: {
>
>   *"mem"*: 8000,
>
>   *"disk"*: 0,
>
>   *"cpus"*: 0
>
> }
>
>   }
>
> Thanks,
>
> Guangya
>
> On Mon, Feb 22, 2016 at 5:11 PM, 陈强 <chenqi...@qiyi.com> wrote:
>
> > Hi all,
> >
> > Does Mesos support to get the available total resource for every roles
> > now? if don't, who are focusing on this? thanks.
> >
> > --
> >
> > Best Regards,
> > ChenQiang
> >
> >
>

Re: 答复: Question about "Framework directly access Meso agent"

[Klaus Ma]

Suteng,

Regarding the storage/data part, agree with Joris to use distributed
FS/Cache for data.
Regarding resource bookkeeping, Master also need to check whether framework
used more resources against its offers. If framework access Mesos agent to
launch tasks, it can not sync up Master/Agent timely.



On Thu, Feb 18, 2016 at 10:22 PM Joris Van Remoortere <jo...@mesosphere.io>
wrote:

> If the main reason for contemplating this design is the size of the task
> payload, have you considered a content addressable storage design?
> For example: why can the task not be launched with URIs that allow the
> agent to download the payload before launching the task? This fans out the
> network load evenly among agents.
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Wed, Feb 17, 2016 at 9:35 PM, Suteng <sut...@huawei.com> wrote:
>
> > Alex,
> > We don't have test the performance of mesos. But we have develop a
> > framework in house, which is like a simplified mesos, use to schedule a
> > large number fine grain computation tasks.  We find that master will be a
> > bottleneck. One reason is our task contain several KB data, and task
> number
> > is quite huge.
> > If we use mesos to replace it, maybe master still be a bottleneck.
> >
> > Master still do the resource bookkeeping, we can decompose launch task to
> > two steps, firstly scheduler tell master which offer he wants, then
> master
> > tell scheduler the address of agent. Secondly, scheduler can directly
> > launch task to the agent, and also can directly send message to agent.
> > Maybe I can do some test about the mesos master launch task throughput,
> > with different number task data.
> >
> >
> > -邮件原件-
> > 发件人: C Rukletsov [mailto:a...@mesosphere.com]
> > 发送时间: 2016年2月17日 18:04
> > 收件人: dev
> > 主题: Re: Question about "Framework directly access Meso agent"
> >
> > Suteng—
> >
> > such optimization makes sense in certain cases (e.g. sending a framework
> > message), but it can be rather tricky in general, because the master has
> to
> > maintain bookkeeping. Moreover, with the upcoming HTTP API it becomes
> > harder for a framework to determine where to send messages to reach a
> > specific agent.
> >
> > Have you done any performance tests and seen master becoming a
> bottleneck?
> >
> > On Wed, Feb 17, 2016 at 5:14 AM, Suteng <sut...@huawei.com> wrote:
> >
> > > Hi,
> > >
> > >
> > >
> > > Currently, Mesos framework’s task related operations lauchTask,
> > > updateStatus and executorSendMessage etc., and resource related
> > > operations resourceOffer etc., all operations are pass through Mesos
> > Master.
> > >
> > > When the cluster and task number become huge, or with optimistic
> > > resource offer, multi-framework concurrently launchTask, maybe Mesos
> > > Master will be a bottleneck.
> > >
> > > Is possible for framework scheduler directly access Mesos agent,
> > > launchTask, updateStatus and SendMessage2Executore to Mesos Agent
> > > directly, bypass the Master?
> > >
> > > Will invoke big conflict with current mechanism?
> > >
> > >
> > >
> > > Looking forward to your comments and opinions.
> > >
> > >
> > >
> > > Best Regards,
> > >
> > > Teng
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Su Teng  00241668
> > >
> > >
> > >
> > > Distributed and Parallel Software Lab
> > >
> > > Huawei Technologies Co., Ltd.
> > >
> > > Email:sut...@huawei.com
> > >
> > >
> > >
> > >
> > >
> >
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: Enable compiler optimization by default?

[Klaus Ma]

+1;

So our CI will also update to use optimisation flags, right?  We need to
highlight this in upgrade document to our user; I used to meet so strange
behaviour after changing -O level.

On Thu, Feb 18, 2016 at 8:51 AM James DeFelice <james.defel...@gmail.com>
wrote:

> +1
> On Feb 17, 2016 7:24 PM, "Neil Conway" <neil.con...@gmail.com> wrote:
>
> > Hi folks,
> >
> > At present, Mesos defaults to compiling with "-O0"; to enable compiler
> > optimizations, the user needs to specify "--enable-optimize".
> >
> > I'd like to propose we change the default, for a few reasons:
> >
> > (1) The autoconf default for CFLAGS/CXXFLAGS is "-O2 -g". Anecdotally,
> > I think most software packages compile with a reasonable level of
> > optimizations enabled by default.
> >
> > (2) I think we should make the default configure flags appropriate for
> > end-users (rather than Mesos developers): developers will be familiar
> > enough with Mesos to tune the configure flags according to their own
> > preferences.
> >
> > (3) The performance consequences of not enabling compiler
> > optimizations can be pretty severe: 5x in a benchmark I just ran, and
> > we've seen between 2x and 30x (!) performance differences for some
> > real-world workloads.
> >
> > Neil
> >
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: build error

[Klaus Ma]

Hi Disha,

According to your log, both `make` and `make check` are passed. It failed
when `make install` because of permission. We should use `root` (by sudo or
root user) to install mesos into default directory; and you can also change
it by `configure --prefix path_to_dir_with_written_permission`

Thanks
Klaus

On Mon, Feb 8, 2016 at 1:27 PM Disha Singh <directionsta...@gmail.com>
wrote:

> A few lines in the end of make check command were :
>
> -*--
>
> [--] 22 tests from ContentType/SchedulerTest (17276 ms total)
>
> [--] Global test environment tear-down
> [==] 920 tests from 117 test cases ran. (652156 ms total)
> [  PASSED  ] 920 tests.
>
>   YOU HAVE 9 DISABLED TESTS
>
> make[3]: Leaving directory `/home/disha/Desktop/mesos/build/src'
> make[2]: Leaving directory `/home/disha/Desktop/mesos/build/src'
> make[1]: Leaving directory `/home/disha/Desktop/mesos/build/src'
>
> *---
>
> After i run make install I get :
>
> --
>
> Making install in .
> make[1]: Entering directory `/home/disha/Desktop/mesos/build'
> make[2]: Entering directory `/home/disha/Desktop/mesos/build'
> make[2]: Nothing to be done for `install-exec-am'.
>  /bin/mkdir -p '/usr/local/lib/pkgconfig'
> /bin/mkdir: cannot create directory ‘/usr/local/lib/pkgconfig’: Permission
> denied
> make[2]: *** [install-pkgconfigDATA] Error 1
> make[2]: Leaving directory `/home/disha/Desktop/mesos/build'
> make[1]: *** [install-am] Error 2
> make[1]: Leaving directory `/home/disha/Desktop/mesos/build'
> make: *** [install-recursive] Error 1
>
> ------
>
> Sorry, for delay in reply.
> Thanks for helping . :)
>
> On Sat, Feb 6, 2016 at 7:27 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > What error did you get?
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
> > On Sat, Feb 6, 2016 at 6:18 PM, Disha Singh <directionsta...@gmail.com>
> > wrote:
> >
> > > It worked fine ,but then some test cases failed even when I didn't
> touch
> > > any files, I am just trying to build mesos the way it is. Is it
> something
> > > that is supposed to happen? I guess no. :(
> > >
> > > On Fri, Jan 29, 2016 at 5:53 AM, Klaus Ma <klaus1982...@gmail.com>
> > wrote:
> > >
> > > > If no requirement on java/python, you can disable them as Vinod said
> > > > (`../configure --disable-java`); it'll also reduce build time :).
> > > >
> > > > 
> > > > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > > > Platform OpenSource Technology, STG, IBM GCG
> > > > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> > > >
> > > > On Fri, Jan 29, 2016 at 4:18 AM, Vinod Kone <vinodk...@apache.org>
> > > wrote:
> > > >
> > > > > On Thu, Jan 28, 2016 at 11:55 AM, Disha Singh <
> > > directionsta...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I am also unable to upgrade maven2 to maven3.
> > > > > >
> > > > >
> > > > > Have you tried asking on the maven user list? You would probably
> get
> > > > better
> > > > > help regarding maven upgrade there.
> > > > >
> > > > > If you are not interested in maven/java bindings, you can try
> > building
> > > > > mesos with --disable-java configure option.
> > > > >
> > > >
> > >
> >
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: Specifying a preferred host with a Resource Request

[Klaus Ma]

Hi Jagadish,

This the requirement for dynamic reservation and persistent volumes :).
Here's the related document:

Reservation: http://mesos.apache.org/documentation/latest/reservation/
Persistent Volume:
http://mesos.apache.org/documentation/latest/persistent-volume/

Thanks
Klaus

On Sun, Feb 7, 2016 at 2:31 AM Jagadish Venkatraman <jagadish1...@gmail.com>
wrote:

> Hi Guangya,
>
> Thanks for the response! Let me provide more background to this request.
>
> *Background:*
> I work on Apache Samza <http://samza.apache.org> , a distributed stream
> processing framework. Currently Samza supports only Yarn as a resource
> manager. (there have been requests to run Samza with mesos). A cluster (200
> nodes 'ish) runs many Samza Jobs (about 3500). Each Samza Job has its own
> framework that requests resources (containers) for the job to run. Each
> such container uses GBs of local state
> <http://radar.oreilly.com/2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html>
>   .
> When such a container(resource) is started on a different host by the
> framework, the local state must be re-bootstrapped.  (this results in a
> long bootstrap time, which is essentially down time).
>
> The same is true for Apache Kafka <http://kafka.apache.org/>, a
> distributed pub-sub logging system.  When a Kafka broker must be restarted
> by the framework, it should ideally be re-started on the same host.
> (otherwise, each broker has to re-bootstrap several GBs of logs from its
> peers before it can start to service a request.)
>
> I'm sure many stateful services have similar requirements.
>
> >> Is it possible that you update your framework logic as this:
> 1) framework get resoruce offer from mesos master
> 2) framework filter the resource offers based on its preferences
>
> I can certainly do that. But, here's my concern:
>
>-  Is the offer for resources to frameworks, 'round robin' across the
>available pool across hosts? I want to ensure that the wait time for a
>resource-wait is bounded.
>- Are there tunables that we can set to be more 'fair' (in terms of
>variety of hosts) when Offers are offered? For example, every framework
>will receive atleast some offers for *every* host (where resources are
>available). Or, all available offers are broadcasted to all frameworks.
>
> Are there alternatives that I can use to support this usecase and ensure
> that the wait time for an available resource is limited (say about a minute
> or two)? . It can still be a best-effort guarantee and not a strict one.
>
>
>
> Thanks again,
> Jagadish
>
> --
> Jagadish
>
>
>
> On Fri, Feb 5, 2016 at 6:46 PM, Guangya Liu <gyliu...@gmail.com> wrote:
>
>> Hi Jagadish,
>>
>> Even though Mesos have the interface of "requestResources", it was not
>> implemented in the built-in allocator at the moment, so the call of 
>> "driver.requestResources
>> (resources);" will not work.
>>
>> Is it possible that you update your framework logic as this:
>> 1) framework get resoruce offer from mesos master
>> 2) framework filter the resource offers based on its preferences
>>
>> The problem for such solution is that the framework sometimes may not get
>> its preferred resources if the preferred resource was offered to other
>> frameworks.
>>
>> Can you please file a JIRA ticket to request implement the API of 
>> "requestResources"?
>> It would be great if you can append some background for your request so
>> that the community can evaluate how to move this forward.
>>
>> Thanks,
>>
>> Guangya
>>
>>
>> On Sat, Feb 6, 2016 at 6:45 AM, Jagadish Venkatraman <
>> jagadish1...@gmail.com> wrote:
>>
>>> I have fair experience in writing frameworks on Yarn. In the Yarn world,
>>> the amClient supports a method where I can specify the preferredHost with
>>> the resource request.
>>>
>>> Is there a way to specify a preferred host with the resource request in
>>> Mesos?
>>>
>>> I currently do:
>>>
>>> driver.requestResources (resources);
>>>
>>> I don't find a way to associate a preferred hostname with a resource
>>> request. A code sample will be really helpful. (for example, I want 1G
>>> mem,
>>> 1cpu core preferrably on host: xyz.aws.com )
>>>
>>> Thanks,
>>> Jagadish
>>>
>>> --
>>> Jagadish V,
>>> Graduate Student,
>>> Department of Computer Science,
>>> Stanford University
>>>
>>
>>
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: build error

[Klaus Ma]

What error did you get?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Sat, Feb 6, 2016 at 6:18 PM, Disha Singh <directionsta...@gmail.com>
wrote:

> It worked fine ,but then some test cases failed even when I didn't touch
> any files, I am just trying to build mesos the way it is. Is it something
> that is supposed to happen? I guess no. :(
>
> On Fri, Jan 29, 2016 at 5:53 AM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > If no requirement on java/python, you can disable them as Vinod said
> > (`../configure --disable-java`); it'll also reduce build time :).
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
> > On Fri, Jan 29, 2016 at 4:18 AM, Vinod Kone <vinodk...@apache.org>
> wrote:
> >
> > > On Thu, Jan 28, 2016 at 11:55 AM, Disha Singh <
> directionsta...@gmail.com
> > >
> > > wrote:
> > >
> > > > I am also unable to upgrade maven2 to maven3.
> > > >
> > >
> > > Have you tried asking on the maven user list? You would probably get
> > better
> > > help regarding maven upgrade there.
> > >
> > > If you are not interested in maven/java bindings, you can try building
> > > mesos with --disable-java configure option.
> > >
> >
>

Re: [Discussion] MESOS-4442: `allocated` may have more resources then `total` in allocator

[Klaus Ma]

Copy that; thanks :).

On Sat, Jan 30, 2016 at 5:04 AM Benjamin Mahler <bmah...@apache.org> wrote:

> Looks like you've seen the comment here:
>
> https://issues.apache.org/jira/browse/MESOS-4442?focusedCommentId=15120754=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15120754
>
> We allow this so that the allocator can be over-allocated. When
> over-allocated, we will not re-offer freed resources until we are no longer
> over-allocated.
>
> On Fri, Jan 22, 2016 at 4:36 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > @team, any comments?
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
> > On Thu, Jan 21, 2016 at 9:31 PM, Klaus Ma <klaus1982...@gmail.com>
> wrote:
> >
> > > Yes, *total*: cpus(*):2 vs. *allocated*: cpus(*):2;cpus(*){REV}:2
> > >
> > > 
> > > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > > Platform OpenSource Technology, STG, IBM GCG
> > > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> > >
> > > On Thu, Jan 21, 2016 at 5:43 PM, Qian Zhang <zhq527...@gmail.com>
> wrote:
> > >
> > >> In the log you posted, it seems total cpus is also 2 rather than 1,
> but
> > it
> > >> seem there are 4 allocated cpus (2 non-revocable and 2 revocable)?
> > >>
> > >> I0121 17:08:09.303431 4284416 hierarchical.cpp:528] Slave
> > >> f2d8b550-ed52-44a4-a35a-1fff81d41391-S0 (9.181.90.153) updated with
> > >> oversubscribed resources  (total: cpus(*):2; mem(*):1024;
> disk(*):1024;
> > >> ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024;
> disk(*):1024;
> > >> ports(*):[31000-32000]; *cpus(*){REV}:2*)
> > >>
> > >>
> > >> Thanks,
> > >> Qian Zhang
> > >>
> > >> On Thu, Jan 21, 2016 at 5:25 PM, Klaus Ma <klaus1982...@gmail.com>
> > wrote:
> > >>
> > >> > Hi team,
> > >> >
> > >> > When I double-check the feature interaction between Optimistic Offer
> > >> Phase
> > >> > 1 & Oversubscription, I found an issue that `allocated` may have
> more
> > >> > resources then `total` in allocator when enable Oversubscription.
> I'd
> > >> like
> > >> > to get your input on whether this is design behaviour, although the
> > >> impact
> > >> > is low: 1.) allocator will not offer this delta resources, 2) QoS
> > >> > Controller will correct it later by killing the executor.
> Personally,
> > >> I'd
> > >> > like to keep this assumption in allocator: slave.total always
> contains
> > >> > slave.allocated.
> > >> >
> > >> > Here's the steps:
> > >> >
> > >> > T1: in cluster, cpus=2: one is revocable and the other one is
> > >> nonRevocable
> > >> > T2: framework1 get offer cpus=2, launch task but estimator report
> > empty
> > >> > resources before executor launched
> > >> > T3: slave.total is updated to cpus=1 in
> > >> > HierarchicalAllocatorProcess::updateSlave
> > >> > T4: in allocate(), slave.total (cpus=1) < slave.allocated (cpus=2)
> > >> >
> > >> > Here's the log I got:
> > >> >
> > >> > I0121 17:08:09.303431 4284416 hierarchical.cpp:528] Slave
> > >> > f2d8b550-ed52-44a4-a35a-1fff81d41391-S0 (9.181.90.153) updated with
> > >> > oversubscribed resources  (total: cpus(*):2; mem(*):1024;
> > disk(*):1024;
> > >> > ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024;
> > disk(*):1024;
> > >> > ports(*):[31000-32000]; *cpus(*){REV}:2*)
> > >> >
> > >> > Please refer to MESOS-4442 for more detail.
> > >> >
> > >> > 
> > >> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > >> > Platform OpenSource Technology, STG, IBM GCG
> > >> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> > >> >
> > >>
> > >
> > >
> >
>
-- 

Regards,

Da (Klaus), Ma (马达), PMP® | Advisory Software Engineer
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Re: build error

[Klaus Ma]

If no requirement on java/python, you can disable them as Vinod said
(`../configure --disable-java`); it'll also reduce build time :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Fri, Jan 29, 2016 at 4:18 AM, Vinod Kone <vinodk...@apache.org> wrote:

> On Thu, Jan 28, 2016 at 11:55 AM, Disha Singh <directionsta...@gmail.com>
> wrote:
>
> > I am also unable to upgrade maven2 to maven3.
> >
>
> Have you tried asking on the maven user list? You would probably get better
> help regarding maven upgrade there.
>
> If you are not interested in maven/java bindings, you can try building
> mesos with --disable-java configure option.
>

Re: [Discussion] MESOS-4442: `allocated` may have more resources then `total` in allocator

[Klaus Ma]

@team, any comments?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Thu, Jan 21, 2016 at 9:31 PM, Klaus Ma <klaus1982...@gmail.com> wrote:

> Yes, *total*: cpus(*):2 vs. *allocated*: cpus(*):2;cpus(*){REV}:2
>
> ----
> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> Platform OpenSource Technology, STG, IBM GCG
> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>
> On Thu, Jan 21, 2016 at 5:43 PM, Qian Zhang <zhq527...@gmail.com> wrote:
>
>> In the log you posted, it seems total cpus is also 2 rather than 1, but it
>> seem there are 4 allocated cpus (2 non-revocable and 2 revocable)?
>>
>> I0121 17:08:09.303431 4284416 hierarchical.cpp:528] Slave
>> f2d8b550-ed52-44a4-a35a-1fff81d41391-S0 (9.181.90.153) updated with
>> oversubscribed resources  (total: cpus(*):2; mem(*):1024; disk(*):1024;
>> ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024;
>> ports(*):[31000-32000]; *cpus(*){REV}:2*)
>>
>>
>> Thanks,
>> Qian Zhang
>>
>> On Thu, Jan 21, 2016 at 5:25 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>>
>> > Hi team,
>> >
>> > When I double-check the feature interaction between Optimistic Offer
>> Phase
>> > 1 & Oversubscription, I found an issue that `allocated` may have more
>> > resources then `total` in allocator when enable Oversubscription. I'd
>> like
>> > to get your input on whether this is design behaviour, although the
>> impact
>> > is low: 1.) allocator will not offer this delta resources, 2) QoS
>> > Controller will correct it later by killing the executor. Personally,
>> I'd
>> > like to keep this assumption in allocator: slave.total always contains
>> > slave.allocated.
>> >
>> > Here's the steps:
>> >
>> > T1: in cluster, cpus=2: one is revocable and the other one is
>> nonRevocable
>> > T2: framework1 get offer cpus=2, launch task but estimator report empty
>> > resources before executor launched
>> > T3: slave.total is updated to cpus=1 in
>> > HierarchicalAllocatorProcess::updateSlave
>> > T4: in allocate(), slave.total (cpus=1) < slave.allocated (cpus=2)
>> >
>> > Here's the log I got:
>> >
>> > I0121 17:08:09.303431 4284416 hierarchical.cpp:528] Slave
>> > f2d8b550-ed52-44a4-a35a-1fff81d41391-S0 (9.181.90.153) updated with
>> > oversubscribed resources  (total: cpus(*):2; mem(*):1024; disk(*):1024;
>> > ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024;
>> > ports(*):[31000-32000]; *cpus(*){REV}:2*)
>> >
>> > Please refer to MESOS-4442 for more detail.
>> >
>> > 
>> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
>> > Platform OpenSource Technology, STG, IBM GCG
>> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
>> >
>>
>
>

Re: [Discussion] MESOS-4442: `allocated` may have more resources then `total` in allocator

[Klaus Ma]

Yes, *total*: cpus(*):2 vs. *allocated*: cpus(*):2;cpus(*){REV}:2


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Thu, Jan 21, 2016 at 5:43 PM, Qian Zhang <zhq527...@gmail.com> wrote:

> In the log you posted, it seems total cpus is also 2 rather than 1, but it
> seem there are 4 allocated cpus (2 non-revocable and 2 revocable)?
>
> I0121 17:08:09.303431 4284416 hierarchical.cpp:528] Slave
> f2d8b550-ed52-44a4-a35a-1fff81d41391-S0 (9.181.90.153) updated with
> oversubscribed resources  (total: cpus(*):2; mem(*):1024; disk(*):1024;
> ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024;
> ports(*):[31000-32000]; *cpus(*){REV}:2*)
>
>
> Thanks,
> Qian Zhang
>
> On Thu, Jan 21, 2016 at 5:25 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > Hi team,
> >
> > When I double-check the feature interaction between Optimistic Offer
> Phase
> > 1 & Oversubscription, I found an issue that `allocated` may have more
> > resources then `total` in allocator when enable Oversubscription. I'd
> like
> > to get your input on whether this is design behaviour, although the
> impact
> > is low: 1.) allocator will not offer this delta resources, 2) QoS
> > Controller will correct it later by killing the executor. Personally, I'd
> > like to keep this assumption in allocator: slave.total always contains
> > slave.allocated.
> >
> > Here's the steps:
> >
> > T1: in cluster, cpus=2: one is revocable and the other one is
> nonRevocable
> > T2: framework1 get offer cpus=2, launch task but estimator report empty
> > resources before executor launched
> > T3: slave.total is updated to cpus=1 in
> > HierarchicalAllocatorProcess::updateSlave
> > T4: in allocate(), slave.total (cpus=1) < slave.allocated (cpus=2)
> >
> > Here's the log I got:
> >
> > I0121 17:08:09.303431 4284416 hierarchical.cpp:528] Slave
> > f2d8b550-ed52-44a4-a35a-1fff81d41391-S0 (9.181.90.153) updated with
> > oversubscribed resources  (total: cpus(*):2; mem(*):1024; disk(*):1024;
> > ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024;
> > ports(*):[31000-32000]; *cpus(*){REV}:2*)
> >
> > Please refer to MESOS-4442 for more detail.
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
>

[Discussion] MESOS-4442: `allocated` may have more resources then `total` in allocator

[Klaus Ma]

Hi team,

When I double-check the feature interaction between Optimistic Offer Phase
1 & Oversubscription, I found an issue that `allocated` may have more
resources then `total` in allocator when enable Oversubscription. I'd like
to get your input on whether this is design behaviour, although the impact
is low: 1.) allocator will not offer this delta resources, 2) QoS
Controller will correct it later by killing the executor. Personally, I'd
like to keep this assumption in allocator: slave.total always contains
slave.allocated.

Here's the steps:

T1: in cluster, cpus=2: one is revocable and the other one is nonRevocable
T2: framework1 get offer cpus=2, launch task but estimator report empty
resources before executor launched
T3: slave.total is updated to cpus=1 in
HierarchicalAllocatorProcess::updateSlave
T4: in allocate(), slave.total (cpus=1) < slave.allocated (cpus=2)

Here's the log I got:

I0121 17:08:09.303431 4284416 hierarchical.cpp:528] Slave
f2d8b550-ed52-44a4-a35a-1fff81d41391-S0 (9.181.90.153) updated with
oversubscribed resources  (total: cpus(*):2; mem(*):1024; disk(*):1024;
ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024;
ports(*):[31000-32000]; *cpus(*){REV}:2*)

Please refer to MESOS-4442 for more detail.

----
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

Re: Install some 3rdparty packages needed for building Mesos modules

[Klaus Ma]

+1 on installing 3rdpart packages.

I used to build a C++ Mesos framework with local/system packages (protobuf,
boost), but protobuf failed because of header file backward compatibility.
I have to go through the Mesos build to include 3rdpart packages and got
following Makefile:
https://github.com/klaus1982/mesos-ping/blob/master/Makefile

It'll be great if we can ship the 3rdpackages when installing Mesos.



Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Jan 20, 2016 at 6:03 AM, Kapil Arya <ka...@mesosphere.io> wrote:

> Hi All,
>
> I wanted to get your opinion on installing the 3rdparty packages glog,
> protobuf, boost and picojson[1] when installing Mesos itself. These
> packages are required to build Mesos modules.
>
> Currently, a module write has to manually install these 3rdparty
> packages, either system-wide or locally, and update the compilation
> flags such as CPPFLAGS to point to the installation which is
> error-prone. Further, one might have a system-wide installation with
> the wrong package version, causing even more headache.
>
> The proposal here is to install these 3rdparty packages when
> installing Mesos. To avoid any conflicts with system-wide or local
> installation, we can install them as follows:
>
> ${PREFIX}/include/mesos/3rdparty -- for header files
> ${PREFIX}//mesos/3rdparty -- for library files (LIBDIR can be
> lib or lib64 depending upon the installation)
>
> where PREFIX refers to the `--prefix` flag for Mesos configure script.
>
> We would then update `mesos.pc` with the correct flags so that a
> module write can simply use `pkg-config` to get all the required
> flags.
>
> I have created an issue
> https://issues.apache.org/jira/browse/MESOS-4434 to track this.
>
> Best,
> Kapil
>
>
> [1]: picojson is currently installed in ${PREFIX}/include. See
> https://issues.apache.org/jira/browse/MESOS-3909
>

Re: Operator HTTP endpoints

[Klaus Ma]

+1 on *Request -> *Info :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Mon, Jan 18, 2016 at 7:38 PM, Alex Rukletsov <a...@mesosphere.com> wrote:

> I agree that the borderline between being related to the request and being
> related to the way the request is processed is vague. I think having
> *Request objects is a good idea, both for documenting the JSON request
> schema and differentiating between the request and storage data.
>
> However, I'm a bit reluctant to nest QuotaInfo in QuotaRequest. Since
> QuotaInfo is what we persist in the master (and the registry), it may have
> fields that are not relevant to the request (e.g., "principal"), hence we
> should validate these fields are not set and document that. I would suggest
> to untie *Request and *Info protos and perform conversions *Request ->
> *Info.
>
> On Fri, Jan 15, 2016 at 10:09 PM, Benjamin Mahler <bmah...@apache.org>
> wrote:
>
> > The distinction between being related to the request vs. how the request
> > should be processed seems arguable, do you feel that we'll be able to
> make
> > the distinction easily?
> >
> > One thought.. if there are top-level Request objects:
> >
> > message QuotaRequest {
> >   optional bool force;
> >   required QuotaInfo quota_info;
> > }
> >
> > Then we could map query parameters to fields, so the following are
> > equivalent:
> >
> > /quota?force=true
> > {
> >   "quota_info": { ...}
> > }
> >
> > /quota
> > {
> >   "force": true,
> >   "quota_info": { ...}
> > }
> >
> > The user can choose between query parameters vs json body as they please,
> > and we still have a single specification of the request.
> >
> > On Thu, Jan 14, 2016 at 10:09 AM, Alex Rukletsov <a...@mesosphere.com>
> > wrote:
> >
> > > Folks,
> > >
> > > I would like to gather your opinions about a concern related to
> operator
> > > HTTP endpoints. From one side we agreed that for simplicity and
> > consistency
> > > we should bake all request data in a single JSON. From the other side,
> > some
> > > parameters, like a force flag, do not really belong to request JSON (as
> > > pointed out by some SRE guys in comment to MESOS-3914 [1]). The force
> > flag
> > > is not really related to the request itself, but more to the way the
> > > request should be processed.
> > >
> > > To my knowledge, currently we use the force flag in two places:
> > >   * Subscribe call in framework API.
> > >   * Quota set request.
> > >
> > > Currently we have the 'force' field in JSON both cases.
> > >
> > > I would like us to agree on the way we write endpoints and clean-up
> > > existing ones *before* we release Mesos 1.0. Looking forward to your
> > > feedback.
> > >
> > > AlexR
> > >
> > > [1] https://issues.apache.org/jira/browse/MESOS-3914
> > >
> >
>

Re: Operator HTTP endpoints

[Klaus Ma]

+1 for adding "force" in JSON body; used to be a JEE developer for several
years, there's a general practice to us: query argument (e.g.
"?force=true") is ONLY used by 'GET', all "data" are in body if action is
'POST' (e.g. UPDATE, DELETE). The reason is complex: security, meaning of
HTTP action, search engine, and so on.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Fri, Jan 15, 2016 at 9:40 AM, Marco Massenzio <m.massen...@gmail.com>
wrote:

> +1 for Lukas's suggestion to have `force` be a query argument (eg,
> ``?force=true``) and the body respect the ``Content-Type`` header.
>
> (although I was part of the conversation of adding the ``force`` flag to
> the SUBSCRIBE message, I forget the exact details of that one - @Vinod may
> be able to chime in).
>
> Thanks for getting the conversation started!
>
> --
> *Marco Massenzio*
> http://codetrips.com
>
> On Thu, Jan 14, 2016 at 11:09 AM, Alex Rukletsov <a...@mesosphere.com>
> wrote:
>
> > Folks,
> >
> > I would like to gather your opinions about a concern related to operator
> > HTTP endpoints. From one side we agreed that for simplicity and
> consistency
> > we should bake all request data in a single JSON. From the other side,
> some
> > parameters, like a force flag, do not really belong to request JSON (as
> > pointed out by some SRE guys in comment to MESOS-3914 [1]). The force
> flag
> > is not really related to the request itself, but more to the way the
> > request should be processed.
> >
> > To my knowledge, currently we use the force flag in two places:
> >   * Subscribe call in framework API.
> >   * Quota set request.
> >
> > Currently we have the 'force' field in JSON both cases.
> >
> > I would like us to agree on the way we write endpoints and clean-up
> > existing ones *before* we release Mesos 1.0. Looking forward to your
> > feedback.
> >
> > AlexR
> >
> > [1] https://issues.apache.org/jira/browse/MESOS-3914
> >
>

Re: 回复： Request Mesos contributor role

[Klaus Ma]

Sorry for the inconvenience :). Beside add contributor's request in
English, the other are something like "Hi", "Keep in touch" ...


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Jan 13, 2016 at 5:17 PM, Alex Rukletsov <a...@mesosphere.com> wrote:

> Could a native speaker please translate it into English?
>
> On Wed, Jan 13, 2016 at 9:25 AM, Klaus Ma <klaus1982...@gmail.com> wrote:
> > 多多沟通 :).
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform OpenSource Technology, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
> > On Wed, Jan 13, 2016 at 4:22 PM, 陈强 <chenqi...@qiyi.com> wrote:
> >
> >> 哈哈，多谢！以后有问题可以一起讨论，沟通。
> >>
> >> On 2016年01月12日 16:26, 黑洞 wrote:
> >>
> >>> 欢迎，很高兴，能在此碰到同行，多多交流。
> >>>   --
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> https://github.com/heidsoft
> >>>
> >>> Research: Cloud computing (cloud security), Big Data (data analysis)
> >>> Key-Learning: Linux / C / C + + / JAVA / Python/ R
> >>> Email: heids...@qq.com
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> -- 原始邮件 --
> >>> 发件人: "陈强"<chenqi...@qiyi.com>;
> >>> 发送时间: 2016年1月12日(星期二) 下午3:32
> >>> 收件人: "dev"<dev@mesos.apache.org>;
> >>> 主题: Re: Request Mesos contributor role
> >>>
> >>>
> >>>
> >>> Hi Tim,
> >>>
> >>> My username is: chenqiang
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On 2016年01月12日 15:16, Du, Fan wrote:
> >>>
> >>>>
> >>>> On 2016/1/12 15:02, Timothy Chen wrote:
> >>>>
> >>>>> Have you created a jira user already? I searched for Du Fan and
> >>>>> couldn't find anything.
> >>>>>
> >>>>> Let me know what's your jira username.
> >>>>>
> >>>> fan.du
> >>>>
> >>>> Thanks Tim.
> >>>>
> >>>> Tim
> >>>>>
> >>>>> On Mon, Jan 11, 2016 at 10:22 PM, Du, Fan <fan...@intel.com> wrote:
> >>>>>
> >>>>>> hi Mesos committer
> >>>>>>
> >>>>>> I want to assign myself to a JIRA issue I'm about to create,
> >>>>>> As from the
> >>>>>> http://mesos.apache.org/documentation/latest/submitting-a-patch/
> >>>>>> Could you please add me to the Mesos contributor list?
> >>>>>>
> >>>>>> thanks a lot.
> >>>>>>
> >>>>>
> >>>>>
> >> --
> >> 陈强
> >>
> >> 技术产品中心
> >>
> >> 上海市徐汇区宜山路1388号民润商务大厦6层 / 201103
> >>
> >> 6/F. Minrun Business Building, No. 1388 Yishan Road
> >>
> >> Xuhui District, Shanghai 201103, P.R.China
> >>
> >> 爱奇艺公司
> >>
> >> QIYI.COM,INC.
> >>
> >> 手机：＋86 15900964316
> >> 电话：＋86 021 5451 9520-8377
> >> 邮箱：chenqi...@qiyi.com
> >> 网址：www.iQIYI.com
> >>
> >>
>

Re: 回复： Request Mesos contributor role

[Klaus Ma]

多多沟通 :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform OpenSource Technology, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Jan 13, 2016 at 4:22 PM, 陈强 <chenqi...@qiyi.com> wrote:

> 哈哈，多谢！以后有问题可以一起讨论，沟通。
>
> On 2016年01月12日 16:26, 黑洞 wrote:
>
>> 欢迎，很高兴，能在此碰到同行，多多交流。
>>   --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> https://github.com/heidsoft
>>
>> Research: Cloud computing (cloud security), Big Data (data analysis)
>> Key-Learning: Linux / C / C + + / JAVA / Python/ R
>> Email: heids...@qq.com
>>
>>
>>
>>
>>
>>
>>
>> -- 原始邮件 --
>> 发件人: "陈强"<chenqi...@qiyi.com>;
>> 发送时间: 2016年1月12日(星期二) 下午3:32
>> 收件人: "dev"<dev@mesos.apache.org>;
>> 主题: Re: Request Mesos contributor role
>>
>>
>>
>> Hi Tim,
>>
>> My username is: chenqiang
>>
>> Thanks.
>>
>>
>> On 2016年01月12日 15:16, Du, Fan wrote:
>>
>>>
>>> On 2016/1/12 15:02, Timothy Chen wrote:
>>>
>>>> Have you created a jira user already? I searched for Du Fan and
>>>> couldn't find anything.
>>>>
>>>> Let me know what's your jira username.
>>>>
>>> fan.du
>>>
>>> Thanks Tim.
>>>
>>> Tim
>>>>
>>>> On Mon, Jan 11, 2016 at 10:22 PM, Du, Fan <fan...@intel.com> wrote:
>>>>
>>>>> hi Mesos committer
>>>>>
>>>>> I want to assign myself to a JIRA issue I'm about to create,
>>>>> As from the
>>>>> http://mesos.apache.org/documentation/latest/submitting-a-patch/
>>>>> Could you please add me to the Mesos contributor list?
>>>>>
>>>>> thanks a lot.
>>>>>
>>>>
>>>>
> --
> 陈强
>
> 技术产品中心
>
> 上海市徐汇区宜山路1388号民润商务大厦6层 / 201103
>
> 6/F. Minrun Business Building, No. 1388 Yishan Road
>
> Xuhui District, Shanghai 201103, P.R.China
>
> 爱奇艺公司
>
> QIYI.COM,INC.
>
> 手机：＋86 15900964316
> 电话：＋86 021 5451 9520-8377
> 邮箱：chenqi...@qiyi.com
> 网址：www.iQIYI.com
>
>

Re: How to contribute to ./support/apply-reviews.py?

[Klaus Ma]

Thanks very much. Yes, I have post a RR at
https://reviews.apache.org/r/42191/, no sure what happen to the working
branch that I can not post review request :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Jan 12, 2016 at 8:56 PM, haosdent <haosd...@gmail.com> wrote:

> Also through review board? https://reviews.apache.org/r/39420/
>
> On Tue, Jan 12, 2016 at 8:42 PM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > Hi team,
> >
> > Is there any guidance on how to contribute to apply-reviews.py? It seems
> I
> > can not post RR as usual.
> > I added a parameters to apply the patches quiet. So it's easy for us to
> > apply ~10 patches with -c.
> >
> > If any suggestion, please let me know.
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform Symphony/DCOS Development & Support, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>

How to contribute to ./support/apply-reviews.py?

[Klaus Ma]

Hi team,

Is there any guidance on how to contribute to apply-reviews.py? It seems I
can not post RR as usual.
I added a parameters to apply the patches quiet. So it's easy for us to
apply ~10 patches with -c.

If any suggestion, please let me know.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

Re: How to contribute to ./support/apply-reviews.py?

[Klaus Ma]

Hi Artem,

Yes, just try `-n` option; it works for me; great to know :).

Both JIRA & PR are closed.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Jan 13, 2016 at 7:26 AM, Artem Harutyunyan <ar...@mesosphere.io>
wrote:

> Hi Klaus,
>
> The workflow is the same (JIRA + ReviewBoard). The script, however, already
> seems to support what you need though (look into `-n` option).
>
> Cheers,
> Artem.
>
> On Tue, Jan 12, 2016 at 4:42 AM, Klaus Ma <klaus1982...@gmail.com> wrote:
>
> > Hi team,
> >
> > Is there any guidance on how to contribute to apply-reviews.py? It seems
> I
> > can not post RR as usual.
> > I added a parameters to apply the patches quiet. So it's easy for us to
> > apply ~10 patches with -c.
> >
> > If any suggestion, please let me know.
> >
> > 
> > Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> > Platform Symphony/DCOS Development & Support, STG, IBM GCG
> > +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >
>

Docker Executor in Mesos

[Klaus Ma]

Hi team,

Currently, if we run docker in mesos, we'll start docker-executor, "docker
run" and container in slave hosts. So why not use one executor to launch
docker tasks? One reason I can image is compatibility of docker API. If
there're thousands of tasks in a powerful task, do you get docker startup
performance issue?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

Do we have document on HTTP endpoints?

[Klaus Ma]

Hi team,

Is there any document to introduce our HTTP endpoints such as
"tasks/state", "slaves/state" and so on. For now, I just get a document
about metrics here (http://mesos.apache.org/documentation/latest/monitoring/
).

Regards

Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

Re: Do we have document on HTTP endpoints?

[Klaus Ma]

MESOS-3831 is the same requirement; waiting for it :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Dec 2, 2015 at 12:30 PM, tommy xiao <xia...@gmail.com> wrote:

> Also I found this ISSUE:  https://issues.apache.org/jira/browse/MESOS-3831
>
> 2015-12-02 12:28 GMT+08:00 tommy xiao <xia...@gmail.com>:
>
> > Currently  this is official docs as you pointed url.
> >
> > 2015-12-02 10:47 GMT+08:00 Klaus Ma <klaus1982...@gmail.com>:
> >
> >> Hi team,
> >>
> >> Is there any document to introduce our HTTP endpoints such as
> >> "tasks/state", "slaves/state" and so on. For now, I just get a document
> >> about metrics here (
> >> http://mesos.apache.org/documentation/latest/monitoring/
> >> ).
> >>
> >> Regards
> >> 
> >> Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
> >> Platform Symphony/DCOS Development & Support, STG, IBM GCG
> >> +86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me
> >>
> >
> >
> >
> > --
> > Deshi Xiao
> > Twitter: xds2000
> > E-mail: xiaods(AT)gmail.com
> >
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>

Re: Dynamic vs. implicit roles

[Klaus Ma]

@Neil, just want to confirm about ACL, do you mean we will load role info
from 3rd part application, e.g. LDAP?

And as I mentioned in both design doc, why not build a RoleManager as
plugin for them? Both features are required following operator:
1. check: check whether role is available
2. create: create role in Master
3. update: update role info
4. destroy: delete the role
5. persist:
6. query: query from role manager.
   master/allocator need role info during the operation

For the life cycle of role, "update" should be period & manual (API/HTT);
others are trigger by API/HTTP.


----
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Dec 1, 2015 at 1:13 PM, James Peach <jor...@gmail.com> wrote:

>
> > On Nov 30, 2015, at 6:53 PM, YongQiao Wang <jamesyongq...@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > Currently, there are two proposals on how to improve role management in
> > Mesos:
> >
> > (a) Dynamic roles (MESOS-3177): roles are stored in the registry
> > and queried/added/deleted/removed via HTTP endpoints. I posted a design
> doc
> > here:
> >
> https://docs.google.com/document/d/1OIgceqpsjV3-_LGF83IMAFnrh1Ea3Zc16w9kWWPpUj4/edit#
> >
> > (b) Implicit Roles (MESOS-3988): any role will be allowed, subject to the
> > ACL/authorization system. In a sense, "all roles" exist, so there is no
> > need to store the set of legal roles or provide endpoints to modify them.
> > Neil also has posted a design doc here:
> >
> https://docs.google.com/document/d/1SCFfrBd4edSY3bVCMrNJYMxIVllD0bHJuGmgG-4vCXA/edit?usp=sharing
>
> Implicit roles seem to behave more like groups in an OS where the role
> string is more gid_t which can be consumed directly by the kernel. I think
> that this is moving in the right direction since you shouldn't need to
> exhaustively know all the roles as long as you can attribute resources to
> them correctly. I can imagine that in future you could have an external
> identity service that signs a binding between frameworks and roles that
> Mesos can verify.
>
> > We have discussed in the previous session, append the discussion history
> as
> > below. Let's have a further discussion to choose a better solution
> between
> > them, any comments and feedbacks would be very welcome!
> >
> > - Original message -
> > From: Yong Qiao Wang/China/IBM
> > To: n...@mesosphere.io
> > Cc: a...@mesosphere.io, b...@mesosphere.io, Qian AZ
> Zhang/China/IBM@IBMCN,
> > yongf...@ca.ibm.com, jamesyongq...@gmail.com
> > Subject: Re: Dynamic vs. implicit roles
> > Date: Tue, Dec 1, 2015 10:27 AM
> >
> > Thanks Neil.
> >
> >> With implicit roles, that would involve:
> >>
> >> 1. Choosing a role name
> >> 2. Configuring weights, ACLs, and quotas for the role.
> >> 3. Configuring applications/frameworks to register using that role.
> >
> > [Yong Qiao] If applications/frameworks do not follow your rules, and
> > register with another role, then how to prevent? and do we will still
> > create this undesirable role in Mesos? Maybe we can only relay on ACLs to
> > avoid this, but according to my understanding, ACLs is not required in
> > Mesos. In addition, I am not sure whether it is make sence to use ACLs
> for
> > role validation.
> >
> > Regards!
> > *Yong Qiao Wang*
> >
> > Re: Dynamic vs. implicit roles
> > 6:55 AM
> > Neil Conway to me (cc), Yong Feng, Qian AZ Zhang, b...@mesosphere.io,
> Adam
> > Bordelon
> > Show more
> > Hi Yong,
> >
> > Thanks for your feedback.
> >
> > On Mon, Nov 30, 2015 at 2:36 PM, Yong Feng <yongf...@ca.ibm.com> wrote:
> >
> > 2. MESOS-3988 treats it as part of framework API. I assume that the new
> > created implicit role only has a kind of default resource plan like
> > weight/quota.
> >
> >
> > This is not the case: if you configure a weight/quota/ACL for a role,
> that
> > configuration will be used whenever a framework tries to register using
> > that role. So you can configure a non-default weight/quota/etc. for an
> > implicit role just as you would with a statically or dynamically
> configured
> > role.
> >
> > We still rely on management API such as quota management to further
> > configure the resource plan.
> >
> > For the use case that in a company/organization, all resources are
> > allocated according to pre-defined budget plan. We will need admin to
> > create role, con

What's the scope/version of v1 HTTP API?

[Klaus Ma]

Hi team,

My understanding to v1 HTTP API is that, it's used for API compatibility
between different mesos version. But what's scope of v1 HTTP API? Before
Mesos 1.0? And how/when should we upgrade API version?


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

Re: Initial leader election

[Klaus Ma]

I'd suggest to open a JIRA to trace issue; I think you can append
master.log & slave.log for owner reference.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Nov 24, 2015 at 8:45 PM, Guilherme Moro <guilherme.m...@ammeon.com>
wrote:

> Hi,
>
> I'm having a problem while trying to create the initial cluster, no leader
> is elected.
> For a start, let me explain my setup:
> 3 nodes
> 3 zookeepers
> 3 mesos-master services, configured as initctl services and controlled by
> puppet, RPM's installed are from the RHEL repository at mesosphere
> (installed through puppet as well), running on RHEL 6.6
> Quorum is set to 2, as expected, all the remaining configs were double
> checked and appears to be correct.
> Most of times I can get the cluster to bootstrap after rebooting the nodes
> (sometimes more than once).
> The whole thing resembles a bit
> https://issues.apache.org/jira/browse/MESOS-2148 and
> https://issues.apache.org/jira/browse/MESOS-2014
>
> Even when I get the master elected, sometimes another couple of reboots or
> restarts of the services are needed to get all the slave nodes added (they
> are the same nodes as the masters).
>
> I can quite easily reproduce this behavior, if someone cares to look at
> logs tell me exactly what to collect and what logging flags I should
> enable.
>
> So, should I maybe open a bug or is there any trick to bootstrap the
> cluster that I'm losing here.
>
> Regards,
>
> Guilherme Moro
>
> --
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the system manager.
> This message contains confidential information and is intended only for the
> individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail.
>
>

Re: Configuration file?

[Klaus Ma]

+1, that's helpful :).

For the detail of implementing such as auto re-load, I think we can let
owner/shepherd to decide :).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Mon, Nov 23, 2015 at 9:40 PM, Adam Avilla <a...@avil.la> wrote:

> +1 I think it would be helpful.
>
> This may be orthogonal / feature creep, but would it be possible to have
> the config file be able to be safely reloaded with a HUP or appropriate
> signal?
>
> On Mon, Nov 23, 2015 at 5:32 AM, Guangya Liu <gyliu...@gmail.com> wrote:
>
> > +1000, introducing a new configuration file for mesos master and slave
> can
> > help end user take the configuration file as the source of all flags.
> >
> > The OpenStack is also using same way to manage all of the flags, it is
> > putting all flags into a configuration file and the configuration file
> > including all flag examples. Most of the flags are disabled by default
> and
> > the end user can just enable those flags based on his requirement.
> >
> > Also the flags in the configuration file can be classified to different
> > groups for a better management, and mesos can also follow this to
> classify
> > those flags to different groups, such as ACL, Cluster, framework etc.
> >
> >
> > On Mon, Nov 23, 2015 at 9:08 PM, Alexander Rojas <
> alexan...@mesosphere.io>
> > wrote:
> >
> > > Hey guys,
> > >
> > > Over the time I’ve been involved in Mesos I’ve seen that we went from a
> > > handful of flags to around 42 supported flags in the master. At this
> > point
> > > I’m wondering if perhaps we should support a configuration file in
> > > conjunction (or instead of) with all the command flags.
> > >
> > > My intuition is that it will make it easier for operators as well as
> for
> > > debuggers to be able to replicate configurations easier.
> > >
> > > Any comments on this idea?
> >
>
>
>
> --
> /adam
>

Re: [jira] [Commented] (MESOS-3946) Test for role management

[Klaus Ma]

I think we can use this ticket for integration test for dynamic role EPIC
:).


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Tue, Nov 24, 2015 at 11:21 AM, Yong Qiao Wang (JIRA) <j...@apache.org>
wrote:

>
> [
> https://issues.apache.org/jira/browse/MESOS-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023691#comment-15023691
> ]
>
> Yong Qiao Wang commented on MESOS-3946:
> ---
>
> The original idea is to add the test for role add/update/remove endpoint
> in this ticket, maybe we should add the test in the related ticket rather
> than issue a new separated ticket. I will mark this as an invalid task
> later. Thanks!
>
> > Test for role management
> > 
> >
> > Key: MESOS-3946
> > URL: https://issues.apache.org/jira/browse/MESOS-3946
> > Project: Mesos
> >  Issue Type: Task
> >Reporter: Yong Qiao Wang
> >Assignee: Yong Qiao Wang
> >
> > Add test for role dynamic configuration.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Re: Add JIRA ticket# to `TODO`s in comments

[Klaus Ma]

+1, JIRA will include more discussion and we can close it when it has been
improved.


Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

On Wed, Nov 11, 2015 at 5:11 PM, Alexander Rojas <alexan...@mesosphere.io>
wrote:

> +1
>
> This also provides a way of removing TODO’s since they are traceable. If
> you look in the code, there are TODO’s which are no relevant anymore or
> probably cannot be understood from their actual context.
>
> > On 08 Nov 2015, at 05:50, Kapil Arya <ka...@mesosphere.io> wrote:
> >
> > Folks,
> >
> > I wanted to bring up a style issue related to the TODO tag in comments. I
> > have filed a Jira ticket (
> https://issues.apache.org/jira/browse/MESOS-3850)
> > with the following description:
> >
> > Currently, we have a TODO() tags to note
> stuff
> > has "should be"/"has to be" done in future. While this provides us with
> > some notion of accounting, it's not enough.
> >
> > The author listed in the TODO comment should be considered the
> "Reporter",
> > but not necessarily the "Assignee". Further, since the stuff "should
> > be"/"has to be" done, why not have a Jira issue tracking it?
> >
> > We can use TODO(MESOS-XXX) or TODO(:MESOS-XXX) or something
> > similar. Finally, we might wan to consider adding this to the style guide
> > to make it a soft/hard requirement.
> >
> >
> > Are there any opinions/suggestions on this one?
> >
> > Best,
> > Kapil
>
>

Re: More Project Structure in JIRA

[Klaus Ma]

+1

On Sun, Oct 25, 2015 at 11:57 PM, Shuai Lin <linshuai2...@gmail.com> wrote:

> +1
>
> On Wed, Oct 21, 2015 at 12:55 AM, Greg Mann <g...@mesosphere.io> wrote:
>
> > +1
> >
> > On Tue, Oct 20, 2015 at 9:50 AM, tommy xiao <xia...@gmail.com> wrote:
> >
> > > +1 Yes please!
> > >
> > > 2015-10-19 16:09 GMT+08:00 Alexander Rojas <alexan...@mesosphere.io>:
> > >
> > > > +1 Yes please!
> > > >
> > > > > On 15 Oct 2015, at 10:11, Bernd Mathiske <be...@mesosphere.io>
> > wrote:
> > > > >
> > > > > Proposal: in extension of today’s limited two-level (epic, task)
> > > > approach, make full use of expressive power already available in JIRA
> > to
> > > > provide more structure for larger projects to facilitate planning,
> > > > tracking, and reporting. This will facilitate dynamically planning of
> > > > sub-projects, which will make us more agile.
> > > > >
> > > > > The general idea is to use links between epics to provide a
> recursive
> > > > hierarchical structure, with which one can span trees or DAGs of
> > > > arbitrarily large projects. This does not mean that we want to plan
> > > > everything in minute detail before starting to work. On the contrary.
> > > > >
> > > > > You can start anywhere in the eventual tree and express part of the
> > > > overall effort, maybe say a short epic with a few task tickets. Then
> > you
> > > > can LATER make this epic a dependency for a larger effort.
> > > > >
> > > > > Conversely, you can subdivide a task in the epic into subtasks.
> > > However,
> > > > this does not mean that you have to literally use the feature
> “subtask”
> > > in
> > > > JIRA for this. Instead, staying recursive in our JIRA grammar, so to
> > > speak,
> > > > convert the task to an epic and then create ordinary tasks in it to
> > > > represent subtasks.
> > > > >
> > > > > Now the task cannot be a task in its parent epic anymore. We fix
> this
> > > by
> > > > putting in a link of type "blocks" to the parent. When you then look
> at
> > > the
> > > > parent, it still holds a number of tasks, and it has one dependency
> on
> > an
> > > > epic (to which you can add more).
> > > > >
> > > > > Thus our dependency tree can grow in all directions. You can also
> > > > rearrange and update it in any shape or form if necessary.
> > > > >
> > > > > Overall, we only use two JIRA elements: epics and tasks (of
> different
> > > > flavors such as bugs, improvements, etc.). Tasks are the leaves,
> > > everything
> > > > else is an epic. Review requests only ever happen for tasks.
> > > > >
> > > > > The epics are there to provide a high level view and to allow
> dynamic
> > > > (“more agilish”, non-waterfall) planning. Granted, you’d also use a
> > tree
> > > if
> > > > you did waterfall. The difference is that you’d spec it all out at
> > once.
> > > My
> > > > observation is that not too few of us do exactly this - outside JIRA
> -
> > > and
> > > > then try to remember what tickets are where in their tree. Let’s make
> > > this
> > > > part of JIRA!
> > > > >
> > > > > Why not use labels? Because they are in a flat name space and we
> want
> > > to
> > > > represent tree structure. How would you know that a label denotes a
> > > > subproject of another label? By memorizing or by depicting a tree
> > outside
> > > > JIRA. Why not use components? Same problem as with labels: flat name
> > > space.
> > > > We can use labels and components these for many other purposes.
> > Separate
> > > > discussion.
> > > > >
> > > > > Aren’t we doing this already? Probably. I have not checked
> > thoroughly.
> > > > There may occasionally be epics that link to other epics. If so, I
> > would
> > > > merely like to encourage us to use this powerful expressive means
> more
> > > > often.
> > > > >
> > > > > Bernd
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Deshi Xiao
> > > Twitter: xds2000
> > > E-mail: xiaods(AT)gmail.com
> > >
> >
>



-- 
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | klaus1982...@gmail.com | http://k82.me

Re: Why do we need slave_id in Kill message

[Klaus Ma]

I think this slave_id is used for status sync up/double check. In 
master, it'll check whether the special slave_id is equal to task's 
slave id; if not equal, master log message and ignore kill request.


On 2015年09月15日 17:46, Qian AZ Zhang wrote:

Hi,

In Kill message (scheduler.proto), I found there is a slave_id field:
   message Kill {
 required TaskID task_id = 1;
 optional SlaveID slave_id = 2;
   }

I am just wondering in which case framework needs to specify this field
when it kills a task, I think master should know the slave id of each task,
can we just use the info in master?


Regards,
Qian Zhang


--
Klaus Ma (马达), PMP® | http://www.cguru.net

Any volunteers to be shepherd of MESOS-2875 (Add containerId to ResourceUsage to enable QoS controller to target a container)?

[Klaus Ma]

Hi team,

I'm working on MESOS-2875, any volunteers to be shepherd of this ticket?
Thanks in advance :).

--
Klaus Ma (马达), PMP® | http://www.cguru.net

RE: Any volunteers to be shepherd of MESOS-2875 (Add containerId to ResourceUsage to enable QoS controller to target a container)?

[Klaus Ma]

np :)


Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
Skype

> Date: Wed, 9 Sep 2015 19:25:10 -0700
> Subject: Re: Any volunteers to be shepherd of MESOS-2875 (Add containerId to 
> ResourceUsage to enable QoS controller to target a container)?
> From: vaibhavkhand...@gmail.com
> To: dev@mesos.apache.org
> 
> Sorry I read wrong .. I cannot be shepherd for the ticket ..
> 
> On Wed, Sep 9, 2015 at 7:13 PM, Vaibhav Khanduja <vaibhavkhand...@gmail.com>
> wrote:
> 
> > I can take this up ...
> >
> > On Wed, Sep 9, 2015 at 7:05 PM, Klaus Ma <kl...@cguru.net> wrote:
> >
> >> Hi team,
> >>
> >> I'm working on MESOS-2875, any volunteers to be shepherd of this ticket?
> >> Thanks in advance :).
> >>
> >> --
> >> Klaus Ma (马达), PMP® | http://www.cguru.net
> >>
> >>
> >
  

Re: How to enable logs when running UT cases

[Klaus Ma]

Thanks all your help, I works :).

Regards,

Klaus Ma (马达), PMP® | Advisory Software Engineer
Platform Symphony & MapReduce Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://www.cguru.net

> On Sep 1, 2015, at 13:57, haosdent <haosd...@gmail.com> wrote:
> 
> How two way to do this:
> If use mesos-tests.sh:
> GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="xxx" --verbose
> 
> If use make check:
> GLOG_v=1 MESOS_VERBOSE=1 make check GTEST_FILTER="xxx"
> 
> On Tue, Sep 1, 2015 at 1:07 PM, Klaus Ma <kl...@cguru.net> wrote:
> 
>> Hi team,
>> 
>> I’m working on MESOS-3070 (Master CHECK failure if a framework uses
>> duplicated task id); the fix was passed in Ubuntu 14.04 but failed at Mac
>> OS, so is there any parameter to enable logs when run UT cases? So I can
>> check what happened in Mac OS.
>> 
>> I export MESOS_VERBOSE, but it seems not work :(.
>> 
>> Regards,
>> 
>> Klaus Ma (马达), PMP®  | http://www.cguru.net
>> 
>> 
> 
> 
> -- 
> Best Regards,
> Haosdent Huang

How to enable logs when running UT cases

[Klaus Ma]

Hi team,

I’m working on MESOS-3070 (Master CHECK failure if a framework uses duplicated 
task id); the fix was passed in Ubuntu 14.04 but failed at Mac OS, so is there 
any parameter to enable logs when run UT cases? So I can check what happened in 
Mac OS.

I export MESOS_VERBOSE, but it seems not work :(.

Regards,

Klaus Ma (马达), PMP®  | http://www.cguru.net

Re: MesosCon Seattle videos now online

[Klaus Ma]

Is there any location to download those videos? And is it OK to 
re-publish those video in other website with Mesos brand?


On 2015年08月28日 00:56, Dave Lester wrote:

Over 50 talks from MesosCon Seattle are now available on YouTube, and organized 
into an official playlist: 
https://www.youtube.com/playlist?list=PLVjgeV_avap2arug3vIz8c6l72rvh9poV

Among the talks, I wanted to highlight our invited keynotes:
* Benjamin Hindman, State of Mesos[1]
* Adrian Cockcroft, Cloud Trends, DevOps and Microservices[2]
* Neha Narula, Trading Simplicity for Performance When Designing
   Distributed Systems[3]
* Peter Bailis, Silence is Golden: Coordination-Avoiding Systems
   Design[4]

Thanks, all! We'l be sending out a survey for conference feedback
shortly. And remember that the CFP for MesosCon Europe[5] (held in
Dublin this October) closes tomorrow (Friday)!

Dave


Links:

   1. 
https://www.youtube.com/watch?v=aV6pdWveN7slist=PLVjgeV_avap2arug3vIz8c6l72rvh9poVindex=54
   2. 
https://www.youtube.com/watch?v=c0wSmr-u5vQindex=40list=PLVjgeV_avap2arug3vIz8c6l72rvh9poV
   3. 
https://www.youtube.com/watch?v=4n1Udu_NJXIindex=42list=PLVjgeV_avap2arug3vIz8c6l72rvh9poV
   4. 
https://www.youtube.com/watch?v=EYJnWttrC9kindex=30list=PLVjgeV_avap2arug3vIz8c6l72rvh9poV
   5. http://events.linuxfoundation.org/events/mesoscon-europe/program/cfp



--
Klaus Ma (马达), PMP® | Advisory Software EngineerPlatform
Symphony  MapReduce Development  Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://www.cguru.net

Re: Lets stop using the CHECK macro in the test harness.

[Klaus Ma]

+1, agree to keep test running.

Regards,

Klaus Ma (马达), PMP®  | http://www.cguru.net

 On Aug 15, 2015, at 14:45, Bernd Mathiske be...@mesosphere.io wrote:
 
 +1, but…
 
 If we are going to touch all our tests, then IMHO while at it we might as 
 well make a jump forward to something better than the current local return 
 void to abort tests in macros. 
 
 If we used exceptions instead, it should be easy to catch those in a wrapper 
 somewhere in the test class and then we could install some 
 test-class-specific or test-superclass-specific or test-specific, but NOT 
 macro-specific (!), extra code that prints out extra diagnostic info iff the 
 test has failed. For example, it can dump the contents of the sandbox.
 
 If you really don’t like exceptions (?), we could make tests return a value 
 instead of void and make the macros indicate that the test failed that way. 
 Then we could also have a failure hook in the wrapper. (This is somewhat 
 inferior, because it still does not support using the macros in nested 
 methods/functions with a different return type. So I prefer exceptions. Not 
 saying I want them everywhere in Mesos. Just in tests! 2c)
 
 I have started to modify one individual AWAIT macro for the above purpose, 
 because I need more info when an unreproducable flaky fetcher cache tests 
 failure happens. I need the extra info dump then and there when it happens, 
 because I cannot get to it it later. The problem with this approach is that 
 it instruments only one macro. Here is what it looks like (ugly!):
 
 #define AWAIT_READY_FOR_WITH_FAILURE_HOOK(actual, duration, onFailureHook) \
  GTEST_PRED_FORMAT2_( \
 AwaitAssertReady, \
 actual, \
 duration, \
 return [=](const char *message) { \
   onFailureHook(); \
   GTEST_FATAL_FAILURE_(message); \
 })
 
 #define AWAIT_READY_WITH_FAILURE_HOOK(actual, onFailureHook) \
  AWAIT_READY_FOR_WITH_FAILURE_HOOK(actual, Seconds(15), onFailureHook)
 
 Then I use it this way:
 
AWAIT_READY_WITH_FAILURE_HOOK(
someFuture,
[=]() { logSandbox(); }); // action on failure by timeout
 
 Before you ask, this approach does not work, i.e. the output on failure does 
 not happen:
 
AWAIT_READY(someFuture
  .onFailed([=]() { logSandbox(); });
 
 ---
 
 Always printing out all the info would IMHO be prohibitively much.
 
 An alternative would be to have two log files, one at a lower log level for 
 success only and one with the highest log level, which buildbot prints only 
 in case something bad happens. etc. We could overwrite the latter after each 
 individual test. This approach would mean extra work on the logging system.
 
 Opinions?
 
 Of course, Paul’s proposal should be tackled in any case. :-)
 
 Bernd
 
 On Aug 15, 2015, at 1:24 AM, Marco Massenzio ma...@mesosphere.io wrote:
 
 +1
 
 *Marco Massenzio*
 
 *Distributed Systems Engineerhttp://codetrips.com http://codetrips.com*
 
 On Fri, Aug 14, 2015 at 3:46 PM, Paul Brett pbr...@twitter.com.invalid
 wrote:
 
 We are currently using the Google log CHECK macros (CHECK_SOME,
 CHECK_NOTNULL etc) in the test harness, usually to verify test setup.  When
 these checks fail, it causes the test harness to abort rather than simply
 move onto the next test. The abort prevents any subsequent tests from
 running, hiding errors and preventing the generation of the XML test
 report.
 
 I would like to propose that we eliminate the use of CHECK in the test
 harness and replace it with the appropriate Google test macros to fail the
 test case.
 ​  I​
 am not proposing that we change the use of CHECK outside the test harness
 (although CHECK calls in master and slave can also kill the test harness).
 
 For void functions, CHECK can
 ​ easily​
 be replaced with the corresponding ASSERT equivalent.
 
 For non-void function, ASSERT cannot be used because it does not return the
 correct data type and hence we need to use a combination of ADD_FAILURE()
 and return.
 
 For example:
 
   CHECK(foo)
 
 would become:
 
   if(!foo) {
   ADD_FAILURE();
   return anything;
   }
 
 If there is general agreement, I will raise tickets to update the Mesos
 testing patterns document and each of the test cases.
 
 ​Thanks
 ​
 
 -- Paul Brett
 
 

Re: Mesos Developer Community Design Discussions

[Klaus Ma]

+1.

Live stream is great :).

Thanks
Klaus

On 08/11/2015 07:42 AM, Khanduja, Vaibhav wrote:

+1

Thx

On 8/10/15, 4:41 PM, Timothy Anderegg timothy.ander...@gmail.com wrote:


Live stream for folks who won't be there would be great, thanks!

Tim

On Mon, Aug 10, 2015 at 7:09 PM, Benjamin Hindman b...@eecs.berkeley.edu
wrote:


The spreadsheet has been updated, sorry about that!

Dave: we didn't want to pull people away from the hackathon and all the
other times were taken, so we figured we'd do the best we can! Enough
people have chatted with me about talking about future features that it
made sense to put something together for anyone in the community to
join.
We'll try and get live streaming set up for folks that aren't flying in
until Wednesday (or at all).

On Mon, Aug 10, 2015 at 3:01 PM Dave Lester d...@davelester.org wrote:


Hi Ben,

It looks like the spreadsheet/google form isn't public outside your
organization, you may need to revise the permissions.

I'm curious, why this particular day? It's a bit unfortunate that

only 8

day notice is being given, since I imagine most people have their
flights for MesosCon already booked and will not likely be in
attendance.

Best,
Dave

On Mon, Aug 10, 2015, at 02:52 PM, Benjamin Hindman wrote:

We've had great developer community meetings over Google Hangouts,

but

MesosCon is right around the corner and we'd like to do it in

person!

I'd like to invite everyone to attend an in person developer

community

meeting on Tuesday, August 18th in Seattle. The focus will be for

Mesos

committers and contributors to get together to discuss big ticket
features to work on during the next 6 to 12 months. I will be

joining

many
other committers at this event and we all look forward to discussing

new

features with everyone!

There is a limited amount of space so please submit your RSVP here:
http://bit.ly/mesosdesignsummit

We look forward to seeing you in Seattle! And don't forget to

register

for
MesosCon
http://events.linuxfoundation.org/events/mesoscon/attend/register

if

you haven't already!

RE: Ask a Shepherd for MESOS-3070 (Master CHECK failure if a framework uses duplicated task id)

[Klaus Ma]

Hi Vinod,

Thanks very much; I'll update the detail of proposal on the ticket.

Regards,

Klaus Ma (马达), PMP® | Advisory Software Engineer
Platform Symphony  MapReduce Development  Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://www.cguru.net


-Original Message-
From: Vinod Kone [mailto:vinodk...@gmail.com] 
Sent: 2015年8月1日 1:18
To: dev
Subject: Re: Ask a Shepherd for MESOS-3070 (Master CHECK failure if a framework 
uses duplicated task id)

On Fri, Jul 31, 2015 at 2:11 AM, Klaus Ma kl...@cguru.net wrote:

 solution for this ticket J.


I'll be happy to shepherd.

Ask a Shepherd for MESOS-3070 (Master CHECK failure if a framework uses duplicated task id)

[Klaus Ma]

Hi team,

 

Is there anyone help to be shepherd of this ticket? I’ve some question 
solution for this ticket J.


Regards,

Klaus Ma (马达), PMP(r) | Advisory Software Engineer
Platform Symphony  MapReduce Development  Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://www.cguru.net

 

RE: Where's Meses design documents

[Klaus Ma]

Wiki seems out of date :). And a central place for links of all documents in 
google doc is fine; it'll also save owner effort on translating from google doc 
to wiki.

Regards,

Klaus Ma (马达), PMP® | http://www.cguru.net

-Original Message-
From: Benjamin Mahler [mailto:benjamin.mah...@gmail.com] 
Sent: 2015年7月28日 5:28
To: dev
Subject: Re: Where's Meses design documents

Vinod, it would be nice to have them within the repo, can't we have an area in 
docs/ for design documents? Even if that just links to active google docs, 
seems nicer than using cwiki?

On Mon, Jul 27, 2015 at 2:05 PM, Vinod Kone vinodk...@gmail.com wrote:

 Ideally, all design docs that are finalized should go to Mesos's cwiki 
 space https://cwiki.apache.org/confluence/display/MESOS/Index.  We 
 have some there already.

 @all: if you've finalized (i.e., corresponding code has been 
 committed) design docs please move them to the cwiki space. Let me 
 know if you need perms.

 On Mon, Jul 27, 2015 at 1:45 PM, Michael Park mcyp...@gmail.com wrote:

  Hi Klaus,
 
  There's no central place where all Mesos design documents live currently.
  They're mostly google docs which are linked from the JIRA epics. 
  Perhaps it's a good idea to have a central place to store the design 
  documents however, so that they can be discovered easier.
 
  MPark.
 
  On Sat, Jul 25, 2015 at 11:43 PM Klaus Ma kl...@cguru.net wrote:
 
   Hi Yong/Alex,
  
   Thanks very much for those info :). I have went through those 
   documents
  to
   learn the overall design of Mesos.
   I think I have to check JIRA for design document of special feature.
  
   Regards,
   
   Klaus Ma (马达), PMP® | http://www.cguru.net
  
On Jul 26, 2015, at 10:07, Yong Feng fengyong...@gmail.com wrote:
   
Hi Klaus
   
I assume you would like to learn how Mesos is designed and
 implemented
feature by feature through design doc of Epic. If that is that 
case,
  you
could start with 
https://github.com/apache/mesos/tree/master/docs
  which
describes Mesos overall architecture, func spec of main 
features,
  release
history, roadmap plan and so on. It actually also cover what 
Alex
  cited.
   
After that, if you have interest on specific topic/feature, you 
could search JIRA for related ticket to check the design idea 
and related discussion.
   
Hope it will help.
   
Thanks,
   
Yong
   
On Sat, Jul 25, 2015 at 9:09 PM, Klaus Ma kl...@cguru.net wrote:
   
I'm asking for the design document of each epic, is there a 
central
   place
to download them?Just go through the design document of 
maintenance,
   it's
helpful; so I'd like to also learn other design documents.
   
Regards,Klaus Ma (马达), PMP® | http://www.cguru.net CallSend 
SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree
  via
SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype
   CreditFree
via Skype
   
Date: Sat, 25 Jul 2015 18:01:06 -0700
Subject: Re: Where's Meses design documents
From: clemmer.alexan...@gmail.com
To: dev@mesos.apache.org
   
Are you asking about the standard documentation (which is 
here[1]),
  or
are you asking about the design documents for every epic that 
has
  been
added historically?
   
[1] http://mesos.apache.org/documentation/latest/
   
On Sat, Jul 25, 2015 at 5:53 PM, Klaus Ma kl...@cguru.net wrote:
Hi team,
I can get design documents from each EPIC/Story, but where 
can I
 get
ALL design document of Mesos?
   
Regards,Klaus Ma (马达), PMP® | http://www.cguru.net 
CallSend SMSCall from mobileAdd to SkypeYou'll need Skype
 CreditFree
via SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need 
Skype CreditFree via Skype
   
   
   
--
Alex
   
Theory is the first term in the Taylor series of practice. --
 Thomas
  M
Cover (1992)
   
   
  
  
 

Re: Where's Meses design documents

[Klaus Ma]

Hi Yong/Alex,

Thanks very much for those info :). I have went through those documents to 
learn the overall design of Mesos.
I think I have to check JIRA for design document of special feature.

Regards,

Klaus Ma (马达), PMP® | http://www.cguru.net 

 On Jul 26, 2015, at 10:07, Yong Feng fengyong...@gmail.com wrote:
 
 Hi Klaus
 
 I assume you would like to learn how Mesos is designed and implemented
 feature by feature through design doc of Epic. If that is that case, you
 could start with https://github.com/apache/mesos/tree/master/docs which
 describes Mesos overall architecture, func spec of main features, release
 history, roadmap plan and so on. It actually also cover what Alex cited.
 
 After that, if you have interest on specific topic/feature, you could
 search JIRA for related ticket to check the design idea and related
 discussion.
 
 Hope it will help.
 
 Thanks,
 
 Yong
 
 On Sat, Jul 25, 2015 at 9:09 PM, Klaus Ma kl...@cguru.net wrote:
 
 I'm asking for the design document of each epic, is there a central place
 to download them?Just go through the design document of maintenance, it's
 helpful; so I'd like to also learn other design documents.
 
 Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
 CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via
 SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree
 via Skype
 
 Date: Sat, 25 Jul 2015 18:01:06 -0700
 Subject: Re: Where's Meses design documents
 From: clemmer.alexan...@gmail.com
 To: dev@mesos.apache.org
 
 Are you asking about the standard documentation (which is here[1]), or
 are you asking about the design documents for every epic that has been
 added historically?
 
 [1] http://mesos.apache.org/documentation/latest/
 
 On Sat, Jul 25, 2015 at 5:53 PM, Klaus Ma kl...@cguru.net wrote:
 Hi team,
 I can get design documents from each EPIC/Story, but where can I get
 ALL design document of Mesos?
 
 Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
 CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree
 via SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype
 CreditFree via Skype
 
 
 
 --
 Alex
 
 Theory is the first term in the Taylor series of practice. -- Thomas M
 Cover (1992)
 
 

Where's Meses design documents

[Klaus Ma]

Hi team,
I can get design documents from each EPIC/Story, but where can I get ALL design 
document of Mesos?

Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
Skype  

RE: Where's Meses design documents

[Klaus Ma]

I'm asking for the design document of each epic, is there a central place to 
download them?Just go through the design document of maintenance, it's helpful; 
so I'd like to also learn other design documents.

Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
Skype

 Date: Sat, 25 Jul 2015 18:01:06 -0700
 Subject: Re: Where's Meses design documents
 From: clemmer.alexan...@gmail.com
 To: dev@mesos.apache.org
 
 Are you asking about the standard documentation (which is here[1]), or
 are you asking about the design documents for every epic that has been
 added historically?
 
 [1] http://mesos.apache.org/documentation/latest/
 
 On Sat, Jul 25, 2015 at 5:53 PM, Klaus Ma kl...@cguru.net wrote:
  Hi team,
  I can get design documents from each EPIC/Story, but where can I get ALL 
  design document of Mesos?
 
  Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
  CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
  SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree 
  via Skype
 
 
 
 -- 
 Alex
 
 Theory is the first term in the Taylor series of practice. -- Thomas M
 Cover (1992)
  

How to build UT only

[Klaus Ma]

Hi team,
Is there anyway to compile the UT code only? When we run make check, it not 
only compile the UT code but also run UT cases. 

Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
Skype  

RE: How to build UT only

[Klaus Ma]

Got that, thanks very much :).

Regards,

Klaus Ma (马达), PMP® | http://www.cguru.net


-Original Message-
From: haosdent [mailto:haosd...@gmail.com] 
Sent: 2015年7月18日 19:39
To: dev@mesos.apache.org
Subject: Re: How to build UT only

```
make check -j8 GTEST_FILTER=-*
```
Would only compile code without run unit tests.

On Sat, Jul 18, 2015 at 6:08 PM, Klaus Ma kl...@cguru.net wrote:

 Hi team,
 Is there anyway to compile the UT code only? When we run make check, 
 it not only compile the UT code but also run UT cases.

 Regards,Klaus Ma (马达), PMP® | http://www.cguru.net CallSend 
 SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
 SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype 
 CreditFree via Skype




--
Best Regards,
Haosdent Huang

Ask shepherd for MESOS-3023

[Klaus Ma]

Hi team,
I’m working on MESOS-3023, a ticket for newbie; is there anyone can be my 
shepherd for this? Thanks in advance :).
Regards,Klaus Ma (马达), PMP® | http://www.cguru.net
CallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
SkypeCallSend SMSCall from mobileAdd to SkypeYou'll need Skype CreditFree via 
Skype  

Ask shepherd for MESOS-3023

[Klaus Ma]

Hi team,

I’m working on MESOS-3023, a ticket for newbie; is there anyone can be my 
shepherd for this? Thanks in advance :).

Regards,

Klaus Ma (马达), PMP® | http://www.cguru.net 

Jekin build steps

[Klaus Ma]

Hi all,

I'm working on MESOS-1416 (mesos-0.19.0 build directory is read-only), so i'd 
like to know the steps to reproduce it. Can anyone help me?

Regards,Da Ma (马达), PMP® | CEL3 Team LeadPlatform Symphony  MapReduce 
Development  Support, STG, IBM GCG+86-10-8245 4084 | mad...@cn.ibm.com | 
http://www.cguru.net