Re: [BULK]Re: New PMC Chair

[Sam Chen]

Congrats ! Well deserved!


On Fri, 30 Apr 2021 at 01:49 Grégoire Seux  wrote:

> Congratulations!
>
> -- ​
> Grégoire
>
> --
> *From:* Andreas Peters
> *Sent:* Thursday, April 29, 2021 6:36 PM
> *To:* d...@mesos.apache.org; Vinod Kone; user
> *Subject:* [BULK]Re: New PMC Chair
>
> Great to hear. :-)
>
> Am 29.04.21 um 16:35 schrieb Vinod Kone:
> > Hi community,
> >
> > Just wanted to let you all know that the board passed the resolution to
> > elect a new PMC chair!
> >
> > Hearty congratulations to *Qian Zhang* for becoming the new Apache Mesos
> > PMC chair and VP of the project.
> >
> > Thanks,
> >
>
>

Re: Welcome Chun-Hung Hsiao as Mesos Committer and PMC Member

[Sam]

Congrats Chun


Regards,
Sam Chen | APJ Country Director | DC/OS Evangelist
>   Build and run modern apps
> at scale using DC/OS


> On Mar 11, 2018, at 1:14 PM, Jie Yu <yujie@gmail.com> wrote:
> 
> Chun

Re: [VOTE] Release Apache Mesos 1.5.0 (rc1)

[Sam]

+1


Regards,
>   


> On Jan 23, 2018, at 11:15 AM, Gilbert Song  wrote:
> 
> Hi all,
> 
> Please vote on releasing the following candidate as Apache Mesos 1.5.0.
> 
> 1.5.0 includes the following:
> 
>   * Support Container Storage Interface (CSI).
>   * Agent reconfiguration policy.
>   * Auto GC docker images in Mesos Containerizer.
>   * Standalone containers.
>   * Support gRPC client.
>   * Non-leading VOTING replica catch-up.
> 
> The CHANGELOG for the release is available at:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.5.0-rc1
> 
> 
> The candidate for Mesos 1.5.0 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/mesos-1.5.0.tar.gz
> 
> The tag to be voted on is 1.5.0-rc1:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.5.0-rc1
> 
> The MD5 checksum of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/mesos-1.5.0.tar.gz.md5
> 
> The signature of the tarball can be found at:
> https://dist.apache.org/repos/dist/dev/mesos/1.5.0-rc1/mesos-1.5.0.tar.gz.asc
> 
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
> 
> The JAR is in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1221
> 
> Please vote on releasing this package as Apache Mesos 1.5.0!
> 
> The vote is open until Thu Jan 25 18:24:36 PST 2018 and passes if a majority 
> of at least 3 +1 PMC votes are cast.
> 
> [ ] +1 Release this package as Apache Mesos 1.5.0
> [ ] -1 Do not release this package because ...
> 
> Thanks,
> Jie and Gilbert

Re: Welcome James Peach as a new committer and PMC memeber!

[Sam]

Congrats, way to go 


Regards,
Sam Chen | APJ Country Director | DC/OS Evangelist
>   Build and run modern apps
> at scale using DC/OS


> On Sep 7, 2017, at 5:08 AM, Yan Xu <xuj...@apple.com> wrote:
> 
> Hi Mesos devs and users,
> 
> Please welcome James Peach as a new Apache Mesos committer and PMC member.
> 
> James has been an active contributor to Mesos for over two years now. He has 
> made many great contributions to the project which include XFS disk isolator, 
> improvement to Linux capabilities support and IPC namespace isolator. He's 
> super active on the mailing lists and slack channels, always eager to help 
> folks in the community and he has been helping with a lot of Mesos reviews as 
> well.
> 
> Here is his formal committer candidate checklist:
> 
> https://docs.google.com/document/d/19G5zSxhrRBdS6GXn9KjCznjX3cp0mUbck6Jy1Hgn3RY/edit?usp=sharing
>  
> 
> Congrats James!
> 
> Yan
> 

Re: Welcome Greg Mann as a new committer and PMC member!

[Sam]

Congrats, well deserved 


Regards,
Sam Chen | APJ Country Director | DC/OS Evangelist
>   Build and run modern apps
> at scale using DC/OS


> On Jun 14, 2017, at 5:44 AM, Neil Conway <neil.con...@gmail.com> wrote:
> 
> Congratulations, Greg!! Very well-deserved. Looking forward to
> continuing to work with you on the project.
> 
> Neil
> 
> 
>> On Tue, Jun 13, 2017 at 2:42 PM, Vinod Kone <vinodk...@apache.org> wrote:
>> Hi folks,
>> 
>> Please welcome Greg Mann as the newest committer and PMC member of the
>> Apache Mesos project.
>> 
>> Greg has been an active contributor to the Mesos project for close to 2
>> years now and has made many solid contributions. His biggest source code
>> contribution to the project has been around adding authentication support
>> for default executor. This was a major new feature that involved quite a few
>> moving parts. Additionally, he also worked on improving the scheduler and
>> executor APIs.
>> 
>> Here is his more formal checklist for your perusal.
>> 
>> https://docs.google.com/document/d/1S6U5OFVrl7ySmpJsfD4fJ3_R8JYRRc5spV0yKrpsGBw/edit
>> 
>> Thanks,
>> Vinod
>> 

Re: Welcome Guangya Liu as Mesos Committer and PMC member!

[Sam]

congratulations Guangya

Sent from my iPhone

> On 17 Dec 2016, at 3:23 AM, Avinash Sridharan  wrote:
> 
> Congrats Guangya !! 
> 
>> On Fri, Dec 16, 2016 at 11:20 AM, Greg Mann  wrote:
>> Congratulations Guangya!!! :D
>> 
>>> On Fri, Dec 16, 2016 at 11:10 AM, Jie Yu  wrote:
>>> Hi folks,
>>> 
>>> Please join me in formally welcoming Guangya Liu as Mesos Committer and PMC
>>> member.
>>> 
>>> Guangya has worked on the project for more than a year now and has been a
>>> very active contributor to the project. I think one of the most important
>>> contribution he has for the community is that he helped grow the Mesos
>>> community in China. He initiated the Xian-Mesos-User-Group and successfully
>>> organized two meetups which attracted more than 100 people from Xi’an
>>> China. He wrote a handful of blogs and articles in Chinese tech media which
>>> attracted a lot of interests in Mesos. He had given several talks about
>>> Mesos at conferences in China.
>>> 
>>> His major coding contribution to the project was the docker volume driver
>>> isolator. He has also been involved in allocator performance improvement,
>>> gpu support for docker containerizer, Mesos Tiers/Optimistic Offer design,
>>> scarce resources discussion, and many others.
>>> 
>>> His formal checklist is here:
>>> https://docs.google.com/document/d/1tot79kyJCTTgJHBhzStFKrVkDK4pX
>>> qfl-LHCLOovNtI/edit?usp=sharing
>>> 
>>> Thanks,
>>> - Jie
>> 
> 
> 
> 
> -- 
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245

Re: Welcome Qian Zhang as a new committer!

[Sam]

Congratulation, Qian.

Sent from my iPhone

> On 9 Oct 2016, at 2:54 AM, Avinash Sridharan  wrote:
> 
> Congrats Qian !! Great work . Well deserved !!
> 
>> On Sat, Oct 8, 2016 at 11:47 AM, Vinod Kone  wrote:
>> Congrats Qian. Looking forward to more contributions!
>> 
>> @vinodkone
>> 
>>> On Oct 8, 2016, at 11:35 AM, haosdent  wrote:
>>> 
>>> Congrats to Qian!!! Looking forward work more with you in the community!  
>>> 
 On Sun, Oct 9, 2016 at 2:20 AM, Jie Yu  wrote:
 Hi folks,
 
 I' happy to announce that the PMC has voted Qian Zhang as a new committer 
 and member of PMC for the Apache Mesos project. Please join me to 
 congratulate him!
 
 A little more about Qian Zhang:
 
 Qian Zhang has been working on the Apache Mesos project for about an year 
 now. He designed and implemented the CNI (Container Network Interface) 
 support in Mesos with Avinash, which standardized the networking 
 integration in Mesos. He also worked with haosdent on the unified cgroups 
 isolator, which greatly simplifies the original cgroups support in Mesos 
 and makes extension to new subsystems so much easier. He was also involved 
 in discussions on quotas and pods, and provided valuable feedback. He is 
 currently working on OCI support in Mesos, trying to enable Mesos to 
 launch OCI containers.
 
 More details can be found in his committer candidate checklist.
 
 Qian, thank you for your great work to the project so far. Would love to 
 see more!
 
 - Jie
>>> 
>>> 
>>> 
>>> -- 
>>> Best Regards,
>>> Haosdent Huang
> 
> 
> 
> -- 
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245

Re: Can Mesos support same multi-frameworks in same cluster?

[Sam]

Hi Brenden,
Very helpful. Appreciated.

Regards,
Sam

Sent from my iPhone

> On Jul 24, 2016, at 1:29 AM, Brenden Matthews <bren...@diddyinc.com> wrote:
> 
> You can pretty much run as many frameworks as you'd like.
> 
> One trick you can use to drive up CPU use (if you want) is to overcommit 
> CPUs. For example, if you have a machine with 16 CPUs, set the number of CPUs 
> to 32 on the Mesos agent, and you'll be overcommitted by 100% (i.e., 1 CPU 
> share will practically be equivalent to 50% of 1 CPU).
> 
> Furthermore, you will want to ensure that allocated resources match used 
> resources, which may help drive up utilization if that's your goal (aka 
> 'right sizing').
> 
>> On Thu, Jul 21, 2016 at 9:35 PM, Sam <usultra...@gmail.com> wrote:
>> Hi Guys,
>> We met one issue in maximizing the CPU and Mem resources utilization in our 
>> servers. For example, We got 100 CPUs and 100G Mem , and we installed Apache 
>> mesos and Kafka \ Hbase frameworks. In our production Env, we got heavy 
>> workload from client in Hbase/Kafka, however our resource utilization 
>> percentage is always under 50%. it's so surprised that Mesos cannot be able 
>> to offer the full resources to frameworks.
>> So we are thinking of deploy same multi-Frameworks to fix this issue, for 
>> Instance, 5 Kafka frameworks to allocate at least 20% of resources, that's 
>> 20 CPUs and 20MEMs each. Which me mans 5 Kafkas will consume 100CPUs and 
>> 100MEMs logically. Let's say using role, quota or reservation of this is the 
>> case.
>> 
>> Our questions are :
>> 1, Is this logic correct? Or ridiculous?
>> 2, if this is positive, how to support and customize frameworks or others to 
>> support this? Which means I can order 5 times in same cluster.
>> 
>> Looking forward to have any suggestions that you got.
>> Appreciated
>> 
>> Regards,
>> Sam
>> 
>> Sent from my iPhone
> 

Can Mesos support same multi-frameworks in same cluster?

[Sam]

Hi Guys,
We met one issue in maximizing the CPU and Mem resources utilization in our 
servers. For example, We got 100 CPUs and 100G Mem , and we installed Apache 
mesos and Kafka \ Hbase frameworks. In our production Env, we got heavy 
workload from client in Hbase/Kafka, however our resource utilization 
percentage is always under 50%. it's so surprised that Mesos cannot be able to 
offer the full resources to frameworks. 
So we are thinking of deploy same multi-Frameworks to fix this issue, for 
Instance, 5 Kafka frameworks to allocate at least 20% of resources, that's 20 
CPUs and 20MEMs each. Which me mans 5 Kafkas will consume 100CPUs and 100MEMs 
logically. Let's say using role, quota or reservation of this is the case.

Our questions are :
1, Is this logic correct? Or ridiculous? 
2, if this is positive, how to support and customize frameworks or others to 
support this? Which means I can order 5 times in same cluster.

Looking forward to have any suggestions that you got.
Appreciated 

Regards,
Sam

Sent from my iPhone

Re: Welcome Anand and Joseph as new committers!

[Sam]

Congratulations Anand & Joseph. 
Looking forward to having chance to talk with you guys. 

Sent from my iPhone

> On Jun 11, 2016, at 5:36 PM, haosdent  wrote:
> 
> Congratulations Anand and Joseph! 
> I always learn a lot form Joseph's nice analyzations under Mesos flaky test 
> tickets, thanks a lot!
> 
>> On Sat, Jun 11, 2016 at 8:15 AM, Brenden Matthews  
>> wrote:
>> Welcome and congrats!
>> 
>>> On Fri, Jun 10, 2016 at 2:11 PM, Benjamin Mahler  wrote:
>>> Welcome Anand and Joseph, thanks for all of your contributions!
>>> 
>>> Looking forward to seeing your ongoing positive influences on the community 
>>> and the project, let's build great software!
>>> 
 On Thu, Jun 9, 2016 at 2:00 PM, Vinod Kone  wrote:
>>> 
 Hi folks,
 
 I'm happy to announce that the PMC has voted in *Anand Mazumdar *and 
 *Joseph
 Wu* as committers and members of PMC for the Apache Mesos project.
 
 A little about the new committers.
 
 Anand has been working on the Apache Mesos project for about an year now
 and has shown incredible commitment to the project and the community. His
 significant contributions include implementing scheduler HTTP API,
 designing and implementing executor HTTP API and helping out with the
 operator HTTP API. His formal committer checklist is here
 
 .
 
 Joseph's passion and dedication to the community is phenomenal. His
 significant contributions include Maintenance Primitives and Container
 Logger Modules. He has also been a valuable contributor and reviewer to our
 testing infrastructure and the Windows work. His formal committer checklist
 is here
 
 .
 
 Please join me in congratulating them on their new roles and especially
 responsibilities :)
 
 On behalf of the PMC,
 Vinod
> 
> 
> 
> -- 
> Best Regards,
> Haosdent Huang

Re: Deploying MySQL and WordPress Docker Containers through Marathon

[Sam]

Guys,
Here is beauty version https://dcos.io/docs/1.7/overview/roadmap/ 

Regards,
Sam

Sent from my iPhone

> On May 13, 2016, at 3:35 PM, Guangya Liu <gyliu...@gmail.com> wrote:
> 
> Just FYI, the dc/os does have near term roadmap to enable support for 
> "external volume storage": 
> https://github.com/dcos/dcos-docs/blob/master/1.7/overview/roadmap.md#pluggable-external-volumes
>  , once this was enabled in open source dc/os, you can use this feature to 
> manage your stateful services.
> 
> The JIRA for the external storage support is here 
> https://issues.apache.org/jira/browse/MESOS-4355
> 
> Thanks,
> 
> Guangya
> 
>> On Fri, May 13, 2016 at 2:16 AM, Wong, Steven <steven.w...@emc.com> wrote:
>> There was a presentation given on this last week at the EMC World 
>> conference, demonstrating migration of a MySQL server across cluster nodes, 
>> with persistent storage on an external volume. For this use case, a cloud 
>> volume (such as AWS EBS), or a networked attached storage volume (such as 
>> ScaleIO) would be used to hold the database.
>> 
>> slide deck: 
>> http://www.slideshare.net/EMCCODE/emc-world-2016-code04-extending-mesos-for-storage-and-external-resources
>> 
>>  
>> 
>> Video of demo: 
>> https://www.youtube.com/watch?v=DL64mdYv5Lg=PLbssOJyyvHuWiBQAg9EFWH570timj2fxt=1
>> 
>>  
>> 
>> Note that the demo is using a very recent version of the DC/OS Marathon UI 
>> 
>> Steve Wong
>> 
>> Developer Advocate
>> 
>> EMC{code} – the open source advocacy group within EMC
>> 
>> 
>> 
>> CODE OPEN, DEPLOY EVERYWHERE
>> 
>> https://github.com/emccode
>> 
>> blog.emccode.com
>> 
>>  
>> 
>> @cantbewong
>> 
>>  
>> 
>> From: suruchi.kum...@accenture.com [mailto:suruchi.kum...@accenture.com] 
>> Sent: Thursday, May 12, 2016 2:59 AM
>> To: user@mesos.apache.org
>> Subject: Deploying MySQL and WordPress Docker Containers through Marathon
>> 
>>  
>> 
>>  
>> 
>> Hi ,
>> 
>>  
>> 
>> In case we are killing one of the instances running through Marathon UI. It 
>> is able to bring up another replacing it.
>> 
>>  
>> 
>> So, would like to know is it a replication of the previous instance which 
>> was killed or it’s a new one.
>> 
>>  
>> 
>> For example, if we are running a MySQL application through Marathon UI  and 
>> try on of its instance to kill. Will the information saved in the database 
>> will be remain in the new one replacing it.
>> 
>>  
>> 
>>  
>> 
>> Thanks
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>>  
>> 
>> 
>> This message is for the designated recipient only and may contain 
>> privileged, proprietary, or otherwise confidential information. If you have 
>> received it in error, please notify the sender immediately and delete the 
>> original. Any other use of the e-mail by you is prohibited. Where allowed by 
>> local law, electronic communications with Accenture and its affiliates, 
>> including e-mail and instant messaging (including content), may be scanned 
>> by our systems for the purposes of information security and assessment of 
>> internal compliance with Accenture policy. 
>> __
>> 
>> www.accenture.com
>> 
> 

Re: Running Mesos agent on ARM (Raspberry Pi)?

[Sam]

Guys, 
I don't understand why deploying Mesos master and slave on Raspberrypi right 
now. Most of scenarios is using raspberry pi as edge server of IoT since 
Raspberrypi low configuration and performance .
Regards of the possibility of deployment, we have experimented before , and it 
works. You have to install Debian on Raspberry pi first , then deploying Mesos 
Master and Slave as docker images. The performance is too low. 
Hope to see what's you guys scenarios .


Regards,
Sam

Sent from my iPhone

> On Apr 25, 2016, at 4:10 PM, tommy xiao <xia...@gmail.com> wrote:
> 
> let it go. it give us alternative solution.
> 
> 2016-04-25 16:05 GMT+08:00 Aaron Carey <aca...@ilm.com>:
>> Out of curiosity... is this for fun or production workloads? I'd be curious 
>> to hear about raspis being used in production!
>> 
>>  --
>> 
>> Aaron Carey
>> Production Engineer - Cloud Pipeline
>> Industrial Light & Magic
>> London
>> 020 3751 9150
>> From: Sharma Podila [spod...@netflix.com]
>> Sent: 22 April 2016 17:53
>> To: user@mesos.apache.org; dev
>> Subject: Running Mesos agent on ARM (Raspberry Pi)?
>> 
>> We are working on a hack to run Mesos agents on Raspberry Pi and are 
>> wondering if anyone here has done that before. From the Google search 
>> results we looked at so far, it seems like it has been compiled, but we 
>> haven't seen an indication that anyone has run it and launched tasks on 
>> them. And does it sound right that it might take 4 hours or so to compile?
>> 
>> We are looking to run just the agents. The master will be on a regular 
>> Ubuntu laptop or a server. 
>> 
>> Appreciate any pointers.
> 
> 
> 
> -- 
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com

Re: Mesos Community Sync 4/7/16 - Notes

[Sam Chen]

Hi Grep,
I am located in China. I would love to participate in next meeting. Could
you please share me how to join in? appreciated.

Regards,
Sam

On Wed, Apr 13, 2016 at 5:07 AM, Greg Mann <g...@mesosphere.io> wrote:

> Hey Mesos community,
> Find below the notes from our last Mesos Community Sync meeting, which
> occurred on 4/7/2016 at 9am PST.
>
> The next community sync will be on 4/21/2016 from 3-4pm PST. See the Mesos
> community calendar <http://mesos.apache.org/community/> for upcoming
> meetings (note that you can add that Google calendar to your own calendar
> to sync future events automatically).
>
> Cheers,
> Greg
>
>
> April 7, 2016
>
> Time: 9am PST
>
> Location: Mesosphere HQ
>
> Attendees:
>
>-
>
>Mesosphere: Jan, Alexander, Bernd, Kevin, Greg, Gilbert, BenjaminB,
>AlexR, MPark, Jörg
>
>
> Agenda/Notes:
>
>-
>
>Ship date Mesos 0.29?
>-
>
>   Mid-May
>   -
>
>   Will 0.29 be the Mesos 1.0 release? Possibly, depending on
>   availability of features that we’ve committed to complete for 1.0
>   -
>
>Module API changes
>-
>
>   Users have raised concerns about the frequency/extent of changes to
>   the Mesos modules API
>   -
>
>   We can help users by calling out these changes prominently in the
>   Changelog
>   -
>
>   Could use protobuf function parameters to avoid breaking interfaces
>   -
>
>   Could include a separate Changelog section for breaking changes
>   -
>
>Email out notice to dev/user mailing list one day before Mesos
>Community Sync meeting.
>-
>
>New features should be listed on Mesos release planning
><https://cwiki.apache.org/confluence/display/MESOS/Mesos+Release+Planning>
>wiki
>-
>
>
>
> https://mesosphere.com/blog/2015/11/11/contributing-to-mesos-where-to-begin/
>
>
> Action Items:
>
>-
>
>[Vinod] Produce an initial roadmap document to review at community
>syncs
>-
>
>[Greg] Send meeting notes to mailing lists
>-
>
>[Jie] Update wiki page
><https://cwiki.apache.org/confluence/display/MESOS/Mesos+Release+Planning>
>which shows major features in each release
>-
>
>[mpark] Set up automatic reminder emails for the community sync
>
>

Issues on Zk configuration in Marathon

[Sam]

Hello guys 
One quick question in Marathon with Mesos,
We are trying to deploy Zk with Marathon to make sure that Zk is always 
available no matter one of nodes crashed. For example : we got Zk1,Zk2 and Zk3, 
Zk1 need to have IP address of Zk2 and Zk3; Zk2 need to have IP address of Zk1 
and Zk3 , same to Zk3 .  The issue is when one of them crashed , and Marathon 
spin up new Zk, how to have old  IP address configuration set into new instance 
? I think this is issue to all App cluster that need to have each other 
configuration respectively. 
Looking forward to having solution to get it done . Appreciate 

Regards,
Sam

Sent from my iPhone

Re: Anyone try Weave in Mesos env ?

[Sam Chen]

Paul,
Appreciated

Regards,
Sam

On Thu, Nov 26, 2015 at 11:09 AM, Paul Bell <arach...@gmail.com> wrote:

> HmmI'm not sure there's really a "fix" for that (BTW: I assume you
> mean to fix high (or long) latency, i.e., to make it lower, faster). A
> network link is a network link, right? Like all hardware, it has its own
> physical characteristics which determine its latency's lower bound, below
> which it is physically impossible to go.
>
> Sounds to me as if you've got the whole Mesos + Docker + Weave thing
> figured out, at least as far as the basic connectivity and addressing is
> concerned. So there's not much more that I can tell you in that regard.
>
> Are you running Weave 1.2 (or above)? It incorporates their "fast path"
> technology based on the Linux kernel's Open vSwitch (*vide*:
> http://blog.weave.works/2015/11/13/weave-docker-networking-performance-fast-data-path/).
> But, remember, there's still the link in between endpoints. One can
> optimize the packet handling within an endpoint, but this could boil down
> to a case of "hurry up and wait".
>
> I would urge you to take this question up with the friendly,
> knowledgeable, and very helpful folks at Weave:
> https://groups.google.com/a/weave.works/forum/#!forum/weave-users .
>
> Cordially,
>
> Paul
>
> On Wed, Nov 25, 2015 at 9:31 PM, Sam <usultra...@gmail.com> wrote:
>
>> Paul,
>> Yup, Weave and Docker.  May I know how did you fix low latency issue over
>> Internet ? By tunnel or ?
>>
>> Regards,
>> Sam
>>
>> Sent from my iPhone
>>
>> > On Nov 26, 2015, at 10:23 AM, Paul <arach...@gmail.com> wrote:
>> >
>> > Happy Thanksgiving to you, too.
>> >
>> > I tend to deploy the several Mesos nodes as VMware VMs.
>> >
>> > However, I've also run a cluster with master on ESXi, slaves on ESXi,
>> slave on bare metal, and an EC2 slave.
>> >
>> > But in my case all applications are Docker containers connected via
>> Weave.
>> >
>> > Does your present deployment involve Docker and Weave?
>> >
>> > -paul
>> >
>> >> On Nov 25, 2015, at 8:55 PM, Sam <usultra...@gmail.com> wrote:
>> >>
>> >> Paul,
>> >> Happy thanksgiving first. We are using Aws, Rackspace as hybrid cloud
>> env , and we deployed Mesos master in AWS , part of Slaves in AWS , part of
>> Slaves in Rackspace .  I am thinking whether it works ? And since it got
>> low latency in networking , can we deploy two masters in both AWS and
>> Rackspace ? And federation ?Appreciated for your reply .
>> >>
>> >> Regards ,
>> >> Sam
>> >>
>> >> Sent from my iPhone
>> >>
>> >>> On Nov 26, 2015, at 9:47 AM, Paul <arach...@gmail.com> wrote:
>> >>>
>> >>> Hi Sam,
>> >>>
>> >>> Yeah, I have significant experience in this regard.
>> >>>
>> >>> We run a Docker containers spread across several Mesos slave nodes.
>> The containers are all connected via Weave. It works very well.
>> >>>
>> >>> Can you describe what you have in mind?
>> >>>
>> >>> Cordially,
>> >>>
>> >>> Paul
>> >>>
>> >>>> On Nov 25, 2015, at 8:03 PM, Sam <usultra...@gmail.com> wrote:
>> >>>>
>> >>>> Guys,
>> >>>> We are trying to use Weave in hybrid cloud Mesos env , anyone got
>> experience on it ? Appreciated
>> >>>> Regards,
>> >>>> Sam
>> >>>>
>> >>>> Sent from my iPhone
>>
>
>

Anyone try Weave in Mesos env ?

[Sam]

Guys,
We are trying to use Weave in hybrid cloud Mesos env , anyone got experience on 
it ? Appreciated 
 Regards,
Sam

Sent from my iPhone

Re: Anyone try Weave in Mesos env ?

[Sam]

Paul,
Happy thanksgiving first. We are using Aws, Rackspace as hybrid cloud env , and 
we deployed Mesos master in AWS , part of Slaves in AWS , part of Slaves in 
Rackspace .  I am thinking whether it works ? And since it got low latency in 
networking , can we deploy two masters in both AWS and Rackspace ? And 
federation ?Appreciated for your reply .

Regards ,
Sam

Sent from my iPhone

> On Nov 26, 2015, at 9:47 AM, Paul <arach...@gmail.com> wrote:
> 
> Hi Sam, 
> 
> Yeah, I have significant experience in this regard.
> 
> We run a Docker containers spread across several Mesos slave nodes. The 
> containers are all connected via Weave. It works very well.
> 
> Can you describe what you have in mind?
> 
> Cordially,
> 
> Paul
> 
>> On Nov 25, 2015, at 8:03 PM, Sam <usultra...@gmail.com> wrote:
>> 
>> Guys,
>> We are trying to use Weave in hybrid cloud Mesos env , anyone got experience 
>> on it ? Appreciated 
>> Regards,
>> Sam
>> 
>> Sent from my iPhone

Re: Anyone try Weave in Mesos env ?

[Sam]

Paul,
Yup, Weave and Docker.  May I know how did you fix low latency issue over 
Internet ? By tunnel or ?

Regards,
Sam 

Sent from my iPhone

> On Nov 26, 2015, at 10:23 AM, Paul <arach...@gmail.com> wrote:
> 
> Happy Thanksgiving to you, too.
> 
> I tend to deploy the several Mesos nodes as VMware VMs.
> 
> However, I've also run a cluster with master on ESXi, slaves on ESXi, slave 
> on bare metal, and an EC2 slave.
> 
> But in my case all applications are Docker containers connected via Weave.
> 
> Does your present deployment involve Docker and Weave? 
> 
> -paul
> 
>> On Nov 25, 2015, at 8:55 PM, Sam <usultra...@gmail.com> wrote:
>> 
>> Paul,
>> Happy thanksgiving first. We are using Aws, Rackspace as hybrid cloud env , 
>> and we deployed Mesos master in AWS , part of Slaves in AWS , part of Slaves 
>> in Rackspace .  I am thinking whether it works ? And since it got low 
>> latency in networking , can we deploy two masters in both AWS and Rackspace 
>> ? And federation ?Appreciated for your reply .
>> 
>> Regards ,
>> Sam
>> 
>> Sent from my iPhone
>> 
>>> On Nov 26, 2015, at 9:47 AM, Paul <arach...@gmail.com> wrote:
>>> 
>>> Hi Sam, 
>>> 
>>> Yeah, I have significant experience in this regard.
>>> 
>>> We run a Docker containers spread across several Mesos slave nodes. The 
>>> containers are all connected via Weave. It works very well.
>>> 
>>> Can you describe what you have in mind?
>>> 
>>> Cordially,
>>> 
>>> Paul
>>> 
>>>> On Nov 25, 2015, at 8:03 PM, Sam <usultra...@gmail.com> wrote:
>>>> 
>>>> Guys,
>>>> We are trying to use Weave in hybrid cloud Mesos env , anyone got 
>>>> experience on it ? Appreciated 
>>>> Regards,
>>>> Sam
>>>> 
>>>> Sent from my iPhone

Re: Mesos + Spark Integration

[Sam Bessalah]

Have you tried to give spark the namenode explicitely
hdfs://namenode_ip:8020/hdfs/  as address?

On Mon, Sep 14, 2015 at 11:09 PM, Rodrick Brown 
wrote:

> I have separate systems for the following services
>
> —  Mesos (3 masters + 3 slaves)
> — Hadoop (2 NN  + 8 slaves)
> — ZooKeeper (3 nodes)
>
> I would like to run my spark jobs on Mesos currently its being executed by
> Yarn via spark-submit.
>
>
> I’ve copied spark to each of the slaves however I’m running into issues
> with my spark job is trying to connect to hadoop via localhost and not via
> hfs:// url
> Is there a good intro guide on gluing everything together I found the
> initial documentation on apache.org very lacking when it comes to
> integratiing spark & Mesos.
>
> Thanks.
>
>
> --
>
> [image: Orchard Platform] 
>
> Rodrick Brown / DevOPs Engineer
> +1 917 445 6839 / rodr...@orchardplatform.com
> 
>
> Orchard Platform
> 101 5th Avenue, 4th Floor, New York, NY 10003
> http://www.orchardplatform.com
>
> Orchard Blog  | Marketplace Lending
> Meetup 
>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient of
> this communication, please delete it immediately and notify the sender by
> return email. Unauthorized reading, dissemination, distribution or copying
> of this communication is prohibited. This communication does not constitute
> an offer to sell or a solicitation of an indication of interest to purchase
> any loan, security or any other financial product or instrument, nor is it
> an offer to sell or a solicitation of an indication of interest to purchase
> any products or services to any persons who are prohibited from receiving
> such information under applicable law. The contents of this communication
> may not be accurate or complete and are subject to change without notice.
> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or completeness
> of the information contained herein. The intended recipient is advised to
> consult its own professional advisors, including those specializing in
> legal, tax and accounting matters. Orchard does not provide legal, tax or
> accounting advice.
>

Re: Metering for Mesos

[Sam Chen]

Haosdent ,
Let me bring one example on the table . We are using Mesos and Marathon ,
and deployed two tier application (web tier is Tomcat , database layer is
mysql) .
We are frustrated in terms of how to charge this service, so that we are
thinking whether mesos or marathon can have metering service to let us
reference .
Hope its clear :)

Sam

On Fri, Aug 7, 2015 at 12:33 PM, haosdent haosd...@gmail.com wrote:

 You mean metering by resource? You could get every task resource usage
 through send http request to state.json .

 On Fri, Aug 7, 2015 at 12:23 PM, Sam Chen usultra...@gmail.com wrote:

 Guys ,
 We are planning to use Mesos as production  platform and based on
 Openstack , My question is , Is there any solution for metering ?  then
 billing .  Since we want to have our platform online and have
  pay-as-you-go mode . Anyone have any suggsetion ?  Very appreciated .


 Sam




 --
 Best Regards,
 Haosdent Huang

Metering for Mesos

[Sam Chen]

Guys ,
We are planning to use Mesos as production  platform and based on Openstack
, My question is , Is there any solution for metering ?  then billing .
Since we want to have our platform online and have  pay-as-you-go mode .
Anyone have any suggsetion ?  Very appreciated .


Sam

Hello Guys

[Sam Chen]

Guys,
We (Linkernetworks) http://www.linkernetworks.com/  is using Mesos now and
our solution and product is based on Mesos .  I  sent serveral emails to
user@mesos.apache.org . however there are nobody reply . Could you guys
help us add into
http://mesos.apache.org/documentation/latest/powered-by-mesos/ .
Appreciated .

Sam

Re: Hello Guys

[Sam Chen]

@haosdent
Appreciated . just done
Add Linkernetworks into powered-by-mesos.md #54

On Fri, Aug 7, 2015 at 11:25 AM, haosdent haosd...@gmail.com wrote:

 Hi, @Sam. You could submit a pull request like this
 https://github.com/apache/mesos/pull/43/files in Github. I think your
 request would be accepted more quickly.

 On Fri, Aug 7, 2015 at 11:17 AM, Sam Chen usultra...@gmail.com wrote:

 Guys,
 We (Linkernetworks) http://www.linkernetworks.com/  is using Mesos now
 and our solution and product is based on Mesos .  I  sent serveral emails
 to user@mesos.apache.org . however there are nobody reply . Could you
 guys help us add into
 http://mesos.apache.org/documentation/latest/powered-by-mesos/ .
 Appreciated .

 Sam




 --
 Best Regards,
 Haosdent Huang

Re: 答复: [DISCUSS] Renaming Mesos Slave

[Sam Salisbury]

Master/Minion +1

On 5 June 2015 at 15:14, CCAAT cc...@tampabay.rr.com wrote:


 +1 master/slave, no change needed.  is the same as
 master/slaveI.E. keep the nomenclature as it currently is

 This means keep the name 'master' and keep the name 'slave'.


 Are you applying fuzzy math or kalman filters to your summations below?

 It looks to me, tallying things up, Master is kept as it is
 and 'Slave' is kept as it is. There did not seem to be any consensus
 on the new names if the pair names are updated. Or you can vote separately
 on each name? On an  real ballot, you enter the choices,
 vote according to your needs, tally the results and publish them.
 Applying a 'fuzzy filter' to what has occurred in this debate so far
 is ridiculous.

 Why not repost the question like this or something on a more fair
 voting preference:

 
 Please vote for your favourite Name-pair in Mesos, for what is currently
 Master-Slave. Note Master-Slave is the no change vote option.

 [] Master-Slave
 [] Mesos-Slave
 [] Mesos-Minion
 [] Master-Minion
 [] Master-Follower
 [] Mesos-Follower
 [] Master-worker
 [] Mesos-worker
 [] etc etc

 -


 Tally the result and go from there.
 James




 On 06/05/2015 04:27 AM, Adam Bordelon wrote:

 Wow, what a response! Allow me to attempt to summarize the sentiment so
 far.

 Let's start with the implicit question,
 _0. Should we rename Mesos Slave?_
 +1 (Explicit approval) 12, including 7 from JIRA
 +0.5 (Implicit approval, suggested alternate name) 18
 -0.5 (Some disapproval, wouldn't block it) 5, including 1 from JIRA
 -1 (Strong disapproval) 16

 _1. What should we call the Mesos Slave node/host/machine?_
 Worker: +10, -2
 Agent: +6
 Follower (+Leader): +4, -1
 Minion: +2, -1
 Drone (+Director/Queen): +2
 Resource-Agent/Provider: +2

 _2. What should we call the mesos-slave process (could be the same)?_
 Pretty much everybody says that it should be the same as the node.

 _3. Do we need to rename Mesos Master too?_
 Most say No, except when slave's new name has a preferred pairing (e.g.
 Follower/Leader)

 _4. How will we phase in the new name and phase out the old name?_
 To calm any fears, we would have to go through a full deprecation cycle,
 introducing the new name in one release, while maintaining
 symlinks/aliases/duplicate-endpoints for the old name. In a subsequent
 release, we can remove the old name/endpoints. As we introduce the new
 Mesos 1.0 HTTP API, we will already be introducing breaking API changes,
 so this would be an ideal time to do a rename.

 Whether or not we decide to officially change the name in the code/APIs,
 some organizations are already using alternative terminologies in their
 presentations/scripts. We could at least try to agree upon a recommended
 alternative name for these purposes.

 _5. How do we vote on this?_
 First, FYI: https://www.apache.org/foundation/voting.html
 It seems there are two potentially separate items to vote on:

 Prop-A: Rename Mesos-Slave in the code/APIs
 Qualifies as a code modification, so a negative (binding) vote
 constitutes a veto. Note that there are no -1s from the Mesos PMC yet.
 After this week of discussion where the community is invited to share
 their thoughts/opinions, we will call for an official VOTE from the PMC
 members. The proposal will pass if there are at least three positive
 votes and no negative ones.

 Prop-B: Recommended Alternative Name for Slave
 This can follow the common format of majority rule. We can gather
 recommendations during this one week discussion period, and then vote on
 the top 2-3 finalists.

 On Thu, Jun 4, 2015 at 8:23 PM, Emilien Kenler eken...@wizcorp.jp
 mailto:eken...@wizcorp.jp wrote:

 +1 for keeping master/slave.

 On Fri, Jun 5, 2015 at 12:00 PM, Panyungao (Wingoal)
 panyun...@huawei.com mailto:panyun...@huawei.com wrote:

 +1  master/slave. 

 __ __

 These are only terminologies in software architecture.  They
 have different definitions from those of social or political
 view. 

 __ __

 *发件人:*zhou weitao [mailto:zhouwtl...@gmail.com
 mailto:zhouwtl...@gmail.com]
 *发送时间:*2015年6月5日10:40
 *收件人:*user@mesos.apache.org mailto:user@mesos.apache.org
 *主题:*Re: [DISCUSS] Renaming Mesos Slave

 __ __

 +1 master/slave, no change needed.

 __ __

 2015-06-05 0:10 GMT+08:00 Ankur Chauhan an...@malloc64.com
 mailto:an...@malloc64.com:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 +1 master/slave

 James made some very good points and there is no technical
 reason for
 wasting time on this.

 On 04/06/2015 08:45, James Vanns wrote:
  +1 master/slave, no change needed.
 
  I couldn't agree more. This is a barmy request; master/slave is
 a
  well understood common convention (if it isn't well 

Re: Register User Group n Bejing ,China

[Sam Chen]

Tommy ,
There are no correct and wrong name of User Group from Community
perspective , Our major goal is to move Mesos forward in China .  Having
said that in my previous email , since we are doing contribution to Mesos
and  running project based on Mesos . That's why I created this User Group
, and I figured it out there are no User Group in
http://mesos.apache.org/community/user-groups/ for now before you guys
jumping in. That's why we created it.
 if you think the one that you created is the right one .  I  am totally no
issue with that . My point is we need to push Mesos to the upstairs  and
make noise in the same time in China .  Mesos is still small baby in China
,let's work together and make it success .

I am already member of your User Group , could you please convert me into
co-organizer ? and i will merge all members into here . thanks.

Sam


On Mon, May 11, 2015 at 7:34 PM, tommy xiao xia...@gmail.com wrote:

 Can't agree more with Dave's option,  Beijing Mesos User Group is correct
 name for local event. How about think merged the group to the Beijing
 Mesos User Group? For Organizer, we can talk about detail merged way by
 offline contact.

 2015-05-11 16:03 GMT+08:00 Dave Lester d...@davelester.org:

  Thus far our user groups have been named and organized around single
 cities. In China we already have MUGs in Shanghai and Shenzhen, so I think
 Beijing Mesos User Group may be a more-logical name than China Mesos
 Meetup. Just my two cents. :)

 Dave

 On Sun, May 10, 2015, at 10:45 PM, Zhongyue Luo wrote:

 I think 16 and 23 doesn't make a difference. Tommy's group has already
 scheduled a meetup and also registered first. I think we should merge with
 http://www.meetup.com/Beijing-Mesos-User-Group/

 On Mon, May 11, 2015 at 12:02 PM, Sam Chen usultra...@gmail.com wrote:

 Dave and Tommy
 We got 23 Mesosers in this User Group , And this Meetup will be funded by
 LinkerNetworks Inc, which is focusing on Mesos and relevant development .
  Do you guys think  it will be good to merge your guys into our big User
 Group .
 Looking forward to hear your guys feedback .


 Sam


 On Mon, May 11, 2015 at 8:48 AM, Dave Lester d...@davelester.org wrote:


 Hi Sam and Tommy,

 It's awesome to hear about your interest in a Beijing MUG!

 Could you guys potentially merge these separate meetup groups so we have
 a single group to link to?

 Thanks,
 Dave

 On Sun, May 10, 2015, at 05:14 PM, tommy xiao wrote:

 Hi Sam,

 nice to meet you by email. meetup.com have already approved my request(
 http://www.meetup.com/mail/178886170210516992/). Could you please have
 some time to talk about beijing mesos user meetup in May 17.

 Mesos Beijing Kickoff Meeting
 (http://www.meetup.com/Beijing-Mesos-User-Group/events/222428331/)


 2015-05-10 23:08 GMT+08:00 Sam Chen usultra...@gmail.com:

 Hi , Guys,
 My name is Sam , I am working in China ,Beijing .  We got bunch of guys
 who is very interested in Mesos . Could you please add our Meetup in your
 guys page
 :http://mesos.apache.org/community/user-groups/

 Our Meetup URL  : http://www.meetup.com/China-mesos-Meetup/  ,Appreciated
 .




 --
 Deshi Xiao
 Twitter: xds2000
 E-mail: xiaods(AT)gmail.com








 --
 *Intel SSG/STO/BDT*
 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,
 China
 +862161166500






 --
 Deshi Xiao
 Twitter: xds2000
 E-mail: xiaods(AT)gmail.com

Register User Group n Bejing ,China

[Sam Chen]

Hi , Guys,
My name is Sam , I am working in China ,Beijing .  We got bunch of guys who
is very interested in Mesos . Could you please add our Meetup in your guys
page
:http://mesos.apache.org/community/user-groups/

Our Meetup URL  : http://www.meetup.com/China-mesos-Meetup/  ,Appreciated .

hi, Guys

[Sam Chen]

I  cannot send email to this email list .

Re: Spark (Streaming?) holding on to Mesos resources

[Sam Bessalah]

Hi Geraard,
isn't this the same issueas this?
https://issues.apache.org/jira/browse/MESOS-1688

On Mon, Jan 26, 2015 at 9:17 PM, Gerard Maas gerard.m...@gmail.com wrote:

 Hi,

 We are observing with certain regularity that our Spark  jobs, as Mesos
 framework, are hoarding resources and not releasing them, resulting in
 resource starvation to all jobs running on the Mesos cluster.

 For example:
 This is a job that has spark.cores.max = 4 and spark.executor.memory=3g

 IDFrameworkHostCPUsMem…5050-16506-1146497FooStreamingdnode-4.hdfs.private713.4
 GB…5050-16506-1146495FooStreaming
 dnode-0.hdfs.private16.4 GB…5050-16506-1146491FooStreaming
 dnode-5.hdfs.private711.9 GB…5050-16506-1146449FooStreaming
 dnode-3.hdfs.private74.9 GB…5050-16506-1146247FooStreaming
 dnode-1.hdfs.private0.55.9 GB…5050-16506-1146226FooStreaming
 dnode-2.hdfs.private37.9 GB…5050-16506-1144069FooStreaming
 dnode-3.hdfs.private18.7 GB…5050-16506-1133091FooStreaming
 dnode-5.hdfs.private11.7 GB…5050-16506-1133090FooStreaming
 dnode-2.hdfs.private55.2 GB…5050-16506-1133089FooStreaming
 dnode-1.hdfs.private6.56.3 GB…5050-16506-1133088FooStreaming
 dnode-4.hdfs.private1251 MB…5050-16506-1133087FooStreaming
 dnode-0.hdfs.private6.46.8 GB
 The only way to release the resources is by manually finding the process
 in the cluster and killing it. The jobs are often streaming but also batch
 jobs show this behavior. We have more streaming jobs than batch, so stats
 are biased.
 Any ideas of what's up here? Hopefully some very bad ugly bug that has
 been fixed already and that will urge us to upgrade our infra?

 Mesos 0.20 +  Marathon 0.7.4 + Spark 1.1.0

 -kr, Gerard.

Re: Looking for volunteers to help improve Mesos documentation

[Sam Taha]

What will be the process to submit changes? Will be the same process that
is document for submitting code patches along with the same review process?

I can help in the area of document get started writing a Framework.

Thanks,
Sam Taha

http://www.grandlogic.com


On Thu, Oct 31, 2013 at 9:26 PM, Dave Lester d...@ischool.berkeley.eduwrote:

 Shingo and Ryosuke: Awesome RE: translation.

 Anyone interested in translating in languages other than Japanese?

 Paco, interesting question as it relates to republishing via O'Reilly
 EPUB. I'm not sure if that's possible, but I suspect that others on the
 user list may know.


 On Thu, Oct 31, 2013 at 4:07 PM, Paco Nathan cet...@gmail.com wrote:

 I'm in.
 experience in writing getting started guides, a variety of markup
 workflows, general editorial work, reviews, etc.

 For that matter, if Apache would allow it, we could build our written
 collateral in Atlas and publish as a free O'Reilly EPUB.


 On Thu, Oct 31, 2013 at 5:45 PM, Dave Lester 
 d...@ischool.berkeley.eduwrote:

 I'd love to recruit new volunteers to help with Mesos documentation. I
 recently took a pass through the docs, but there's a lot of room for
 improvement. A few things that stand out to me include improving the
 getting started guide, reviewing docs before releases are cut, and perhaps
 language translation.

 Who is interested in helping, and what would you like to contribute?
 Please respond on the thread so folks can follow along.

 Dave

Powered by Mesos page

[Sam Taha]

Would it be possible to add Grand Logic and JobServer to the organizations
and products using and built on Mesos?

Thanks,
Sam Taha

http://www.grandlogic.com

Application Dependency Managaement ala YARN

[Sam Taha]

I am looking to implement a custom executor, but I do not want to require
users to distribute the JARs and other related resources associated with my
executor onto every slave node.

I see from the code that I can addUris and even .tgz that get downloaded
and extracted during the execution of the executor, but I am not clear if
this will be practical if my JARs and other resources needed by the java
executor will be in the 10s or 100s of megabytes.

It is important that I don't require the user to install anything on the
slave nodes other than the base Mesos package since my Framework
application is very dynamic and code jars can vary by job in most cases. So
I really want everything dynamically downloaded per job run and I would
also like the resources to be cached if they have not changed...etc.

Is there any plan to support something like what exist in YARN which deals
with these of lifecycle resource manage application dependencies across the
cluster? Here is theYARN overview of what I am kind of asking about:

http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/

Thanks in advance.

Thanks,
Sam Taha

http://www.grandlogic.com

Re: process isolation

[Sam Taha]

See comments from Ben Mahler on related question about isolation and using
cgroups with and without cpu subsystems and cfs enforced:

If using process isolation nothing is enforced.

If using cgroups isolation:

  with no subsystems: nothing is enforced.

  with the 'cpu' subsystem: this will enforce your cgroup gets at least 0.5
cpu resources. Think of this as a lower bound guarantee on your cpu quota.
You can use more if the system is otherwise idle.

  with the 'cpu' subsystem and with --cgroups_enable_cfs: this will enforce
your application gets at least 0.5 cpu resources and no more than 0.5 cpu
resources. This adds an upper bound on your cpu consumption.

So, fractions make sense and are definitely supported. Let me know if you
need more clarification here.

You may be interested in reading about cpu.shares:
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.html

Ben


On Mon, Sep 30, 2013 at 1:20 PM, Sam Taha taha...@gmail.com wrote:

 What does it mean to request fractiional CPU resources like 0.5 or 1.5. Is
 this meaningful when making requests and how does it relate to isolation 
 levels.
 For example, you can't really pin 1.5 cores in a cgroup.

 Should I really bother letting users/jobs request less than 1 cpu?

 Thanks,
 Sam Taha

 http://www.grandlogic.com




On Mon, Oct 21, 2013 at 10:17 AM, Vinod Kone vinodk...@gmail.com wrote:

 Yes.

 @vinodkone
 Sent from my mobile

  On Oct 21, 2013, at 5:05 AM, Paul Mackles pa...@loopr.com wrote:
 
  Hi - I just wanted to confirm my understanding of something... with
 process isolation, Mesos will not do anything if a given executor exceeds
 its resource allocation. In other words, if I accept a resource with 1GB of
 memory and then my executor uses 3GB, Mesos won't detect that the process
 exceeded its allocation and kill the process. For that, you need to enable
 cgroups at which point allocation limits are enforced by the OS. Did I get
 that right?
 
  --
  Thanks,
  Paul

JobServer scheduling and processing jobs on Mesos

[Sam Taha]

Greetings All,

Wanted to announce that JobServer, a job scheduling/processing/management
application, is now integrated with Mesos.

I put out a brief post on my blog:
http://grandlogic.blogspot.com/2013/10/jobserver-with-mesos.html

This initial integration with Mesos went a bit better than I expected.
JobServer was pretty well suited to have Mesos plugged into it and it fit
well with JobServer's architecture.

This initial version only has support for the built-in Mesos command
executor. I plan to add a custom JobServer executor in the future so jobs
can execute with the full plugin API of JobServer (called soafaces). And
there is still more work around failover, retrying jobs and resiliency to
get Mesos on par with the native JobServer fearures, but this is a pretty
good start.

This version is pretty capable but still need some work. You can create
jobs and schedule them just like any other JobServer job and track them in
the monitoring/reporting UIs like any other native JobServer job. You are
just limited to executing shell scripts per the Mesos command executor. But
overall Mesos jobs have many of the capabilities as native JobServer jobs,
they just now run on a Mesos cluster. Also, you can mix and match Mesos
clusters with existing statically defined distributed Agent model in
JobServer, so users can migrate gradually to Mesos cluster or have a hybrid
environment if they choose.

JobServer has the concept of Partitions each with its own resources and
queues. Users can create many Partitions and jobs run in a Partition with
defined resource limits. So Mesos was a perfect fit for this and obviously
provided many awesome clustering and resource utilization features that the
static distributed Agent model that exists in JobServer did not have.

I am still looking at refining how I fairly distribute the offered jobs
within my Framework so that all the Partitions get a fair crack at the
offers. The algo I think is pretty cool but I expect to tweak it a bit. I
also added a highlevel Mesos Admin page that lets admins see what is going
on, on the Mesos side of things, and how request are being matched up with
offers (and if they are not for some reason), to help troubleshoot resource
matching issues...etc.

I am planning more reporting features as I integrate more with Mesos, so
looking forward to all new features like integrating isolation controls as
well as gathering/reporting usage stats...etc.

Please give JobServer a try if you can and provide any feedback. The
current released version is the JobServer Standard edition, which is pretty
easy to setup and self contained. Just untar and setup Mesos URL and you
should be set to go for the most part and you can then create jobs that run
any script on Mesos. Make sure you configure your Partition to use Mesos
as the cluster type in the Admin UIs.

http://www.grandlogic.com/downloads_content/downloads/rawDownload-v3-6.html

Thanks for everyone's help on the mailing list.

Thanks,
Sam Taha

http://www.grandlogic.com

Re: Resource utilization stats

[Sam Taha]

Thanks Ben. I will see if I can incorporate this in my reporting GUI. Do
you know how long lived these are stats are?

Ideally, I am thinking it would be nice to return basic usage stats maybe
via the TaskStatus so that I can capture them when they are returned when
the TASK_FINISHED event is fired. I capture stats in my local app database
as I track the events per job.

Thanks,
Sam Taha

http://www.grandlogic.com


On Fri, Oct 18, 2013 at 1:34 PM, Benjamin Mahler
benjamin.mah...@gmail.comwrote:

 Hey Sam,

 You can use the json monitoring endpoint on the slave to retrieve cpu
 usage information.

 $ curl localhost:5051/monitor/statistics.json

 Take a look at the data, this is how the webui displays cpu usage
 information.


 On Thu, Oct 17, 2013 at 12:10 PM, Vinod Kone vinodk...@gmail.com wrote:

 MESOS-581 is tracking cpu/mem usage of master/slave process themselves.

 I think what you are looking for is close to
 https://issues.apache.org/jira/browse/MESOS-62


 On Thu, Oct 17, 2013 at 12:06 PM, Sam Taha taha...@gmail.com wrote:

 https://issues.apache.org/jira/browse/MESOS-581


 On Thu, Oct 17, 2013 at 3:05 PM, Sam Taha taha...@gmail.com wrote:

 Looks like it is not implemented



 On Thu, Oct 17, 2013 at 7:55 AM, Sam Taha taha...@gmail.com wrote:

 Is there currently a way for a Framework to get back CPU/memory usage
 statistics as a task is running or after the task has finished? From Java
 API specifically?

 Thanks,
 Sam Taha

 http://www.grandlogic.com

Re: Check status of Framework connection

[Sam Taha]

Note, I am not using Zookeeper in my test environment. I am just connecting
to a single master node. Obviously with zookeeper there would be master
failover if the primary went down, but there could be situations when the
entire master/zookeeper cluster are down, so the scenario still applies I
think. But I would think either way I should get a callback disconnect if
the single master or master/zookeeper cluster are shutdown either manually
or crash?

Thanks,
Sam Taha

http://www.grandlogic.com


On Fri, Oct 18, 2013 at 4:42 PM, Sam Taha taha...@gmail.com wrote:

 When my Framework has connected to the master, I do not seem to get the
 Scheduler.disconnected() callback when I kill the Master. Is this expected
 behavior or should my Framework get a disconnected() callback if I kill the
 master?

 Thanks,
 Sam Taha

 http://www.grandlogic.com


 On Wed, Oct 9, 2013 at 1:38 PM, Vinod Kone vinodk...@gmail.com wrote:

 You should get a disconnected() callback when the master is disconnected.


 On Wed, Oct 9, 2013 at 10:28 AM, Sam Taha taha...@gmail.com wrote:

 From java Framework, is their a way to to just ping the master to see if
 my Scheduler/Driver is properly connected to mesos? None of the methods in
 the SchedulerDriver look appropriate for just a simple round trip check if
 the Framework is connected.

 I need to do this in cases where I have declined ALL offers for a period
 of time, so I am not getting anything back from mesos during this time, but
 I still need to give the user feedback if the Framework is actually
 connected or not.

 Relying on Scheduler.registered() does not give current connection
 status, since I may have connected hours ago and I may have declined
 offers, but I still need to check if my Framework is connected.

 Thanks,
 Sam Taha

 http://www.grandlogic.com

Re: Check status of Framework connection

[Sam Taha]

Thanks Vinod, that explains what I am seeing. On a related question. Let's
say the master/zookeeper is down for a period of time and then restarted
and my Framework is running during that time (and previously connected to
mesos). Will the master try to reconnect with my Framework again on its own
and I will start receiving offers? Or will I have to reconnect from my end
by periodically checking if the master/zookeeper is back up again?

Thanks,
Sam Taha

http://www.grandlogic.com


On Fri, Oct 18, 2013 at 5:38 PM, Vinod Kone vinodk...@gmail.com wrote:

 You are right. disconnected() callback is not called when you use a stand
 alone master *without* zookeeper. This is an artifact of how the scheduler
 driver works without zookeeper in the mix.

  That said, if you do use zookeeper the disconnected() callback should be
 always called, even when the whole zk cluster is down or the master is down.


 On Fri, Oct 18, 2013 at 2:22 PM, Sam Taha taha...@gmail.com wrote:

 Note, I am not using Zookeeper in my test environment. I am just
 connecting to a single master node. Obviously with zookeeper there would be
 master failover if the primary went down, but there could be situations
 when the entire master/zookeeper cluster are down, so the scenario still
 applies I think. But I would think either way I should get a callback
 disconnect if the single master or master/zookeeper cluster are shutdown
 either manually or crash?

 Thanks,
 Sam Taha

 http://www.grandlogic.com


 On Fri, Oct 18, 2013 at 4:42 PM, Sam Taha taha...@gmail.com wrote:

 When my Framework has connected to the master, I do not seem to get the
 Scheduler.disconnected() callback when I kill the Master. Is this expected
 behavior or should my Framework get a disconnected() callback if I kill the
 master?

 Thanks,
 Sam Taha

 http://www.grandlogic.com


 On Wed, Oct 9, 2013 at 1:38 PM, Vinod Kone vinodk...@gmail.com wrote:

 You should get a disconnected() callback when the master is
 disconnected.


 On Wed, Oct 9, 2013 at 10:28 AM, Sam Taha taha...@gmail.com wrote:

 From java Framework, is their a way to to just ping the master to see
 if my Scheduler/Driver is properly connected to mesos? None of the methods
 in the SchedulerDriver look appropriate for just a simple round trip check
 if the Framework is connected.

 I need to do this in cases where I have declined ALL offers for a
 period of time, so I am not getting anything back from mesos during this
 time, but I still need to give the user feedback if the Framework is
 actually connected or not.

 Relying on Scheduler.registered() does not give current connection
 status, since I may have connected hours ago and I may have declined
 offers, but I still need to check if my Framework is connected.

 Thanks,
 Sam Taha

 http://www.grandlogic.com

Matching a single Offer with multiple Requests

[Sam Taha]

Simple example scenario:

If my Framework/Scheduler gets an Offer for say 2 cpu and 10G (from a
single Slave/OfferID) and let's say I have two job requests that each need
1 cpu and 5G each. Now, can I make both requests against the same Offer
(same OfferID) or can I only make one request even though there are enough
resources for the second job?

As I am looping through the Offers and comparing them to my requests, how
should I keep track that my first request already consumed and
(launchedTask) resources of 1cpu and 5G on this OfferID (should I update
the Offer/Resource to reflect the change as I loop to the next request in
my list)? Or can I only make one request per OfferID and just wait to
presented with the left over resources on the next time Mesos callbacks?

Basically my question is if I have not consumed the full resources for that
Offer, do I have to wait for the next time Mesos presents me with the rest
of the Offer resource in order to launch my second request or do I track
all this on my Framework side as I am matching Offers to Requests?

Thanks,
Sam Taha

http://www.grandlogic.com

Re: Matching a single Offer with multiple Requests

[Sam Taha]

Sorry, just noticed that SchedulerDriver.launchTasks() takes a List of
Tasks, so I guess you can launch multiple job/Task requests against the
same OfferID if you make them all in the same launchTasks() call.

Is this an all or nothing batch call if one of them say requests more than
what is available on the slave?

Also is it kosher to make multiple launchTasks() calls for the same OfferID
or do they have to batched in one launchTasks() call? Just want to make
sure I follow recommended API usage and not get my self in deep water or
any unexpected behavior for how resources are allocated.

Thanks,
Sam Taha

http://www.grandlogic.com


On Tue, Oct 1, 2013 at 12:32 PM, Sam Taha taha...@gmail.com wrote:

 Simple example scenario:

 If my Framework/Scheduler gets an Offer for say 2 cpu and 10G (from a
 single Slave/OfferID) and let's say I have two job requests that each need
 1 cpu and 5G each. Now, can I make both requests against the same Offer
 (same OfferID) or can I only make one request even though there are enough
 resources for the second job?

 As I am looping through the Offers and comparing them to my requests, how
 should I keep track that my first request already consumed and
 (launchedTask) resources of 1cpu and 5G on this OfferID (should I update
 the Offer/Resource to reflect the change as I loop to the next request in
 my list)? Or can I only make one request per OfferID and just wait to
 presented with the left over resources on the next time Mesos callbacks?

 Basically my question is if I have not consumed the full resources for
 that Offer, do I have to wait for the next time Mesos presents me with the
 rest of the Offer resource in order to launch my second request or do I
 track all this on my Framework side as I am matching Offers to Requests?

 Thanks,
 Sam Taha

 http://www.grandlogic.com

Re: Design advice

[Sam Taha]

Yes that was my thinking for the slave/execution side of things. My
partitions should map pretty well to slave executors/tasks and the
cgroup/isolation capabilities of mesos will be great benefit. However, I
still have some cases where users want to run each job (even jobs in the
same partition grouping) to under its own linux process so I am hoping I
can make this a runtime decision whether to use a shared partition executor
(on the given slave) or launch a new executor just for that one job and
then terminate the executor at then end of job run (I think I can use the
default command executor for this...I think).

I will take your advice and build a central framework for now and have it
dispatch resources to each Partition thread.

Thanks for you help.

Sam Taha
http://www.grandlogic.com


On Tue, Sep 10, 2013 at 1:01 PM, Vinod Kone vinodk...@gmail.com wrote:

 Sam,

 Glad to see you are interested in building a framework on top of Mesos!

 From your description, it looks like your partitions can be directly
 mapped to Mesos executors and jobs to Mesos tasks. In Mesos, each
 executor is run under a cgroup with a given set of resources. This enables
 multiple executors to be run on the same host with isolation. Also, an
 executor can run multiple jobs/tasks underneath it.

 In your system, do users need to specifically control the partitions?  I
 would imagine it would be one less thing for users to worry about if Mesos
 can take care of it? A nice thing about using executors is that the total
 no.of resources (for executor and tasks underneath it) can be dynamically
 adjusted. So you don't have to statically over/under provision resources
 for your partitions.

 As per multiple frameworks per jvm, while its definitely possible I
 wouldn't recommend it. I think you would be much better off running one
 central framework and running different partitions as executors.

 HTH,


 On Tue, Sep 10, 2013 at 9:41 AM, Sam Taha taha...@gmail.com wrote:

 I am hoping to get some advice on how to build out my framework scheduler
 for a job scheduling/processing platform, so I don't paint myself in a
 corner with any potential limitations in mesos resource allocation or the
 java SchedulerDriver capabilities. Some background:

 - I have a JVM based job scheduling engine that schedules jobs based on
 repeating time patterns (like cron) and also allows for on-demand job
 execution.
 - The system supports users creating an number of partitions/queues that
 have resource preferences defined by the user. So one partitions may run a
 certain set of jobs and another partition may run another set of jobs.
 Currently the resources are statically allocated to one or more computers
 and the user can define the max number of jobs that can be run on each host
 in the Partition.
 - The scheduling engine is all run out of a single JVM process that is
 heavily threaded/concurrent.

 So my initial thinking is to create a mesos Sched/Framework instance per
 partition/queue thread in the my engine (there could be many partitions
 since this is user created/controlled via GUI). Each SchedulerDriver
 (living in the JVM) would request some set of resources from Mesos (as
 defined by the size of of the queue of jobs running in the Partition) when
 jobs become ready to run on reoccurring time schedules.

 Now my question is, if it is practical to have multiple mesos Schedulers
 Frameworks in a single JVM and each with different resource requirements.
 Or should I build a single central mesos scheduler in the JVM and have all
 my partitions/queues make request to this central mesos Scheduler that
 talks to the mesos master?

 I am not sure how mesos scales if for example you have many framework
 schedulers running out of the same JVM/process? And in general if you have
 a large number of Scheduler instances will this cause resource distribution
 problems with on one scheduler getting too many resources allocatedetc.

 So the bottom line question is should have create/instantiate a single
 central Framework Scheduler in my engine (that proxies requests/offers with
 mesos) or can I create one per Partition thread.

 The per partition thread approach can give me finer resource control
 (request, Filters,etc) but I am not sure if this is practical.

 Thanks,
 Sam Taha
 http://www.grandlogic.com