Re: A Redis Framework for Apache Mesos

[Avinash Sridharan]

On Mon, Jul 4, 2016 at 12:11 AM, DhilipKumar Sankaranarayanan <
s.dhilipku...@gmail.com> wrote:

> *"*
> *Would be nice to get on the frameworks page:*
>
> *http://mesos.apache.org/documentation/latest/frameworks/
>  *
> *?*
> *"*
> Sure i would love to do that, where should i raise a PR?
>
I think you can raise a PR on github. The preferred route though is to
generate an RB request on Apache review board (https://reviews.apache.org/r/)
. One of the Shepherds could help with this.
+Vinod ^^

> *"Also is it already part of the DC/OS universe ?"*
>
> Yes we have reasonable documentation
> with regards to DCOS integration,
> Please give it a try and it will be awesome to get your feedback on that.
>
> Regards,
> Dhilip
>
> On Sun, Jul 3, 2016 at 6:09 PM, Avinash Sridharan 
> wrote:
>
>> Would be nice to get on the frameworks page:
>> http://mesos.apache.org/documentation/latest/frameworks/
>> ?
>>
>> Also is it already part of the DC/OS universe ?
>>
>> On Sun, Jul 3, 2016 at 2:29 PM, Christoph Heer 
>> wrote:
>>
>>> Hi,
>>>
>>> it looks really cool. Can you maybe explain why do you use etcd for
>>> leader election instead of the normally already existing Zookeeper cluster?
>>> Do you use some special etcd features?
>>>
>>> Best regards
>>> Christoph
>>>
>>> > On 03 Jul 2016, at 17:49, tommy xiao  wrote:
>>> >
>>> > Cool. thanks for your sharing.
>>> >
>>> > 2016-07-03 23:44 GMT+08:00 DhilipKumar Sankaranarayanan <
>>> s.dhilipku...@gmail.com>:
>>> > Hello All,
>>> >
>>> >
>>> >
>>> > We have built a framework for provisioning redis-servers in Apache
>>> Mesos enabled infrastructure. It would be awesome to get communities
>>> feedback. While mesos ecosystem is strengthening its capability in storage
>>> layer, redis could be an addition to the lineup.  This is primarily
>>> intended for providers who would like to host redis as a service in their
>>> infrastructure.
>>> >
>>> >
>>> >
>>> > There is an elaborate README about the project which should help in
>>> setting it up: https://github.com/mesos/mr-redis. Please let us know if
>>> there is anything missing in the documentation by raising a PR or opening
>>> an issue or even writing to us.
>>> >
>>> >
>>> >
>>> > This project is also packaged as  mr-redis with DCOS so should be
>>> pretty straight forward to install it via DCOS CLI or DCOS GUI.
>>> >
>>> >
>>> >
>>> > (A Step by Step guide is provided in the README for your convenience)
>>> >
>>> >
>>> >
>>> > Salient Features of this project include:
>>> >
>>> > 1)  Create multiple redis clusters (Master-Slave Cluster) with ease
>>> >
>>> > 2)  Redis instances recover in seconds and not in minutes
>>> >
>>> > 3)  If a Master fails a slaves is automatically promoted as the
>>> New master, all the old slaves now replicate from the newly master plus a
>>> new slave is added to the cluster.  All this without using redis-sentinel
>>> in your datacenter and all these happens in a couple of seconds.
>>> >
>>> > 4)  A CLI to perform basic operations such as create / status /
>>> delete redis instances on the fly.  CLI is cross compiled for Windows and
>>> Darwin users too.
>>> >
>>> > 5)  Scheduler is a HTTP REST server which also can respond to
>>> simple Angular UI we have built to get started with.  Instructions on how
>>> to setup the UI is here.
>>> https://github.com/mesos/mr-redis/tree/master/ui/app
>>> >
>>> >
>>> >
>>> > We also had an opportunity to talk about this during the recent
>>> MesosCon 2016:
>>> https://www.youtube.com/watch?v=xe-Gom5tOl0=32=PLGeM09tlguZQVL7ZsfNMffX9h1rGNVqnC
>>> >
>>> >
>>> >
>>> > Future Work:
>>> >
>>> > · Implement a proxy technique to expose one single endpoint
>>> for the redis instance.  (Work in progress)
>>> >
>>> > · Implement Memory Cgroups per redis PROCS
>>> >
>>> > · Add support for Redis 3.0 cluster instances (Adding shards
>>> to a running redis instance)
>>> >
>>> > · Implement integration test suite and benchmarking suite to
>>> the framework.
>>> >
>>> >
>>> >
>>> > Special thanks to Adobe.io team who expressed interest in
>>> collaborating in the development of this product. I’m sure it’s going to be
>>> great working with all of you folks.
>>> >
>>> >
>>> >
>>> > Advanced happy Independence day America and happy week ahead rest of
>>> the world.
>>> >
>>> >
>>> >
>>> > Looking forward to hear from you all,
>>> >
>>> > Dhilip
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Deshi Xiao
>>> > Twitter: xds2000
>>> > E-mail: xiaods(AT)gmail.com
>>>
>>>
>>
>>
>> --
>> Avinash Sridharan, Mesosphere
>> +1 (323) 702 5245
>>
>
>


-- 
Avinash Sridharan, Mesosphere
+1 (323) 702 5245

Re: Documentation for ACCEPT HTTP API

[Giulio Eulisse]

Dear Jay,

thank you for your reply.

Yes, I am aware of those pages and that's what I used so far. As I said,
they are actually quite clear and allowed me to get a simple "reject all
offers" framework up and running in one evening.

However, if you look at the scheduler-http-api page, you will see that the
documentation for the "ACCEPT" message is completely lacking the
description of what should go into the "operations" field of the JSON
payload, unless I am missing something completely trivial. My question is
therefore where I can find the schema of what should go there (and possibly
a simple example).

-- 
Ciao,
Giulio

On Mon, Jul 4, 2016 at 1:24 PM Jay JN Guo  wrote:

> Hi Giulio,
>
> For scheduler/executor HTTP API, please refer to:
> http://mesos.apache.org/documentation/latest/scheduler-http-api/
> http://mesos.apache.org/documentation/latest/executor-http-api/
>
> If you find anything missing there, let us know.
>
> Also, we are working on Operator HTTP API documentation and hopefully
> could get it out soon.
>
> cheers,
> /Jay
>
>
> - Original message -
> From: Giulio Eulisse 
> To: "user@mesos.apache.org" 
> Cc:
> Subject: Documentation for ACCEPT HTTP API
>
> Date: Mon, Jul 4, 2016 6:01 PM
>
> Dear all,
>
> I've started writing a simple framework using node.js and the HTTP
> Scheduler API. I've managed to subscribe to the event stream, parse
> messages and decline offers quite easily, however I'm having a bit of
> trouble accepting the offers and launching tasks, since I cannot find any
> complete example for the JSON format the various operations should have. I
> assume I can reverse engineer mesos.proto and do a bit of trial and error,
> but I was wondering if I was simply missing some proper documentation. Any
> suggestions?
>
> --
> Ciao,
> Giulio
>
>

Re: Documentation for ACCEPT HTTP API

[Jay JN Guo]

Hi Giulio,
 
For scheduler/executor HTTP API, please refer to:
http://mesos.apache.org/documentation/latest/scheduler-http-api/
http://mesos.apache.org/documentation/latest/executor-http-api/
 
If you find anything missing there, let us know.
 
Also, we are working on Operator HTTP API documentation and hopefully could get it out soon.
 
cheers,
/Jay
 
- Original message -From: Giulio Eulisse To: "user@mesos.apache.org" Cc:Subject: Documentation for ACCEPT HTTP APIDate: Mon, Jul 4, 2016 6:01 PM 
Dear all,
 
I've started writing a simple framework using node.js and the HTTP Scheduler API. I've managed to subscribe to the event stream, parse messages and decline offers quite easily, however I'm having a bit of trouble accepting the offers and launching tasks, since I cannot find any complete example for the JSON format the various operations should have. I assume I can reverse engineer mesos.proto and do a bit of trial and error, but I was wondering if I was simply missing some proper documentation. Any suggestions?
 
-- 
Ciao,
Giulio
 

Documentation for ACCEPT HTTP API

[Giulio Eulisse]

Dear all,

I've started writing a simple framework using node.js and the HTTP
Scheduler API. I've managed to subscribe to the event stream, parse
messages and decline offers quite easily, however I'm having a bit of
trouble accepting the offers and launching tasks, since I cannot find any
complete example for the JSON format the various operations should have. I
assume I can reverse engineer mesos.proto and do a bit of trial and error,
but I was wondering if I was simply missing some proper documentation. Any
suggestions?

-- 
Ciao,
Giulio

Re: Mesos on hybrid AWS - Best practices?

[o...@magnetic.io]

I agree about seperate clusters and tooling on top. This is exactly where 
several of our customers are using Vamp (vamp.io) for: gradual and controlled 
(canary) moving from legacy/current environments/applications (often on own 
DC’s) to container-based modern environments (often on public clouds like AWS). 
Vamp’s gateways can manage the canary routing based on HAproxy, and our 
integration with DC/OS can handle container deployments, (auto)scaling and 
routing/lb’ing on the modern DC/OS cluster.

Cheers, Olaf

Olaf Molenveld
co-founder / CEO
-
VAMP: the Canary test and release platform for containers by magnetic.io
E: o...@magnetic.io
T: +31653362783
Skype: olafmol
www.vamp.io 
www.magnetic.io 






> On 30 Jun 2016, at 19:05, Sharma Podila  wrote:
> 
> I would second the suggestion of separate Mesos clusters for DC and AWS, with 
> a layer on top for picking one or either based on the job SLAs and resource 
> requirements.
> The local storage on cloud instances are more ephemeral than I'd expect the 
> DC instances to be. So, persistent storage of job metadata needs 
> consideration. Using something like DynamoDB may work, however, depending on 
> the scale of your operations, you may have to plan for EC2 rate limiting its 
> API calls and/or paying for higher IOPS for data storage/access. 
> Treating the cloud instances as immutable infrastructure has additional 
> benefits. For example, we deploy new Mesos master ASG for version upgrades, 
> let them join the quorum, and then "tear down" the old master ASG. Same for 
> agents. Although, for agent migration our framework does coordinate migration 
> of jobs from old agent ASG to new one with some SLAs on not too many 
> instances of a service being down at a time. Sort of what the maintenance 
> primitives from Mesos aim to address.
> 
> 
> On Thu, Jun 30, 2016 at 9:41 AM, Ken Sipe  > wrote:
> I would suggest a cluster on AWS and a cluster on-prem.Then tooling on 
> top to manage between the 2.
> It is unlikely that a failure of a task on-prem should have a scheduled 
> replacement on AWS or vise versa.It is likely that you will end up 
> creating constraints to statically partition the clusters anyway IMO.
> 2 Clusters eliminates most of your proposed questions.
> 
> ken
> 
> > On Jun 30, 2016, at 10:57 AM, Florian Pfeiffer  > > wrote:
> >
> > Hi,
> >
> > the last 2 years I managed a mesos cluster with bare-metal on-premise. Now 
> > at my new company, the situation is a little bit different, and I'm 
> > wondering if there are some kind of best practices:
> > The company is in the middle of a transition from on-premise to AWS. The 
> > old stuff is still running in the DC, the newer micro services are running 
> > within autoscales groups on AWS and other AWS services like DynamoDB, 
> > Kinesis and Lambda are also on the rise.
> >
> > So in my naive view of the world (where no problems occur. never!) I'm 
> > thinking that it would be great to span a hybrid mesos cluster over AWS 
> > to leverage the still available resources in the DC which gets more and 
> > more underutilized over the time.
> >
> > Now my naive world view slowly crumbles, and I realize that I'm missing the 
> > experience with AWS. Questions that are already popping up (beside all 
> > those Questions, where I currently don't know that I will have them...) are:
> > * Is Virtual Private Gateway to my VPC enough, or do I need to aim for a 
> > Direct Connect?
> > * Put everything into one Account, or use a Multi-Account strategy? (Mainly 
> > to prevent things running amok and drag stuff down while running into an 
> > account wide shared limit?)
> > * Will e.g. DynamoDb be "fast" enough if it's accessed from the Datacenter.
> >
> > I'll appreciate any feedback or lessons learned about that topic :)
> >
> > Thanks,
> > Florian
> >
> 
> 

Re: A Redis Framework for Apache Mesos

[DhilipKumar Sankaranarayanan]

We use ETCD to store the state of the scheduler (mr-redis) not to perform
master election like "Mesos Master".  The Scheduler itself is stateless it
stores plenty of information with respect to the redis instances in etcd.
 we already have a request from
a user to start supporting zookeeper from the community.  I haven't started
working on that yet.  Any volunteers are welcome.

Plus etcd is not very hard to setup either, probably few commands
 and you have a running
instance of etcd.

Hope this makes sense,
Dhilip

On Sun, Jul 3, 2016 at 2:29 PM, Christoph Heer 
wrote:

> Hi,
>
> it looks really cool. Can you maybe explain why do you use etcd for leader
> election instead of the normally already existing Zookeeper cluster? Do you
> use some special etcd features?
>
> Best regards
> Christoph
>
> > On 03 Jul 2016, at 17:49, tommy xiao  wrote:
> >
> > Cool. thanks for your sharing.
> >
> > 2016-07-03 23:44 GMT+08:00 DhilipKumar Sankaranarayanan <
> s.dhilipku...@gmail.com>:
> > Hello All,
> >
> >
> >
> > We have built a framework for provisioning redis-servers in Apache Mesos
> enabled infrastructure. It would be awesome to get communities feedback.
> While mesos ecosystem is strengthening its capability in storage layer,
> redis could be an addition to the lineup.  This is primarily intended for
> providers who would like to host redis as a service in their infrastructure.
> >
> >
> >
> > There is an elaborate README about the project which should help in
> setting it up: https://github.com/mesos/mr-redis. Please let us know if
> there is anything missing in the documentation by raising a PR or opening
> an issue or even writing to us.
> >
> >
> >
> > This project is also packaged as  mr-redis with DCOS so should be pretty
> straight forward to install it via DCOS CLI or DCOS GUI.
> >
> >
> >
> > (A Step by Step guide is provided in the README for your convenience)
> >
> >
> >
> > Salient Features of this project include:
> >
> > 1)  Create multiple redis clusters (Master-Slave Cluster) with ease
> >
> > 2)  Redis instances recover in seconds and not in minutes
> >
> > 3)  If a Master fails a slaves is automatically promoted as the New
> master, all the old slaves now replicate from the newly master plus a new
> slave is added to the cluster.  All this without using redis-sentinel in
> your datacenter and all these happens in a couple of seconds.
> >
> > 4)  A CLI to perform basic operations such as create / status /
> delete redis instances on the fly.  CLI is cross compiled for Windows and
> Darwin users too.
> >
> > 5)  Scheduler is a HTTP REST server which also can respond to simple
> Angular UI we have built to get started with.  Instructions on how to setup
> the UI is here. https://github.com/mesos/mr-redis/tree/master/ui/app
> >
> >
> >
> > We also had an opportunity to talk about this during the recent MesosCon
> 2016:
> https://www.youtube.com/watch?v=xe-Gom5tOl0=32=PLGeM09tlguZQVL7ZsfNMffX9h1rGNVqnC
> >
> >
> >
> > Future Work:
> >
> > · Implement a proxy technique to expose one single endpoint for
> the redis instance.  (Work in progress)
> >
> > · Implement Memory Cgroups per redis PROCS
> >
> > · Add support for Redis 3.0 cluster instances (Adding shards to
> a running redis instance)
> >
> > · Implement integration test suite and benchmarking suite to the
> framework.
> >
> >
> >
> > Special thanks to Adobe.io team who expressed interest in collaborating
> in the development of this product. I’m sure it’s going to be great working
> with all of you folks.
> >
> >
> >
> > Advanced happy Independence day America and happy week ahead rest of the
> world.
> >
> >
> >
> > Looking forward to hear from you all,
> >
> > Dhilip
> >
> >
> >
> >
> >
> > --
> > Deshi Xiao
> > Twitter: xds2000
> > E-mail: xiaods(AT)gmail.com
>
>

Re: A Redis Framework for Apache Mesos

[DhilipKumar Sankaranarayanan]

*"*
*Would be nice to get on the frameworks page:*

*http://mesos.apache.org/documentation/latest/frameworks/
 *
*?*
*"*
Sure i would love to do that, where should i raise a PR?

*"Also is it already part of the DC/OS universe ?"*

Yes we have reasonable documentation
with regards to DCOS integration,
Please give it a try and it will be awesome to get your feedback on that.

Regards,
Dhilip

On Sun, Jul 3, 2016 at 6:09 PM, Avinash Sridharan 
wrote:

> Would be nice to get on the frameworks page:
> http://mesos.apache.org/documentation/latest/frameworks/
> ?
>
> Also is it already part of the DC/OS universe ?
>
> On Sun, Jul 3, 2016 at 2:29 PM, Christoph Heer 
> wrote:
>
>> Hi,
>>
>> it looks really cool. Can you maybe explain why do you use etcd for
>> leader election instead of the normally already existing Zookeeper cluster?
>> Do you use some special etcd features?
>>
>> Best regards
>> Christoph
>>
>> > On 03 Jul 2016, at 17:49, tommy xiao  wrote:
>> >
>> > Cool. thanks for your sharing.
>> >
>> > 2016-07-03 23:44 GMT+08:00 DhilipKumar Sankaranarayanan <
>> s.dhilipku...@gmail.com>:
>> > Hello All,
>> >
>> >
>> >
>> > We have built a framework for provisioning redis-servers in Apache
>> Mesos enabled infrastructure. It would be awesome to get communities
>> feedback. While mesos ecosystem is strengthening its capability in storage
>> layer, redis could be an addition to the lineup.  This is primarily
>> intended for providers who would like to host redis as a service in their
>> infrastructure.
>> >
>> >
>> >
>> > There is an elaborate README about the project which should help in
>> setting it up: https://github.com/mesos/mr-redis. Please let us know if
>> there is anything missing in the documentation by raising a PR or opening
>> an issue or even writing to us.
>> >
>> >
>> >
>> > This project is also packaged as  mr-redis with DCOS so should be
>> pretty straight forward to install it via DCOS CLI or DCOS GUI.
>> >
>> >
>> >
>> > (A Step by Step guide is provided in the README for your convenience)
>> >
>> >
>> >
>> > Salient Features of this project include:
>> >
>> > 1)  Create multiple redis clusters (Master-Slave Cluster) with ease
>> >
>> > 2)  Redis instances recover in seconds and not in minutes
>> >
>> > 3)  If a Master fails a slaves is automatically promoted as the New
>> master, all the old slaves now replicate from the newly master plus a new
>> slave is added to the cluster.  All this without using redis-sentinel in
>> your datacenter and all these happens in a couple of seconds.
>> >
>> > 4)  A CLI to perform basic operations such as create / status /
>> delete redis instances on the fly.  CLI is cross compiled for Windows and
>> Darwin users too.
>> >
>> > 5)  Scheduler is a HTTP REST server which also can respond to
>> simple Angular UI we have built to get started with.  Instructions on how
>> to setup the UI is here.
>> https://github.com/mesos/mr-redis/tree/master/ui/app
>> >
>> >
>> >
>> > We also had an opportunity to talk about this during the recent
>> MesosCon 2016:
>> https://www.youtube.com/watch?v=xe-Gom5tOl0=32=PLGeM09tlguZQVL7ZsfNMffX9h1rGNVqnC
>> >
>> >
>> >
>> > Future Work:
>> >
>> > · Implement a proxy technique to expose one single endpoint for
>> the redis instance.  (Work in progress)
>> >
>> > · Implement Memory Cgroups per redis PROCS
>> >
>> > · Add support for Redis 3.0 cluster instances (Adding shards to
>> a running redis instance)
>> >
>> > · Implement integration test suite and benchmarking suite to
>> the framework.
>> >
>> >
>> >
>> > Special thanks to Adobe.io team who expressed interest in collaborating
>> in the development of this product. I’m sure it’s going to be great working
>> with all of you folks.
>> >
>> >
>> >
>> > Advanced happy Independence day America and happy week ahead rest of
>> the world.
>> >
>> >
>> >
>> > Looking forward to hear from you all,
>> >
>> > Dhilip
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Deshi Xiao
>> > Twitter: xds2000
>> > E-mail: xiaods(AT)gmail.com
>>
>>
>
>
> --
> Avinash Sridharan, Mesosphere
> +1 (323) 702 5245
>