Hi Gurvinder, our team have an experience with Mesos on CoreOS with fleet, and we decided to switch to bare metal deployments and here are our main reasons.
First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. From DevOps point of view it is hard to control such thing as ZooKeeper restarts (and ensemble rolling restarts as well) which is the core service discovery mechanism for Mesos. You have to put some sidekick services to provide peer discovery mechanics and it doesn’t look very robust. Very common use case with Mesos is running Docker on top of it either with Marathon or with Aurora. But Docker service needs to be installed on worker nodes. So you’re coming to Docker-into-Docker situation which cancels all advantages of both transparent resource management and simple deployment configuration. One more point on Mesos inside Docker here, is that you have to attach Mesos data directories from container to a host. Given that you’re already running Mesos container in privileged mode and sharing directories with state with host there is no more reasons to run Mesos inside a container. And consider container restart (not just failure) with registry corruption and following synchronization issues. Another our use case with multi-region cluster deployments showed some issues with etcd heartbeat/leader election timeouts, which need to be increased in order to handle bigger latencies between data centers. If timeouts increase fleet starts to work in unpredictable way, loosing and finding peer nodes again which is not appropriate in production environment. You can take a look at this configuration for Mesos-CoreOS-HA as well https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha -- Anton Kirillov Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote: > Hi Micheal, > > Yes I tested the tutorial and it works fine for testing. Later on I used > fleet to run mesos workers on all coreos machines too. I was wondering > how the landscape is looking in the community regarding coreos. As is > there any interest from community or mesos team to support coreos in > general. If yes the then how you see where Mesos fits in with Fleet, > Kubernetes. > > My current understanding is that Fleet is useful for lightweight > scheduling, where as Mesos and kubernetes are kind of serving the > similar purpose. Mesos has been here for a while and more feature > complete than kubernetes. But Kubernetes has more tight integeration > with coreos like use etcd for co-ordination, flannel for networking. I > wonder what's the plan are when it comes to Mesos for such. I have seen > the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806) > > I understand that the landscape is changing fast but its good to know > about Mesos roadmap in this regard. Also would love to know if anybody > using Coreos with Mesos beyond testing. > > Thanks, > Gurvinder > On 03/09/2015 11:35 PM, Michael Park wrote: > > Hi Gurvinder, > > > > We got started on this work at Mesosphere and there's a tutorial > > <http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/> > > on > > how to do a single-node setup. We ran the mesos-master and slaves in > > docker containers which led to this JIRA ticket > > <https://issues.apache.org/jira/browse/MESOS-2115>. I haven't been able > > to follow-up on this article recently, and I'd like to hear about others > > who have made further progress as well. > > > > At the time, we were thinking that using fleet shouldn't be too > > difficult since it uses the systemd unit files but didn't quite get > > around to it. > > > > Perhaps you'll find the tutorial to be a decent starting point. > > > > Thanks, > > > > MPark. > > > > On 9 March 2015 at 17:52, Gurvinder Singh <gurvinder.si...@uninett.no > > (mailto:gurvinder.si...@uninett.no) > > <mailto:gurvinder.si...@uninett.no>> wrote: > > > > Hi, > > > > I am wondering if anybody in the community has looked into or are > > running mesos on top of coreos. I would be interested to hear out your > > experiences around following areas > > > > - Users management on coreos cluster and containers running with Mesos > > - Are you using fleet to run mesos or run it as service in > > cloud-config > > and don't use fleet at all > > - Networking among hosts flannel or ? > > - Any other interesting insights you found considering such setup > > > > Thanks, > > Gurvinder > > > > >
