Re: mesos on coreos
Thanks Alex for the information and others too for sharing their experiences. - Gurvinder On 03/11/2015 07:50 PM, Alex Rukletsov wrote: Gurvinder, no, there are no publicly available binaries, neither is documentation at this point. We will publish either or both as soon as it is rock solid. On Wed, Mar 11, 2015 at 2:08 AM, Gurvinder Singh gurvinder.si...@uninett.no mailto:gurvinder.si...@uninett.no wrote: On 03/10/2015 11:41 PM, Tim Chen wrote: Hi all, As Alex said you can run Mesos in CoreOS without Docker if you put in the dependencies in. Tim, is there any documentation of using Mesos outside container in CoreOS available or binary available which we can wget in cloud-init file to fulfill dependencies. As we would like to test it out Mesos on CoreOS outside docker. - Gurvinder It is a common ask though to run Mesos-slave in a Docker container in general, either on CoreOS or not. It's definitely a bit involved as you need to mount in a directory for persisting work dir and also mounting in /sys/fs for cgroups, also you should use the --pid=host flag since Docker 1.5 so it shares the host pid namespace. Although you get a lot less isolation, there are still motivations to run slave in Docker regardless. One thing that's missing from the mesos docker containerizer is that it won't be able to recover tasks on restart, and I have a series of patches pending review to fix that. Tim On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov a...@mesosphere.io mailto:a...@mesosphere.io mailto:a...@mesosphere.io mailto:a...@mesosphere.io wrote: My 2¢. First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. You can run mesos-slave on CoreOS node without putting it into a Docker container. —Alex
Re: mesos on coreos
On 03/10/2015 11:41 PM, Tim Chen wrote: Hi all, As Alex said you can run Mesos in CoreOS without Docker if you put in the dependencies in. Tim, is there any documentation of using Mesos outside container in CoreOS available or binary available which we can wget in cloud-init file to fulfill dependencies. As we would like to test it out Mesos on CoreOS outside docker. - Gurvinder It is a common ask though to run Mesos-slave in a Docker container in general, either on CoreOS or not. It's definitely a bit involved as you need to mount in a directory for persisting work dir and also mounting in /sys/fs for cgroups, also you should use the --pid=host flag since Docker 1.5 so it shares the host pid namespace. Although you get a lot less isolation, there are still motivations to run slave in Docker regardless. One thing that's missing from the mesos docker containerizer is that it won't be able to recover tasks on restart, and I have a series of patches pending review to fix that. Tim On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov a...@mesosphere.io mailto:a...@mesosphere.io wrote: My 2¢. First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. You can run mesos-slave on CoreOS node without putting it into a Docker container. —Alex
Re: mesos on coreos
Some current issues are listed under https://issues.apache.org/jira/browse/MESOS-2115 See also previous email discussions: http://www.mail-archive.com/user%40mesos.apache.org/msg02123.html http://www.mail-archive.com/user%40mesos.apache.org/msg01617.html On Tue, Mar 10, 2015 at 4:58 PM, craig w codecr...@gmail.com wrote: Is there any documentation describing what's necessary to run mesos master and slaves in Docker containers? You already mentioned a few things (mounting work dir, /sys/fs, etc). Thanks On Tue, Mar 10, 2015 at 6:41 PM, Tim Chen t...@mesosphere.io wrote: Hi all, As Alex said you can run Mesos in CoreOS without Docker if you put in the dependencies in. It is a common ask though to run Mesos-slave in a Docker container in general, either on CoreOS or not. It's definitely a bit involved as you need to mount in a directory for persisting work dir and also mounting in /sys/fs for cgroups, also you should use the --pid=host flag since Docker 1.5 so it shares the host pid namespace. Although you get a lot less isolation, there are still motivations to run slave in Docker regardless. One thing that's missing from the mesos docker containerizer is that it won't be able to recover tasks on restart, and I have a series of patches pending review to fix that. Tim On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov a...@mesosphere.io wrote: My 2¢. First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. You can run mesos-slave on CoreOS node without putting it into a Docker container. —Alex -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: mesos on coreos
Hi Micheal, Yes I tested the tutorial and it works fine for testing. Later on I used fleet to run mesos workers on all coreos machines too. I was wondering how the landscape is looking in the community regarding coreos. As is there any interest from community or mesos team to support coreos in general. If yes the then how you see where Mesos fits in with Fleet, Kubernetes. My current understanding is that Fleet is useful for lightweight scheduling, where as Mesos and kubernetes are kind of serving the similar purpose. Mesos has been here for a while and more feature complete than kubernetes. But Kubernetes has more tight integeration with coreos like use etcd for co-ordination, flannel for networking. I wonder what's the plan are when it comes to Mesos for such. I have seen the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806) I understand that the landscape is changing fast but its good to know about Mesos roadmap in this regard. Also would love to know if anybody using Coreos with Mesos beyond testing. Thanks, Gurvinder On 03/09/2015 11:35 PM, Michael Park wrote: Hi Gurvinder, We got started on this work at Mesosphere and there's a tutorial http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/ on how to do a single-node setup. We ran the mesos-master and slaves in docker containers which led to this JIRA ticket https://issues.apache.org/jira/browse/MESOS-2115. I haven't been able to follow-up on this article recently, and I'd like to hear about others who have made further progress as well. At the time, we were thinking that using fleet shouldn't be too difficult since it uses the systemd unit files but didn't quite get around to it. Perhaps you'll find the tutorial to be a decent starting point. Thanks, MPark. On 9 March 2015 at 17:52, Gurvinder Singh gurvinder.si...@uninett.no mailto:gurvinder.si...@uninett.no wrote: Hi, I am wondering if anybody in the community has looked into or are running mesos on top of coreos. I would be interested to hear out your experiences around following areas - Users management on coreos cluster and containers running with Mesos - Are you using fleet to run mesos or run it as service in cloud-config and don't use fleet at all - Networking among hosts flannel or ? - Any other interesting insights you found considering such setup Thanks, Gurvinder
Re: mesos on coreos
Thanks Anton for sharing your experience. Response in line. On 03/10/2015 01:01 PM, Anton Kirillov wrote: Hi Gurvinder, our team have an experience with Mesos on CoreOS with fleet, and we decided to switch to bare metal deployments and here are our main reasons. First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. I agree. That was the main reason I asked about closer integeration of mesos with coreos. If you look here kubernetes (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml) run natively in coreos, not in a container. As it is started by cloud-init process. So something similar for mesos will resolve this issue. Although kubernetes is possible due to simple go binary with no dependency, I looked at the Mesos library dependency and compare with library on coreos. Only 2 are missing (libmesos-version.so, libsasl2.so). So I think it is possible for mesos to follow the same model as kubernetes to run natively. From DevOps point of view it is hard to control such thing as ZooKeeper restarts (and ensemble rolling restarts as well) which is the core service discovery mechanism for Mesos. You have to put some sidekick services to provide peer discovery mechanics and it doesn’t look very robust. That's why I mention about the JIRA where mesos can use etcd. The sidekick could be solved by using the flannel (https://github.com/coreos/flannel) to make each container addressable inside your cluster. Very common use case with Mesos is running Docker on top of it either with Marathon or with Aurora. But Docker service needs to be installed on worker nodes. So you’re coming to Docker-into-Docker situation which cancels all advantages of both transparent resource management and simple deployment configuration. One more point on Mesos inside Docker here, is that you have to attach Mesos data directories from container to a host. Given that you’re already running Mesos container in privileged mode and sharing directories with state with host there is no more reasons to run Mesos inside a container. And consider container restart (not just failure) with registry corruption and following synchronization issues. Another our use case with multi-region cluster deployments showed some issues with etcd heartbeat/leader election timeouts, which need to be increased in order to handle bigger latencies between data centers. If timeouts increase fleet starts to work in unpredictable way, loosing and finding peer nodes again which is not appropriate in production environment. I have not experience with multi region deployment. As such scenario for zookeeper can also be hard, where as consul claims to address this issue. The reason for asking is that with coreos we have a small footprint, upto date OS which can boot the mesos to manage whole cluster. By using docker, we can have multi tenancy support too. Just ideas :P - Gurvinder You can take a look at this configuration for Mesos-CoreOS-HA as well https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha -- Anton Kirillov Sent with Sparrow http://www.sparrowmailapp.com/?sig On Tuesday, March 10, 2015 at 11:08 AM, Gurvinder Singh wrote: Hi Micheal, Yes I tested the tutorial and it works fine for testing. Later on I used fleet to run mesos workers on all coreos machines too. I was wondering how the landscape is looking in the community regarding coreos. As is there any interest from community or mesos team to support coreos in general. If yes the then how you see where Mesos fits in with Fleet, Kubernetes. My current understanding is that Fleet is useful for lightweight scheduling, where as Mesos and kubernetes are kind of serving the similar purpose. Mesos has been here for a while and more feature complete than kubernetes. But Kubernetes has more tight integeration with coreos like use etcd for co-ordination, flannel for networking. I wonder what's the plan are when it comes to Mesos for such. I have seen the JIRA for etcd (https://issues.apache.org/jira/browse/MESOS-1806) I understand that the landscape is changing fast but its good to know about Mesos roadmap in this regard. Also would love to know if anybody using Coreos with Mesos beyond testing. Thanks, Gurvinder On 03/09/2015 11:35 PM, Michael Park wrote: Hi Gurvinder, We got started on this work at Mesosphere and there's a tutorial http://mesosphere.com/docs/tutorials/mesosphere-on-a-single-coreos-instance/ on how to do a single-node setup. We ran the mesos-master and slaves in docker containers which led to this JIRA ticket https://issues.apache.org/jira/browse/MESOS-2115. I haven't been able to follow-up on this article recently, and I'd like to hear about others who have made further progress as well. At the time, we were
Re: mesos on coreos
On 03/10/2015 03:57 PM, Anton Kirillov wrote: Gurvinder, your points are really interesting to consider, but as for me it is still looks pretty like a bit narrow solution, because not all widespread systems are having frameworks to run on Mesos. But it really depends on your goals. We have pretty specific use cases, one of them is using Spark on Mesos to achieve HA alongside with Cassandra as datastore. So we install Mesos Slave with Cassandra node on the same machine to achieve greater data locality. And Docker is pretty poor choice for running Cassandra in it but there is no other way to run Cassandra on CoreOS (afaik) We also plan to run different frameworks on our cluster and Spark, Cassandra being one of them. I would like to know what are the issues you faced while running cassandra in docker. As docker with volume attached has almost bare metal performance. Another idea is that when you go to “big iron” OS footprint doesn’t matter a lot while you have multi-core and huge RAM hardware. It looks like premature optimization. Optimization is one thing, but with CoreOS you get more determinstic updates of OS with rollback option which can be quite useful when running large infrastructure. The current discussion is to get to know about community feeling about these ideas, It is not to say that it's the best solution :) - Gurvinder My points just come from recent experience and more problem-oriented. But it would be really nice to see Mesos as native CoreOS service to experiment with. -- Anton Kirillov Sent with Sparrow http://www.sparrowmailapp.com/?sig On Tuesday, March 10, 2015 at 3:12 PM, Gurvinder Singh wrote: Thanks Anton for sharing your experience. Response in line. On 03/10/2015 01:01 PM, Anton Kirillov wrote: Hi Gurvinder, our team have an experience with Mesos on CoreOS with fleet, and we decided to switch to bare metal deployments and here are our main reasons. First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. I agree. That was the main reason I asked about closer integeration of mesos with coreos. If you look here kubernetes (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/coreos/cloud-configs/master.yaml) run natively in coreos, not in a container. As it is started by cloud-init process. So something similar for mesos will resolve this issue. Although kubernetes is possible due to simple go binary with no dependency, I looked at the Mesos library dependency and compare with library on coreos. Only 2 are missing (libmesos-version.so, libsasl2.so). So I think it is possible for mesos to follow the same model as kubernetes to run natively. From DevOps point of view it is hard to control such thing as ZooKeeper restarts (and ensemble rolling restarts as well) which is the core service discovery mechanism for Mesos. You have to put some sidekick services to provide peer discovery mechanics and it doesn’t look very robust. That's why I mention about the JIRA where mesos can use etcd. The sidekick could be solved by using the flannel (https://github.com/coreos/flannel) to make each container addressable inside your cluster. Very common use case with Mesos is running Docker on top of it either with Marathon or with Aurora. But Docker service needs to be installed on worker nodes. So you’re coming to Docker-into-Docker situation which cancels all advantages of both transparent resource management and simple deployment configuration. One more point on Mesos inside Docker here, is that you have to attach Mesos data directories from container to a host. Given that you’re already running Mesos container in privileged mode and sharing directories with state with host there is no more reasons to run Mesos inside a container. And consider container restart (not just failure) with registry corruption and following synchronization issues. Another our use case with multi-region cluster deployments showed some issues with etcd heartbeat/leader election timeouts, which need to be increased in order to handle bigger latencies between data centers. If timeouts increase fleet starts to work in unpredictable way, loosing and finding peer nodes again which is not appropriate in production environment. I have not experience with multi region deployment. As such scenario for zookeeper can also be hard, where as consul claims to address this issue. The reason for asking is that with coreos we have a small footprint, upto date OS which can boot the mesos to manage whole cluster. By using docker, we can have multi tenancy support too. Just ideas :P - Gurvinder You can take a look at this configuration for Mesos-CoreOS-HA as well https://github.com/akirillov/mesos-deploy/tree/master/mesos-coreos-ha -- Anton Kirillov Sent with Sparrow
Re: mesos on coreos
Hi all, As Alex said you can run Mesos in CoreOS without Docker if you put in the dependencies in. It is a common ask though to run Mesos-slave in a Docker container in general, either on CoreOS or not. It's definitely a bit involved as you need to mount in a directory for persisting work dir and also mounting in /sys/fs for cgroups, also you should use the --pid=host flag since Docker 1.5 so it shares the host pid namespace. Although you get a lot less isolation, there are still motivations to run slave in Docker regardless. One thing that's missing from the mesos docker containerizer is that it won't be able to recover tasks on restart, and I have a series of patches pending review to fix that. Tim On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov a...@mesosphere.io wrote: My 2¢. First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. You can run mesos-slave on CoreOS node without putting it into a Docker container. —Alex
Re: mesos on coreos
Is there any documentation describing what's necessary to run mesos master and slaves in Docker containers? You already mentioned a few things (mounting work dir, /sys/fs, etc). Thanks On Tue, Mar 10, 2015 at 6:41 PM, Tim Chen t...@mesosphere.io wrote: Hi all, As Alex said you can run Mesos in CoreOS without Docker if you put in the dependencies in. It is a common ask though to run Mesos-slave in a Docker container in general, either on CoreOS or not. It's definitely a bit involved as you need to mount in a directory for persisting work dir and also mounting in /sys/fs for cgroups, also you should use the --pid=host flag since Docker 1.5 so it shares the host pid namespace. Although you get a lot less isolation, there are still motivations to run slave in Docker regardless. One thing that's missing from the mesos docker containerizer is that it won't be able to recover tasks on restart, and I have a series of patches pending review to fix that. Tim On Tue, Mar 10, 2015 at 3:16 PM, Alex Rukletsov a...@mesosphere.io wrote: My 2¢. First of all, it doesn’t look like a great idea to package resource manager into Docker putting one more abstraction layer between a resource itself and resource manager. You can run mesos-slave on CoreOS node without putting it into a Docker container. —Alex -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: mesos and coreos?
Nope. First, mesos is not a framework. A framework is what you use in your application to help build the app itself, like spring, rails, or django. Mesos is more fundamental. - mesos gathers all the resources (cpus/mems/disks) of the nodes in your cluster and make it a resource pool - your app doesn't even know it's scheduled and managed (e.g. started/stopped) by mesos (to be exact, by any framework running on mesos, like marathon) So you can think mesos as an distributed operating system , just as mesosphere's slogan says. On Mon, Jan 19, 2015 at 6:27 AM, Victor L vlyamt...@gmail.com wrote: Does that mean mesos is framework to prepare my app to take advantage of clustering environment? On Sun, Jan 18, 2015 at 1:43 PM, Tom Arnfeld t...@duedil.com wrote: The way I see it, Mesos is an API and framework for building and running distributed systems. CoreOS is an API and framework for running them. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Sun, Jan 18, 2015 at 3:01 PM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks
Re: mesos and coreos?
On 01/18/2015 04:25 PM, Ranjib Dey wrote: you are right, OS is same , which is Linux kernerl, But the Ubuntu/CoreOS/Redhat etc distinction are in userspace (i.e tools other than the kernel), and hence you can have coreos running ubuntu/redhat containers. CoreOS is a gentoo knock_off [1,2.3] I have not explored CoreOS yet, but I have not found any reason why any of the common linux distros cannot run on top of CoreOS, including gentoo or even a mixture of different linux distros. You cant have container specific kernels, drivers, time susbsytem etc. But you can certainly have different distros (redhat, ubuntu etc are different distro, not OS). So if a minimal kernel is used with coreOS the distro inside of a particular container cannot have different loadable modules inside of different containers? CoreOS eases management of container , an immutable minimal rootfs , backed by tools (etcd, systemd, fleet, flannel) etc that facilitates building large scale systems. For example, etcd is almost a replacement of zookeeper (you can use it for leader election, distributed locks etc). Fleet is a distributed init system. I thought CoreOS used Systemd? [4] I really which CoreOS was using openrc, and systemd could be used inside of selective containers with different linux distros. CoreOS does not provide a sceduler, which mesos does. Also coreos is not a resource scheduling system, which mesos is. You have containarize things to run on coreos (currently its docker, i think it will rocket in future). While thats not a mandate for mesos. Neither CoreOS not Mesos gives you `distributed systems`, you can distribute your workload using mesos or coreos (mesos will autoschedule things for you). Generally the word `distributed systems` used to describe things like zookeeper, etcd, cassandra, riak, serf etc, where the members are aware of each other, without any external components. Most of them also uses sound theoretical foundations like paxos, raft etc for attaining different types of consistency, partition tolerance etc. Mesos and CoreOS address orthogonal issues, and they can definitely complement each other. CoreOS eases updating kernel, manageing app deployments due to host OS and app separation. While mesos eases scaling and usage issues by autoscheduling. Mesos can use coreos for its containment layer (docker/rocket), as well as use etcd (from coreos) to do the leader election bit instead of zookeeper (which is pain to run over WAN, pain to dynamically resize etc). But there are major work involve. It will be interesting to see how all of these and other possibilities mature. What about mesos+spark on top of a coreOS infrastructure. Anyone has any experience with Apache_spark running on coreOS? regards ranjib James [1] https://coreos.com/docs/sdk-distributors/sdk/building-development-images/#updating-portage-stable-ebuilds-from-gentoo [2] https://github.com/coreos/coreos-overlay [3] https://github.com/coreos/portage-stable [4] https://coreos.com/using-coreos/systemd/
Re: mesos and coreos?
i thought coreos was an operating system and not an API On 18 January 2015 at 18:43, Tom Arnfeld t...@duedil.com wrote: The way I see it, Mesos is an API and framework for building and running distributed systems. CoreOS is an API and framework for running them. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Sun, Jan 18, 2015 at 3:01 PM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks
Re: mesos and coreos?
ok a distribution, with some stuff pre bundled... On 18 January 2015 at 19:15, scott@heroku sc...@heroku.com wrote: Afaik mesos is much more flexible than fleet, which is the scheduling system on Coreos If you can successfully schedule your workloads with fleet you don't need mesos. If not mesos can do more than fleet. Sent from my iPhone On Jan 18, 2015, at 10:29 AM, Victor L vlyamt...@gmail.com wrote: Hope this helps some It doesn't as it doesn't even try to answer my question. Let me re- phrase it: what does mesos on the coreos cluster do that coreos itself doesn't do already? On Sun, Jan 18, 2015 at 10:00 AM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks
Re: mesos and coreos?
CoreOS in and of itself does not try to compete directly with Mesos. Fleet vs Mesos is a much better comparison. The biggest difference there is that Mesos is battle proven at scale (100K+ node deployments running in PROD for 1+ yrs). Fleet is not proven at scale. HTH, JJ. On Jan 18, 2015, at 11:15 AM, scott@heroku sc...@heroku.com wrote: Afaik mesos is much more flexible than fleet, which is the scheduling system on Coreos If you can successfully schedule your workloads with fleet you don't need mesos. If not mesos can do more than fleet. Sent from my iPhone On Jan 18, 2015, at 10:29 AM, Victor L vlyamt...@gmail.com wrote: Hope this helps some It doesn't as it doesn't even try to answer my question. Let me re- phrase it: what does mesos on the coreos cluster do that coreos itself doesn't do already? On Sun, Jan 18, 2015 at 10:00 AM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks
Re: mesos and coreos?
The way I see it, Mesos is an API and framework for building and running distributed systems. CoreOS is an API and framework for running them. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Sun, Jan 18, 2015 at 3:01 PM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks
Re: mesos and coreos?
Coreos places focus on the OS to deploy services as containers. It’s distributed key store is meant to share config in a cluster and to aid in basic scheduling via fleet, which is like cluster wide systemd. It’s scheduler is basic (but can be made to be more complex if you were to use these base tools). On the other hand, Mesos has a more complex featureful scheduler, works as-an application, and has more first class controls over managing jobs (cgroups, etc…) There is not complete overlap between these two systems. They do not necessarily compete with each other. But they do have features which try to address distributed application design/deployment. - J On Jan 18, 2015, at 1:29 PM, Victor L vlyamt...@gmail.com wrote: Hope this helps some It doesn't as it doesn't even try to answer my question. Let me re- phrase it: what does mesos on the coreos cluster do that coreos itself doesn't do already? On Sun, Jan 18, 2015 at 10:00 AM, Jason Giedymin jason.giedy...@gmail.com mailto:jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com mailto:vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks
Re: mesos and coreos?
I think CoreOS provides a good single node OS for executing containers and Fleet provides very simple scheduling and placement and etcd provides discovery primitives. I think Mesos besides being ore proven to scale and handle failure scenarios, it also provides more primitives for users to write Mesos frameworks that can provide more information and events for applications to be smarter about how it wants to react to these. Mesos also provides more isolation choices, more statistics available, and also provides a community and existing frameworks that all users can leverage already. Tim On Jan 18, 2015, at 2:27 PM, Victor L vlyamt...@gmail.com wrote: Does that mean mesos is framework to prepare my app to take advantage of clustering environment? On Sun, Jan 18, 2015 at 1:43 PM, Tom Arnfeld t...@duedil.com wrote: The way I see it, Mesos is an API and framework for building and running distributed systems. CoreOS is an API and framework for running them. -- Tom Arnfeld Developer // DueDil (+44) 7525940046 25 Christopher Street, London, EC2A 2BS On Sun, Jan 18, 2015 at 3:01 PM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks
Re: mesos and coreos?
One other thing I'd like to point out, many people say CoreOS is great because it autoupdates on its own, but you need to realize that the containers that run on top of CoreOS don't run coreos, they run Ubuntu, Fedora, etc, and if there is a security issue (think openssl, etc), you have to rebuild all your containers again to apply the missing updates. Thanks On Sun, Jan 18, 2015 at 2:28 PM, Jason Giedymin jason.giedy...@gmail.com wrote: Coreos places focus on the OS to deploy services as containers. It’s distributed key store is meant to share config in a cluster and to aid in basic scheduling via fleet, which is like cluster wide systemd. It’s scheduler is basic (but can be made to be more complex if you were to use these base tools). On the other hand, Mesos has a more complex featureful scheduler, works as-an application, and has more first class controls over managing jobs (cgroups, etc…) There is not complete overlap between these two systems. They do not necessarily compete with each other. But they do have features which try to address distributed application design/deployment. - J On Jan 18, 2015, at 1:29 PM, Victor L vlyamt...@gmail.com wrote: Hope this helps some It doesn't as it doesn't even try to answer my question. Let me re- phrase it: what does mesos on the coreos cluster do that coreos itself doesn't do already? On Sun, Jan 18, 2015 at 10:00 AM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks -- Diego Medina Lift/Scala consultant di...@fmpwizard.com http://fmpwizard.telegr.am
Re: mesos and coreos?
Not the case. I can run ubuntu containers on a micro linux install with docker. The difference is that they all have the same kernel no matter what distro your container is using On Jan 18, 2015 4:09 PM, Michael Dilworth m...@computer.org wrote: a bit of an aside, but i am under the impression that containers are not another OS, but the same as the underlying host. So you cant have an Ubuntu container on a CoreOS host.. unless you use a hypervisor.. the container is coreos too. mike On 18 January 2015 at 20:56, Diego Medina di...@fmpwizard.com wrote: One other thing I'd like to point out, many people say CoreOS is great because it autoupdates on its own, but you need to realize that the containers that run on top of CoreOS don't run coreos, they run Ubuntu, Fedora, etc, and if there is a security issue (think openssl, etc), you have to rebuild all your containers again to apply the missing updates. Thanks On Sun, Jan 18, 2015 at 2:28 PM, Jason Giedymin jason.giedy...@gmail.com wrote: Coreos places focus on the OS to deploy services as containers. It’s distributed key store is meant to share config in a cluster and to aid in basic scheduling via fleet, which is like cluster wide systemd. It’s scheduler is basic (but can be made to be more complex if you were to use these base tools). On the other hand, Mesos has a more complex featureful scheduler, works as-an application, and has more first class controls over managing jobs (cgroups, etc…) There is not complete overlap between these two systems. They do not necessarily compete with each other. But they do have features which try to address distributed application design/deployment. - J On Jan 18, 2015, at 1:29 PM, Victor L vlyamt...@gmail.com wrote: Hope this helps some It doesn't as it doesn't even try to answer my question. Let me re- phrase it: what does mesos on the coreos cluster do that coreos itself doesn't do already? On Sun, Jan 18, 2015 at 10:00 AM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks -- Diego Medina Lift/Scala consultant di...@fmpwizard.com http://fmpwizard.telegr.am
Re: mesos and coreos?
On Sun, Jan 18, 2015 at 4:08 PM, Michael Dilworth m...@computer.org wrote: a bit of an aside, but i am under the impression that containers are not another OS, but the same as the underlying host. So you cant have an Ubuntu container on a CoreOS host.. unless you use a hypervisor.. the container is coreos too. Hi, sorry for going off topic. I also thought the same, that a base coreos server would have coreos containers, but that's not the case. Coreos prides itself for being a minimum server OS, for example, there is no running java or ruby or python directly on coreos, to run any app in java, you need a container that has the jvm on it. This link gives you more info http://stackoverflow.com/questions/18786209/what-is-the-relationship-between-the-docker-host-os-and-the-container-base-image there they talk about having a host ubuntu and docker containers that can be fedore, they just share the kernel version PS, I tried this myself to make sure, I had https://github.com/fmpwizard/coreosdemo which is based on the golang img which is based on debian https://github.com/docker-library/golang/blob/c1baf037d71331eb0b8d4c70cff4c29cf124c5e0/1.4/wheezy/Dockerfile and it run just fine on a coreos cluster on digita ocean Thanks mike On 18 January 2015 at 20:56, Diego Medina di...@fmpwizard.com wrote: One other thing I'd like to point out, many people say CoreOS is great because it autoupdates on its own, but you need to realize that the containers that run on top of CoreOS don't run coreos, they run Ubuntu, Fedora, etc, and if there is a security issue (think openssl, etc), you have to rebuild all your containers again to apply the missing updates. Thanks On Sun, Jan 18, 2015 at 2:28 PM, Jason Giedymin jason.giedy...@gmail.com wrote: Coreos places focus on the OS to deploy services as containers. It’s distributed key store is meant to share config in a cluster and to aid in basic scheduling via fleet, which is like cluster wide systemd. It’s scheduler is basic (but can be made to be more complex if you were to use these base tools). On the other hand, Mesos has a more complex featureful scheduler, works as-an application, and has more first class controls over managing jobs (cgroups, etc…) There is not complete overlap between these two systems. They do not necessarily compete with each other. But they do have features which try to address distributed application design/deployment. - J On Jan 18, 2015, at 1:29 PM, Victor L vlyamt...@gmail.com wrote: Hope this helps some It doesn't as it doesn't even try to answer my question. Let me re- phrase it: what does mesos on the coreos cluster do that coreos itself doesn't do already? On Sun, Jan 18, 2015 at 10:00 AM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks -- Diego Medina Lift/Scala consultant di...@fmpwizard.com http://fmpwizard.telegr.am -- Diego Medina Lift/Scala consultant di...@fmpwizard.com http://fmpwizard.telegr.am
Re: mesos and coreos?
you are right, OS is same , which is Linux kernerl, But the Ubuntu/CoreOS/Redhat etc distinction are in userspace (i.e tools other than the kernel), and hence you can have coreos running ubuntu/redhat containers. You cant have container specific kernels, drivers, time susbsytem etc. But you can certainly have different distros (redhat, ubuntu etc are different distro, not OS). CoreOS eases management of container , an immutable minimal rootfs , backed by tools (etcd, systemd, fleet, flannel) etc that facilitates building large scale systems. For example, etcd is almost a replacement of zookeeper (you can use it for leader election, distributed locks etc). Fleet is a distributed init system. CoreOS does not provide a sceduler, which mesos does. Also coreos is not a resource scheduling system, which mesos is. You have containarize things to run on coreos (currently its docker, i think it will rocket in future). While thats not a mandate for mesos. Neither CoreOS not Mesos gives you `distributed systems`, you can distribute your workload using mesos or coreos (mesos will autoschedule things for you). Generally the word `distributed systems` used to describe things like zookeeper, etcd, cassandra, riak, serf etc, where the members are aware of each other, without any external components. Most of them also uses sound theoretical foundations like paxos, raft etc for attaining different types of consistency, partition tolerance etc. Mesos and CoreOS address orthogonal issues, and they can definitely complement each other. CoreOS eases updating kernel, manageing app deployments due to host OS and app separation. While mesos eases scaling and usage issues by autoscheduling. Mesos can use coreos for its containment layer (docker/rocket), as well as use etcd (from coreos) to do the leader election bit instead of zookeeper (which is pain to run over WAN, pain to dynamically resize etc). But there are major work involve. regards ranjib
Re: mesos and coreos?
some excellent posts clearing it all up, much appreciated :) Mike On 18 January 2015 at 21:28, Brian Devins badev...@gmail.com wrote: Not the case. I can run ubuntu containers on a micro linux install with docker. The difference is that they all have the same kernel no matter what distro your container is using On Jan 18, 2015 4:09 PM, Michael Dilworth m...@computer.org wrote: a bit of an aside, but i am under the impression that containers are not another OS, but the same as the underlying host. So you cant have an Ubuntu container on a CoreOS host.. unless you use a hypervisor.. the container is coreos too. mike On 18 January 2015 at 20:56, Diego Medina di...@fmpwizard.com wrote: One other thing I'd like to point out, many people say CoreOS is great because it autoupdates on its own, but you need to realize that the containers that run on top of CoreOS don't run coreos, they run Ubuntu, Fedora, etc, and if there is a security issue (think openssl, etc), you have to rebuild all your containers again to apply the missing updates. Thanks On Sun, Jan 18, 2015 at 2:28 PM, Jason Giedymin jason.giedy...@gmail.com wrote: Coreos places focus on the OS to deploy services as containers. It’s distributed key store is meant to share config in a cluster and to aid in basic scheduling via fleet, which is like cluster wide systemd. It’s scheduler is basic (but can be made to be more complex if you were to use these base tools). On the other hand, Mesos has a more complex featureful scheduler, works as-an application, and has more first class controls over managing jobs (cgroups, etc…) There is not complete overlap between these two systems. They do not necessarily compete with each other. But they do have features which try to address distributed application design/deployment. - J On Jan 18, 2015, at 1:29 PM, Victor L vlyamt...@gmail.com wrote: Hope this helps some It doesn't as it doesn't even try to answer my question. Let me re- phrase it: what does mesos on the coreos cluster do that coreos itself doesn't do already? On Sun, Jan 18, 2015 at 10:00 AM, Jason Giedymin jason.giedy...@gmail.com wrote: The value of coreos that immediately comes to mind since I do much work with these tools: - the small foot print, it is a minimal os, meant to run containers. So it throws everything not needed for that out. - containers are the launch vehicle, thus deps are in container land. I can run and test containers with ease, not having to worry about multiple OSes. - with etcd and fleet, coordinating the launch and modification of both machines and cluster make it a breeze. Allowing you to do dynamic mesos scaling up or down. I add nodes at will, across multiple cloud platforms, ready to launch multitude of containers or just mesos. - security. There is a defined write strategy. You cannot write willy nilly to any location. - all the above further allow auto OS updates, which is supported today on all platforms that deploy coreos. This means more frequent updates since the os is minimal, which should increase the security effectiveness when compared to big box superstore OSes like Redhat or Ubuntu. Some platforms charge quite a bit for managed updates of this frequency and level of testing. Coreos allows me to keep apps in a configured container that I trust, tested, and works time and time again. I see coreos as a compliment. As a fyi I'm available for questions, debugging, and client work in this area. Hope this helps some, from real world usage. Sent from my iPad On Jan 18, 2015, at 9:16 AM, Victor L vlyamt...@gmail.com wrote: I am confused: what's the value of mesos on the top of coreos cluster? Mesos provides distributed resource management, fault tolerance, etc., but doesn't coreos provides the same things already? Thanks -- Diego Medina Lift/Scala consultant di...@fmpwizard.com http://fmpwizard.telegr.am
