Systemd After=network.target vs. After=network-online.target
Hello, I have used Mesosphere Zookeeper RPM and it uses After=network.target in its zookeeper.service systemd unit file. This doesn't guarantee that Zookeeper service will start after network is available. To ensure this After=network-online.target should be used. I spent some time with this when Zookeeper was disconnected after reboots because of delays during reboot on my VMs. If Centos starts up fast, which it typically does on a clean install, it doesn't manifest as a problem. Actually I have experienced it on RHEL with some custom company layer on top which introduced these delays. I would recommend to change it to avoid some bad user experience with slow VMs. As well I think it is better for production. Or are there any reasons why not? I used mesosphere-zookeeper.x86_64 3.4.6-0.1.20141204175332.centos7 version. Regards, Petr
RE: Can I consider other framework tasks as a resource? Does it make sense?
It is very helpful. I will take a deeper look on Fenzo. Isn’t pretty much everything external knowledge to a scheduler? CPU, mem, net, storage… these all information has to somehow get into scheduler. But for these there is an internal support by Mesos via resource offers and it is what I think you mean by internal vs. external. What I’m thinking is that there is already a mechanism in Mesos how to get information into scheduler but it is not extendable by custom resource types. Thinking about offered resources I have also realized that there is a common trait to them – they are consumable. When one task accepts some resources they are not available to other tasks. Hence probably if I would like to represent other constraints as resources they would have to have this property. Then, in theory, they could by plugged into Mesos resources mechanism. Possibly not all constraints can be modelled as consumables and the approach through pluggable scheduling library like Fenzo might be more flexible. My original question was basically about what counts as scheduling so that when I need to model some constraint how to place a task I would know where it belongs in my framework’s code. It seems to be answered. Thanks a lot. From: Sharma Podila [mailto:spod...@netflix.com] Sent: 15. prosince 2016 1:59 To: user@mesos.apache.org Subject: Re: Can I consider other framework tasks as a resource? Does it make sense? In general, placing a task based on certain constraints (e.g., locality with other tasks) is a scheduling concern. The complexity in your scenario is that the constraint specification requires knowledge external to your scheduler. If you are able to route that external information (on what and where other frameworks' tasks are running) into your scheduler, then, you should be able to achieve the locality constraints in your scheduler. If your scheduler happens to be running on the JVM, our open source Fenzo scheduling library can be useful. Or at least provide one idea on how your could write a scheduler that deals with such constraints. In Fenzo, for example, you'd write a custom plugin to handle the locality by using the external information, that I refer to above, to "score" agents that fit your task better. Fenzo will then pick the best agent to launch your task for locality. One limitation is the fact that you'd have little to no control on ensuring that the agents on which those other frameworks' tasks are running on will have additional resources available to fit your tasks. And that offers from those agents will arrive at your scheduler. Some variation of "delay scheduling" can help the latter by rejecting offers from agents that do not contain the tasks of interest from other frameworks. On Wed, Dec 14, 2016 at 10:33 AM, Petr Novak wrote: Thanks a lot for the input. “Y scheduler can accept a rule how to check readiness on startup” Based on it seems like +1 that I can consider it as a responsibility of a scheduler. Cheers, Petr From: Alex Rukletsov [mailto:a...@mesosphere.com] Sent: 14. prosince 2016 13:01 To: user Subject: Re: Can I consider other framework tasks as a resource? Does it make sense? Task dependency is probably too vague to discuss specifically. Mesos currently does not explicitly support arbitrary task dependencies. You mentioned colocation, one type of dependency, so let's look at it. If I understood you correctly, you would like to colocate a task from framework B to the same node where a task from framework A is running. The first problem is to get a list of such nodes (and keep them updated, because task may crash, migrate and so on). This can be done, say, by using Mesos DNS or alike. The second problem is to ensure that framework gets enough resources from that nodes. A possible solution here is to put both frameworks A and B into the same role and use dynamic reservations to ensure enough resources are laid away for both tasks. Disadvantages: you should know about all dependencies upfront, frameworks should be in the same role. Now the question is, why would you need to colocate workloads? I would say this is something you should avoid if possible, like any extra constraint that complicate the system. Probably the only 100% legitimate use case for colocation is data locality. Solving this particular problem seems easier than to address arbitrary task dependencies. If all you try to achieve is making sure a specific service represented by a framework X is running and ready in the cluster, you can do that by running specific checks before starting a depending framework Y or launching a new task in this framework. If your question is about whether Y should know about X and know how to check readiness of X in the cluster, I'd say you'd better keep that abstracted: Y scheduler can accept a rule how to check
RE: Multi-agent machine
The use case? Development setup on a single VitrualBox VM to simulate a cluster? To safe laptop resources? Or something more behind? From: Charles Allen [mailto:charles.al...@metamarkets.com] Sent: 9. prosince 2016 21:11 To: user Subject: Multi-agent machine Is it possible to setup a machine such that multiple mesos agents are running on the same machine and registering with the same master? For example, with different cgroup roots or different default working directory.
RE: Can I consider other framework tasks as a resource? Does it make sense?
Thanks a lot for the input. “Y scheduler can accept a rule how to check readiness on startup” Based on it seems like +1 that I can consider it as a responsibility of a scheduler. Cheers, Petr From: Alex Rukletsov [mailto:a...@mesosphere.com] Sent: 14. prosince 2016 13:01 To: user Subject: Re: Can I consider other framework tasks as a resource? Does it make sense? Task dependency is probably too vague to discuss specifically. Mesos currently does not explicitly support arbitrary task dependencies. You mentioned colocation, one type of dependency, so let's look at it. If I understood you correctly, you would like to colocate a task from framework B to the same node where a task from framework A is running. The first problem is to get a list of such nodes (and keep them updated, because task may crash, migrate and so on). This can be done, say, by using Mesos DNS or alike. The second problem is to ensure that framework gets enough resources from that nodes. A possible solution here is to put both frameworks A and B into the same role and use dynamic reservations to ensure enough resources are laid away for both tasks. Disadvantages: you should know about all dependencies upfront, frameworks should be in the same role. Now the question is, why would you need to colocate workloads? I would say this is something you should avoid if possible, like any extra constraint that complicate the system. Probably the only 100% legitimate use case for colocation is data locality. Solving this particular problem seems easier than to address arbitrary task dependencies. If all you try to achieve is making sure a specific service represented by a framework X is running and ready in the cluster, you can do that by running specific checks before starting a depending framework Y or launching a new task in this framework. If your question is about whether Y should know about X and know how to check readiness of X in the cluster, I'd say you'd better keep that abstracted: Y scheduler can accept a rule how to check readiness on startup. On Wed, Dec 14, 2016 at 5:14 AM, haosdent wrote: Hi, @Petr. > Like if I want to run my task collocated with some other tasks on the same > node I have to make this decision somewhere. Do you mean "POD" here? For my cases, if there are some dependencies between my tasks, I use database, message queue or zookeeper to implement my requirement. On Wed, Dec 14, 2016 at 3:09 AM, Petr Novak wrote: Hello, I want to execute tasks which requires some other tasks from other framework(s) already running. I’m thinking where such logic/strategy/policy belongs in principle. I understand scheduling as a process to decide where to execute task according to some resources availability, typically CPU, mem, net, hdd etc. If my task require other tasks running could I generalize and consider that those tasks from other frameworks are kind of required resources and put this logic/strategy decisions into scheduler? Like if I want to run my task collocated with some other tasks on the same node I have to make this decision somewhere. Does it make any sense? I’m asking because I have never thought about other frameworks/tasks as “resources” so that I could put them into scheduler to satisfy my understanding of a scheduler. Or it rather belongs higher like to a framework, or lower to an executor? Should scheduler be dedicated to decisions about resources which are offered and am I mixing concepts? Or I just should keep distinction between resources and requirements/policies or whatever but anyway does this kind of logic still belongs to scheduler or it should be somewhere else? I’m trying to understand which logic should be in scheduler and what should go somewhere else. Many thanks, Petr -- Best Regards, Haosdent Huang
RE: Can I consider other framework tasks as a resource? Does it make sense?
Hi, My question is conceptual about what is scheduler in a cluster-wide context and what counts as scheduler responsibility. The actual implementation is not that important but rather where you put that dependency logic, who decides if dependencies are satisfied. It is not about PODs because dependent services doesn’t have to run in the same pod. It seems to me that Mesos scheduler is defined like that it decides if and where a given task should be executed based on offered resources. In many application offered resources are possibly not enough to decide if to run a task. E.g. state of dependent services, licenses etc. I’m asking if there is any discussion, guidelines, best practices where these decisions belong when designing a framework. Can it still be a scheduler responsibility? Should it be in some other module, e.g. policy engine. I just try to understand where scheduler responsibilities start and end. If I would define scheduler as an entity deciding if and where (and how?) to execute a task based on state of resources and once decided the actual execution is delegated to an executor and I would like to consider all that logic above as a scheduler responsibility, then I have to either extend my definition of a scheduler or I have to be able to map everything as a resource. After all set of running services could be considered as a resource. REST is as well trying to represent everything as a resource. I’m probably trying to think too much, Petr From: haosdent [mailto:haosd...@gmail.com] Sent: 14. prosince 2016 5:14 To: user Subject: Re: Can I consider other framework tasks as a resource? Does it make sense? Hi, @Petr. > Like if I want to run my task collocated with some other tasks on the same > node I have to make this decision somewhere. Do you mean "POD" here? For my cases, if there are some dependencies between my tasks, I use database, message queue or zookeeper to implement my requirement. On Wed, Dec 14, 2016 at 3:09 AM, Petr Novak wrote: Hello, I want to execute tasks which requires some other tasks from other framework(s) already running. I’m thinking where such logic/strategy/policy belongs in principle. I understand scheduling as a process to decide where to execute task according to some resources availability, typically CPU, mem, net, hdd etc. If my task require other tasks running could I generalize and consider that those tasks from other frameworks are kind of required resources and put this logic/strategy decisions into scheduler? Like if I want to run my task collocated with some other tasks on the same node I have to make this decision somewhere. Does it make any sense? I’m asking because I have never thought about other frameworks/tasks as “resources” so that I could put them into scheduler to satisfy my understanding of a scheduler. Or it rather belongs higher like to a framework, or lower to an executor? Should scheduler be dedicated to decisions about resources which are offered and am I mixing concepts? Or I just should keep distinction between resources and requirements/policies or whatever but anyway does this kind of logic still belongs to scheduler or it should be somewhere else? I’m trying to understand which logic should be in scheduler and what should go somewhere else. Many thanks, Petr -- Best Regards, Haosdent Huang
Can I consider other framework tasks as a resource? Does it make sense?
Hello, I want to execute tasks which requires some other tasks from other framework(s) already running. I'm thinking where such logic/strategy/policy belongs in principle. I understand scheduling as a process to decide where to execute task according to some resources availability, typically CPU, mem, net, hdd etc. If my task require other tasks running could I generalize and consider that those tasks from other frameworks are kind of required resources and put this logic/strategy decisions into scheduler? Like if I want to run my task collocated with some other tasks on the same node I have to make this decision somewhere. Does it make any sense? I'm asking because I have never thought about other frameworks/tasks as "resources" so that I could put them into scheduler to satisfy my understanding of a scheduler. Or it rather belongs higher like to a framework, or lower to an executor? Should scheduler be dedicated to decisions about resources which are offered and am I mixing concepts? Or I just should keep distinction between resources and requirements/policies or whatever but anyway does this kind of logic still belongs to scheduler or it should be somewhere else? I'm trying to understand which logic should be in scheduler and what should go somewhere else. Many thanks, Petr
RE: Implementation examples for framework using V1 APIs for Scala/Java
I meant source code for the presentation. Petr From: Petr Novak [mailto:oss.mli...@gmail.com] Sent: 17. listopadu 2016 12:52 To: user@mesos.apache.org Subject: RE: Implementation examples for framework using V1 APIs for Scala/Java Hi, It is like what I have in mind. But I’m contemplating if akka-streams is an overkill for a framework. I would like to see a comparison implementation with/without akka-streams/RxJava. There is no source code on github for it, right? Many thanks, Petr From: Tomek Janiszewski [mailto:jani...@gmail.com] Sent: 13. listopadu 2016 11:38 To: user@mesos.apache.org Subject: Re: Implementation examples for framework using V1 APIs for Scala/Java Hi Here are slides from Dario Rexin "Writing a Mesos HTTP API Client" presentation http://schd.ws/hosted_files/mesosconeu2016/e6/mesoscon_eu_2016.pdf at MesosCon EU. Unfortunately event was not recorded. — Tomek niedz., 13.11.2016, 11:26 użytkownik Petr Novak napisał: Hello, Are there any examples/guides I can take a look? Many thanks, Petr
RE: Implementation examples for framework using V1 APIs for Scala/Java
Is there any Mesosphere framework in production build for v1 API as of yet? For some important service like Spark, Kafka, Cassandra and such. I would like to adopt Mesos in our company but I have never work with streamed chunked HTTP API before so I don't know how imperative Java/Scala code looks for it. If you have any example on hand I would be thankful. Is RxJava, akka-streams and alike required to work with v1 API conveniently? Is non-functional, imperative approach too cumbersome to handle? While I have some experience with Observables the rest of my team has none and it takes some time to switch the mindset. I have to evaluate for it. Regards, Petr From: Petr Novak [mailto:oss.mli...@gmail.com] Sent: 13. listopadu 2016 11:26 To: user@mesos.apache.org Subject: Implementation examples for framework using V1 APIs for Scala/Java Hello, Are there any examples/guides I can take a look? Many thanks, Petr
RE: Implementation examples for framework using V1 APIs for Scala/Java
Nice one to play with. Thanks. Is mesos-rxjava official attempt to replace the old Java lib (libmesos based) to interact with Mesos? Or is it just kind of non-stable experiment? Petr From: haosdent [mailto:haosd...@gmail.com] Sent: 14. listopadu 2016 14:28 To: user Subject: Re: Implementation examples for framework using V1 APIs for Scala/Java You may refer the example framework in mesos-rxjava https://github.com/mesosphere/mesos-rxjava/tree/master/mesos-rxjava-example/mesos-rxjava-example-framework as well. On Sun, Nov 13, 2016 at 10:36 PM, David Greenberg wrote: There's also a book (disclaimer: I am the author) about framework development. The examples are in Java. Here's a link: http://shop.oreilly.com/product/mobile/0636920039952.do On Sun, Nov 13, 2016 at 2:38 AM Tomek Janiszewski wrote: Hi Here are slides from Dario Rexin "Writing a Mesos HTTP API Client" presentation http://schd.ws/hosted_files/mesosconeu2016/e6/mesoscon_eu_2016.pdf at MesosCon EU. Unfortunately event was not recorded. — Tomek niedz., 13.11.2016, 11:26 użytkownik Petr Novak napisał: Hello, Are there any examples/guides I can take a look? Many thanks, Petr -- Best Regards, Haosdent Huang
RE: Implementation examples for framework using V1 APIs for Scala/Java
Hi, I have already read this book some time ago. It was updated for the v1 APIs? Thanks, Petr From: David Greenberg [mailto:dsg123456...@gmail.com] Sent: 13. listopadu 2016 15:36 To: user@mesos.apache.org Subject: Re: Implementation examples for framework using V1 APIs for Scala/Java There's also a book (disclaimer: I am the author) about framework development. The examples are in Java. Here's a link: http://shop.oreilly.com/product/mobile/0636920039952.do On Sun, Nov 13, 2016 at 2:38 AM Tomek Janiszewski wrote: Hi Here are slides from Dario Rexin "Writing a Mesos HTTP API Client" presentation http://schd.ws/hosted_files/mesosconeu2016/e6/mesoscon_eu_2016.pdf at MesosCon EU. Unfortunately event was not recorded. — Tomek niedz., 13.11.2016, 11:26 użytkownik Petr Novak napisał: Hello, Are there any examples/guides I can take a look? Many thanks, Petr
RE: Implementation examples for framework using V1 APIs for Scala/Java
Hi, It is like what I have in mind. But I’m contemplating if akka-streams is an overkill for a framework. I would like to see a comparison implementation with/without akka-streams/RxJava. There is no source code on github for it, right? Many thanks, Petr From: Tomek Janiszewski [mailto:jani...@gmail.com] Sent: 13. listopadu 2016 11:38 To: user@mesos.apache.org Subject: Re: Implementation examples for framework using V1 APIs for Scala/Java Hi Here are slides from Dario Rexin "Writing a Mesos HTTP API Client" presentation http://schd.ws/hosted_files/mesosconeu2016/e6/mesoscon_eu_2016.pdf at MesosCon EU. Unfortunately event was not recorded. — Tomek niedz., 13.11.2016, 11:26 użytkownik Petr Novak napisał: Hello, Are there any examples/guides I can take a look? Many thanks, Petr
Implementation examples for framework using V1 APIs for Scala/Java
Hello, Are there any examples/guides I can take a look? Many thanks, Petr
Re: Can Marathon ensure single instance of a service at any give time?
Thanks everybody for the great input. If I understand it correctly it doesn't help in this case, it just blindly restart service somewhere else once it looses heartbeat. Partition doesn't happen only because network failure it can be as simple as JVM "stop the world" with large heap or pretty much whatever. In cases when 2 and more potentially running services can throw havoc on my cluster I have to implement advanced coordination myself. It probably make sense because production level implementation for Zoo has to use LeaderSelector and equivalent and the actual logic is probably quite connected to business logic in service, e.g. when to ensure that service is still leader at this exact moment right before performing an action. Unsure if there is enough general use cases for a simple "racing for a lock on service startup" generic implementation. Petr
Can Marathon ensure single instance of a service at any give time?
Hello, if I need to run single stateless instance or only a single leader doing a work at any given time. Something I would typically implement using Zoo Curator LeaderSelector. Can I use Marathon to ensure this without having to implement mutual exclusion myself? Let's assume that other parts of the architecture aren't designed well to support more running workers at a time. Currently we have a service which updates cache, it runs on one node and when it fails it is restarted and PID file is used to ensure single instance. Pretty naive implementation, possibly doesn;t work in all edge cases. If I would like to allow it to restart on any node in a cluster can I use Marathon to simplify the implementation or it warrants more involved implementation using Zoo. Does Mesos provide any other helpers to simplify this use case? Or is Marathon designed only to run stateless services which can possibly run in multiple instances? Many thanks, Petr
Re: Mesos integration with OpenStack HEAT AutoScaling
Thanks everybody for answers. Our use case is to run our BigData platform on top of Mesos, pretty classic setup of Kafka, Spark, ES, HDFS + custom services currently with future extensions for other components as decided. It seems doable with Magnum API. We will currently run mostly barebone but we already know we will need some deployments on top of Openstack where Openstack will be in charge to primarily control resources. We are not in hurry with it so we have some time to work out details. Good to know options exists we will follow to review them closely as more detailed requirements arise. On Tue, Feb 16, 2016 at 9:28 AM, o...@magnetic.io wrote: > Hi Everybody, > > i would also suggest using the Magnum API to do this. For our > Canary-testing/releasing & autoscaling platform Vamp (www.vamp.io) we’re > currently setting up a collaboration project to develop a Vamp-Magnum > driver. This way Vamp workflows can orchestrate and coordinate the scaling > activities between IaaS and Mesos/Marathon (or any other supported > scheduler). This coordination is essential as challenges like bin-packing > and other optimisation strategies will become evident very quickly. > > If there’s interest in collaboration on this please give me a ping! > > cheers, Olaf > > Olaf Molenveld > co-founder / CEO > - > *magnetic.io <http://magnetic.io>: innovating enterprises* > *VAMP: canary test and release platform for containers* > E: o...@magnetic.io > T: +31653362783 > Skype: olafmol > www.magnetic.io > www.vamp.io > > On 16 Feb 2016, at 09:20, Guangya Liu wrote: > > Hi Peter, > > Have you ever tried Magnum (https://github.com/openstack/magnum) which is > the container service in OpenStack leveraging HEAT to integrate with > Kubernetes, Swarm and Mesos. With Magnum, you do not need to maintain your > own HEAT template but just let Magnum do this for you, it is more simple > than using HEAT directly. > > The Magnum can now supports both scale up and scale down, when scale down, > the Magnum will select the node which does not have container or have the > least containers. > > The mesos now support "Host Maintain" ( > https://github.com/apache/mesos/blob/master/docs/maintenance.md) which > can be leveraged by HEAT or Magnum, when HEAT or Magnum want to scale down > a host, we can call some cloud-init script to first maintain the host > before HEAT delete it. The host maintain will emit "InverseOffer" and you > can update the framework to handle "InverseOffer" for the host which is > going to be scale down. > > Thanks, > > Guangya > > > On Tue, Feb 16, 2016 at 4:02 PM, Petr Novak wrote: > >> Hello, >> we are considering adopting Mesos but at the same time we need to run it >> on top of OpenStack at some places. My main questions is about how and if >> autoscaling defined via HEAT templates works together. And has to be done. >> I assume that scaling up is not much a problem - when Mesos detects more >> resources it notifies frameworks which might scale based on their buildin >> strategies, though I assume it can't be defined in HEAT templates. Scaling >> down has to go through some cooperation between Mesos and HEAT. Do I have >> to update Mesos frameworks source code to somehow listen to OpenStack >> events or something like this? >> >> Is there any ongoing effort from Mesosphere and OpenStack to integrate >> more closely in this regard? >> >> Many thanks for any points regarding other possible problems and any >> clarification, >> Petr >> > > >
Mesos integration with OpenStack HEAT AutoScaling
Hello, we are considering adopting Mesos but at the same time we need to run it on top of OpenStack at some places. My main questions is about how and if autoscaling defined via HEAT templates works together. And has to be done. I assume that scaling up is not much a problem - when Mesos detects more resources it notifies frameworks which might scale based on their buildin strategies, though I assume it can't be defined in HEAT templates. Scaling down has to go through some cooperation between Mesos and HEAT. Do I have to update Mesos frameworks source code to somehow listen to OpenStack events or something like this? Is there any ongoing effort from Mesosphere and OpenStack to integrate more closely in this regard? Many thanks for any points regarding other possible problems and any clarification, Petr
