[Question] Distributed Load Testing with Mesos and Gatling
Hi all, In the past weeks, I've been thinking in leveraging Mesos to schedule distributed load tests. Recently, the Kubernetes community recently shared one way to accomplish this (here: https://cloud.google.com/solutions/distributed-load-testing-using-kubernetes). One problem, at least for me, with this approach is that the load testing tool needs to coordinate the distributed scenario, and combine the data, if it doesn't, then the load clients will trigger at different times, and then later an aggregation step of the data would be handled by the user, or some external batch job, or script. This is not a problem for load generators like Tsung, or Locust, but could be a little more complicated for Gatling, since they already provide a distributed model, and coordinate the distributed tasks, and Gatling does not. To me, the approach the Kubernetes team suggests is really a hack using the 'ReplicationController' to spawn multiple replicas, which could be easily achieved using the same approach with Marathon (or Kubernetes on Mesos). I was thinking of building a Mesos framework, that would take the input, or load simulation file, and would schedule jobs across the cluster (perhaps with dedicated resources too minimize variance) using Gatling. A Mesos framework will be able to provide a UI/API to take the input jobs, and report status of multiple jobs. It can also provide a way to sync/orchestrate the simulation, and finally provide a way to aggregate the simulation data in one place, and serve the generated HTML report. Boiled down to its primitive parts, it would spin multiple Gatling (java) processes across the cluster, use something like a barrier (not sure what to use here) to wait for all processes to be ready to execute, and finally copy, and rename the generated simulations logs from each Gatling process to one node/place, that is finally aggregated and compiled to HTML report by a single Gatling process. First of all, is there anything in the Mesos community that does this already? If not, do you think this is feasible to accomplish with a Mesos framework, and would you recommend to go with this approach? Does Mesos offers a barrier-like features to coordinate jobs, and can I somehow move files to a single node to be processed? Finally, I've never written a non-trivial Mesos framework, how should I go about, or find more documentation, to get started? I'm looking for best practices, pitfalls, etc. Thank you for your time, Carlos
Re: step-step guide for New-to-Mesos
Sorry, marthon should be marathon https://mesosphere.github.io/marathon/ On Wed, Jul 1, 2015 at 9:23 PM, haosdent haosd...@gmail.com wrote: Hi, @Hajira Next step is to run tasks in containers in Mesos. You want to run somethinng like web application in docker or others? You could try marthon or other exist framework first. I think you don't need to write a framework. On Wed, Jul 1, 2015 at 9:07 PM, Hajira Jabeen hajirajab...@gmail.com wrote: Hello, Being new to Mesos (and everything related to big data), I have been able to install mesos and run example frameworks. Next step is to run tasks in containers in Mesos. Do I have to write a framework for this , or just change the ContainerInfo etc. fields in Mesos.proto file ? Is there any step-step working guide ? Mesos documentation assumes a lot background knowledge, that I do not have .. Any help and pointers will be appreciated .. Regards Hajira On 30 June 2015 at 00:23, Andras Kerekes andras.kere...@ishisystems.com wrote: Hi, Is there a preferred way to do service discovery in Mesos via mesos-dns running on CoreOS? I’m trying to implement a simple app which consists of two docker containers and one of them (A) depends on the other (B). What I’d like to do is to tell container A to use a fix dns name (containerB.marathon.mesos in case of mesos-dns) to find the other service. There are at least 3 different ways I think it can be done, but the 3 I found all have some shortcomings. 1. Use SRV records to get the port along with the IP. Con: I’d prefer not to build the logic of handling SRV records into the app, it can be a legacy app that is difficult to modify 2. Use haproxy on slaves and connect via a well-known port on localhost. Cons: the Marathon provided script does not run on CoreOS, also I don’t know how to run haproxy on CoreOS outside of a docker container. If it is running in a docker container, then how can it dynamically allocate ports on localhost if a new service is discovered in Marathon/Mesos? 3. Use dedicated port to bind the containers to. Con: I can have only as many instances of a service as many slaves I have because they bind to the same port. What other alternatives are there? Thanks, Andras -- Best Regards, Haosdent Huang -- Best Regards, Haosdent Huang
Re: step-step guide for New-to-Mesos
Hey Hajira, you may find this blog entry useful: https://mesosphere.com/blog/2015/04/02/continuous-deployment-with-mesos-marathon-docker/ A bit older, but more specific to Docker, please have a look here: https://mesosphere.github.io/marathon/docs/native-docker.html generally speaking, there is a lot of info available at: http://docs.mesosphere.com you could find useful too. Obviously, you can launch containers using a very simple framework, but that's largely not necessary; and most certainly, don't change the mesos.proto contents (this will prevent a lot of stuff from working): that is meant to be a read-only file (well, unless one is doing development on Mesos itself). We have made RENDLER publicly available as an example framework: https://github.com/mesosphere/RENDLER HTH *Marco Massenzio* *Distributed Systems Engineer* On Wed, Jul 1, 2015 at 9:23 AM, haosdent haosd...@gmail.com wrote: Sorry, marthon should be marathon https://mesosphere.github.io/marathon/ On Wed, Jul 1, 2015 at 9:23 PM, haosdent haosd...@gmail.com wrote: Hi, @Hajira Next step is to run tasks in containers in Mesos. You want to run somethinng like web application in docker or others? You could try marthon or other exist framework first. I think you don't need to write a framework. On Wed, Jul 1, 2015 at 9:07 PM, Hajira Jabeen hajirajab...@gmail.com wrote: Hello, Being new to Mesos (and everything related to big data), I have been able to install mesos and run example frameworks. Next step is to run tasks in containers in Mesos. Do I have to write a framework for this , or just change the ContainerInfo etc. fields in Mesos.proto file ? Is there any step-step working guide ? Mesos documentation assumes a lot background knowledge, that I do not have .. Any help and pointers will be appreciated .. Regards Hajira On 30 June 2015 at 00:23, Andras Kerekes andras.kere...@ishisystems.com wrote: Hi, Is there a preferred way to do service discovery in Mesos via mesos-dns running on CoreOS? I’m trying to implement a simple app which consists of two docker containers and one of them (A) depends on the other (B). What I’d like to do is to tell container A to use a fix dns name (containerB.marathon.mesos in case of mesos-dns) to find the other service. There are at least 3 different ways I think it can be done, but the 3 I found all have some shortcomings. 1. Use SRV records to get the port along with the IP. Con: I’d prefer not to build the logic of handling SRV records into the app, it can be a legacy app that is difficult to modify 2. Use haproxy on slaves and connect via a well-known port on localhost. Cons: the Marathon provided script does not run on CoreOS, also I don’t know how to run haproxy on CoreOS outside of a docker container. If it is running in a docker container, then how can it dynamically allocate ports on localhost if a new service is discovered in Marathon/Mesos? 3. Use dedicated port to bind the containers to. Con: I can have only as many instances of a service as many slaves I have because they bind to the same port. What other alternatives are there? Thanks, Andras -- Best Regards, Haosdent Huang -- Best Regards, Haosdent Huang
Re: step-step guide for New-to-Mesos
Hi, @Hajira Next step is to run tasks in containers in Mesos. You want to run somethinng like web application in docker or others? You could try marthon or other exist framework first. I think you don't need to write a framework. On Wed, Jul 1, 2015 at 9:07 PM, Hajira Jabeen hajirajab...@gmail.com wrote: Hello, Being new to Mesos (and everything related to big data), I have been able to install mesos and run example frameworks. Next step is to run tasks in containers in Mesos. Do I have to write a framework for this , or just change the ContainerInfo etc. fields in Mesos.proto file ? Is there any step-step working guide ? Mesos documentation assumes a lot background knowledge, that I do not have .. Any help and pointers will be appreciated .. Regards Hajira On 30 June 2015 at 00:23, Andras Kerekes andras.kere...@ishisystems.com wrote: Hi, Is there a preferred way to do service discovery in Mesos via mesos-dns running on CoreOS? I’m trying to implement a simple app which consists of two docker containers and one of them (A) depends on the other (B). What I’d like to do is to tell container A to use a fix dns name (containerB.marathon.mesos in case of mesos-dns) to find the other service. There are at least 3 different ways I think it can be done, but the 3 I found all have some shortcomings. 1. Use SRV records to get the port along with the IP. Con: I’d prefer not to build the logic of handling SRV records into the app, it can be a legacy app that is difficult to modify 2. Use haproxy on slaves and connect via a well-known port on localhost. Cons: the Marathon provided script does not run on CoreOS, also I don’t know how to run haproxy on CoreOS outside of a docker container. If it is running in a docker container, then how can it dynamically allocate ports on localhost if a new service is discovered in Marathon/Mesos? 3. Use dedicated port to bind the containers to. Con: I can have only as many instances of a service as many slaves I have because they bind to the same port. What other alternatives are there? Thanks, Andras -- Best Regards, Haosdent Huang
step-step guide for New-to-Mesos
Hello, Being new to Mesos (and everything related to big data), I have been able to install mesos and run example frameworks. Next step is to run tasks in containers in Mesos. Do I have to write a framework for this , or just change the ContainerInfo etc. fields in Mesos.proto file ? Is there any step-step working guide ? Mesos documentation assumes a lot background knowledge, that I do not have .. Any help and pointers will be appreciated .. Regards Hajira On 30 June 2015 at 00:23, Andras Kerekes andras.kere...@ishisystems.com wrote: Hi, Is there a preferred way to do service discovery in Mesos via mesos-dns running on CoreOS? I’m trying to implement a simple app which consists of two docker containers and one of them (A) depends on the other (B). What I’d like to do is to tell container A to use a fix dns name (containerB.marathon.mesos in case of mesos-dns) to find the other service. There are at least 3 different ways I think it can be done, but the 3 I found all have some shortcomings. 1. Use SRV records to get the port along with the IP. Con: I’d prefer not to build the logic of handling SRV records into the app, it can be a legacy app that is difficult to modify 2. Use haproxy on slaves and connect via a well-known port on localhost. Cons: the Marathon provided script does not run on CoreOS, also I don’t know how to run haproxy on CoreOS outside of a docker container. If it is running in a docker container, then how can it dynamically allocate ports on localhost if a new service is discovered in Marathon/Mesos? 3. Use dedicated port to bind the containers to. Con: I can have only as many instances of a service as many slaves I have because they bind to the same port. What other alternatives are there? Thanks, Andras
MesosCon Seattle 2015 Keynote Speakers Announced
The countdown until #MesosCon has begun: 7 weeks until MesosCon! MesosCon is an annual conference organized by the Apache Mesos community, bringing together users and developers to share and learn about the project and its growing ecosystem. Register now[1] and join us in Seattle this August! Here’s a rundown of several conference announcements: *Keynote Speakers** * **We’ve curated a stellar line-up of invited keynote speakers, including: * Adrian Crockroft (Battery Ventures) * Benjamin Hindman (Mesosphere) * Neha Narula (MIT) * Peter Bailis (Berkeley) You don’t want to miss these! We’ll also feature platinum sponsored keynotes from Cisco, Intel, and Mesosphere. *Mesos Hackathon** * **The Mesos community will once again hold a day- long hackathon, this year on Wednesday, August 19th and sponsored by SignalFx. More details will be announced soon including the location and registration; stay tuned and hold the date! *Lightning Talks** * **Want to present at MesosCon, but missed the call for full sessions? We’ve opened a CFP for lightning talks[2]; the application deadline is July 15th. *Diversity Scholarship Program** * **The MesosCon scholarship program[3] provides support to women, people of color, and people with disabilities who may not otherwise have the opportunity to attend MesosCon for financial reasons. Equal access and diversity are important to MesosCon, and we aim to remove this obstacle for underrepresented attendee groups. We want you at MesosCon, and we want to help you get there! Applications are beginning to be reviewed, and will be reviewed on a rolling basis. Apply today! We also encourage MesosCon attendees to donate toward the diversity scholarship program directly during the time of registration using the online system. *Stay In Touch** * **As we make future announcements about the conference, stay in the loop and help us get the word out! We’ll be tweeting from @ApacheMesos and @MesosCon, blogging on the Apache Mesos blog, and sharing updates on the project’s user@ mailing lists. *About MesosCon** * **MesosCon will take place August 20 - 21, 2015 at Sheraton Seattle in Seattle, WA with a hackathon taking place the day prior to the event. The standard attendee registration fee will be US$399 (Through July 31, 2015), and late registration US$499 beginning August 1, 2015. Students are eligible for a special registration fee of US$99 (Note that a valid student id will be required at onsite check-in). Hope to see you at this year’s MesosCon! Links: 1. http://events.linuxfoundation.org/events/mesoscon/attend/register 2. http://events.linuxfoundation.org/events/mesoscon/program/lighting-talks 3. http://events.linuxfoundation.org/events/mesoscon/attend/scholarship
mesos cluster can't fit federation cluster
Hello, I would like to deploy master nodes in a private zone, and setup mesos slaves in another datacenter. But the multi-datacenter mode can't work. it need slave node can reach master node in public network ip. But in production zone, the gateway ip is not belong to master nodes. Does anyone have same experience on multi-datacenter deployment case? I prefer kubernets cluster proposal. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/proposals/federation-high-level-arch.png -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
Re: mesos cluster can't fit federation cluster
Hi Tommy, not sure what your use-case is, but you are correct, the master/slave nodes need to have bi-directional connectivity. However, there is no fundamental reason why those have to be public IPs - so long as they are routable (either via DNS discovery and / or VPN or other network-layer mechanisms) that will work. (I mean, without even thinking too hard about this - so I may be entirely wrong here - you could place a couple of Nginx/HAproxy nodes with two NICs, one visible to the Slaves, the other in the VPC subnet, and forward all traffic? I'm sure I'm missing something here :) When you launch the master nodes, you specify the NICs they need to listen to via the --ip option, while the slave nodes have the --master flag that should have either a hostname:port of ip:port argument: so long as they are routable, this *should* work (although, admittedly, I've never tried this personally). One concern I would have in such an arrangement though, would be about network partitioning: if the DC/DC connectivity were to drop, you'd suddenly lose all master/slave connectivity; it's also not clear to me that having sectioned the Masters from the Slaves would give you better availability and/or reliability and/or security? It would be great to understand the use-case, so we could see what could be added (if anything) to Mesos going forward. *Marco Massenzio* *Distributed Systems Engineer* On Wed, Jul 1, 2015 at 9:15 AM, tommy xiao xia...@gmail.com wrote: Hello, I would like to deploy master nodes in a private zone, and setup mesos slaves in another datacenter. But the multi-datacenter mode can't work. it need slave node can reach master node in public network ip. But in production zone, the gateway ip is not belong to master nodes. Does anyone have same experience on multi-datacenter deployment case? I prefer kubernets cluster proposal. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/proposals/federation-high-level-arch.png -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
Re: service discovery in Mesos on CoreOS
If I understand right, the root trouble is mesos-slave-in-docker. I know CoreOS little, do u run mesos-slave on CoreOS as following? docker run --rm -it --name mesos-slave --net host --volume /var/run/docker.sock:/var/run/docker.sock --entrypoint mesos-slave or map the CoreOS docker.sock into mesos-slave container to discovery service better. If I said nothing, sorry for my interruption. 2015-06-30 21:10 GMT+08:00 Andras Kerekes andras.kere...@ishisystems.com: Would not using Bamboo to update haproxy config have the same problems I described for the Marathon provided script? It would still run in a separate container. *From:* zhou weitao [mailto:zhouwtl...@gmail.com] *Sent:* Monday, June 29, 2015 10:51 PM *To:* user@mesos.apache.org *Subject:* Re: service discovery in Mesos on CoreOS 2015-06-30 6:23 GMT+08:00 Andras Kerekes andras.kere...@ishisystems.com: Hi, Is there a preferred way to do service discovery in Mesos via mesos-dns running on CoreOS? I’m trying to implement a simple app which consists of two docker containers and one of them (A) depends on the other (B). What I’d like to do is to tell container A to use a fix dns name (containerB.marathon.mesos in case of mesos-dns) to find the other service. There are at least 3 different ways I think it can be done, but the 3 I found all have some shortcomings. 1. Use SRV records to get the port along with the IP. Con: I’d prefer not to build the logic of handling SRV records into the app, it can be a legacy app that is difficult to modify 2. Use haproxy on slaves and connect via a well-known port on localhost. Cons: the Marathon provided script does not run on CoreOS, also I don’t know how to run haproxy on CoreOS outside of a docker container. If it is running in a docker container, then how can it dynamically allocate ports on localhost if a new service is discovered in Marathon/Mesos? Do you know this repo? https://github.com/QubitProducts/bamboo . And here our corp one https://github.com/Dataman-Cloud/bamboo branched from the above. 3. Use dedicated port to bind the containers to. Con: I can have only as many instances of a service as many slaves I have because they bind to the same port. What other alternatives are there? Thanks, Andras
Re: service discovery in Mesos on CoreOS
As others has mentioned earlier, definitely don't use the mesos- prefix to name your docker containers since at the time we did the integration docker labels wasn't merged. Also you'll need to run mesos-slave with --pid=host, bind mount in the docker socket, and also bind mount a host directory into the slave as its work directory so the slave can recover its tasks when it restarts. Tim On Tue, Jun 30, 2015 at 11:11 PM, zhou weitao zhouwtl...@gmail.com wrote: If I understand right, the root trouble is mesos-slave-in-docker. I know CoreOS little, do u run mesos-slave on CoreOS as following? docker run --rm -it --name mesos-slave --net host --volume /var/run/docker.sock:/var/run/docker.sock --entrypoint mesos-slave or map the CoreOS docker.sock into mesos-slave container to discovery service better. If I said nothing, sorry for my interruption. 2015-06-30 21:10 GMT+08:00 Andras Kerekes andras.kere...@ishisystems.com : Would not using Bamboo to update haproxy config have the same problems I described for the Marathon provided script? It would still run in a separate container. *From:* zhou weitao [mailto:zhouwtl...@gmail.com] *Sent:* Monday, June 29, 2015 10:51 PM *To:* user@mesos.apache.org *Subject:* Re: service discovery in Mesos on CoreOS 2015-06-30 6:23 GMT+08:00 Andras Kerekes andras.kere...@ishisystems.com : Hi, Is there a preferred way to do service discovery in Mesos via mesos-dns running on CoreOS? I’m trying to implement a simple app which consists of two docker containers and one of them (A) depends on the other (B). What I’d like to do is to tell container A to use a fix dns name (containerB.marathon.mesos in case of mesos-dns) to find the other service. There are at least 3 different ways I think it can be done, but the 3 I found all have some shortcomings. 1. Use SRV records to get the port along with the IP. Con: I’d prefer not to build the logic of handling SRV records into the app, it can be a legacy app that is difficult to modify 2. Use haproxy on slaves and connect via a well-known port on localhost. Cons: the Marathon provided script does not run on CoreOS, also I don’t know how to run haproxy on CoreOS outside of a docker container. If it is running in a docker container, then how can it dynamically allocate ports on localhost if a new service is discovered in Marathon/Mesos? Do you know this repo? https://github.com/QubitProducts/bamboo . And here our corp one https://github.com/Dataman-Cloud/bamboo branched from the above. 3. Use dedicated port to bind the containers to. Con: I can have only as many instances of a service as many slaves I have because they bind to the same port. What other alternatives are there? Thanks, Andras
Questions about Framework/Scheduler
Hey, I am new to mesos and just start to investigate it. I have a fundament question about Framework Assume i am using a long live framework, how the mesos master detect the framework in unavailable , such as some network error or some internal error from the framework ? (I can not find it at mater.cpp. Could you please point out the source code for me ?)The framework has been registered to the master successfully, and has successfully run for a whole. Thanks Ying
Re: Questions about Framework/Scheduler
See Master::exited() https://github.com/apache/mesos/blob/0.22.1/src/master/master.cpp#L878 which derives from Process::exited() https://github.com/apache/mesos/blob/0.22.1/3rdparty/libprocess/include/process/process.hpp#L55 In the event of a temporary network partition, the Mesos master will continue trying to send offer/status/etc. messages to the framework scheduler. Since status messages are reliable at-least once delivery, they are actually queued up (per task) on the slave until an acknowledgement is received from scheduler. On Wed, Jul 1, 2015 at 5:08 PM, Ying Ji jiyin...@gmail.com wrote: Hey, I am new to mesos and just start to investigate it. I have a fundament question about Framework Assume i am using a long live framework, how the mesos master detect the framework in unavailable , such as some network error or some internal error from the framework ? (I can not find it at mater.cpp. Could you please point out the source code for me ?)The framework has been registered to the master successfully, and has successfully run for a whole. Thanks Ying
