Thanks Tomas, I was finally able to deploy Apache Storm 0.9.3-rc1 through Docker clustering on Marathon i.e I deployed Storm Docker images through Marathon. In this setup I am using HAProxy with Marathon (as recommended<https://www.youtube.com/watch?v=hZNGST2vIds> by Mesosphere).
This setup allows us to deploy any App (Storm in our case) in High Availability mode. Secondly since Marathon and Mesos are only concerned about running containers, they are not aware of what we are running. By this setup we achieve both the fault tolerance for machine failures and isolation of Storm processes by using Docker. I have added a project on GitHub (https://github.com/obaidsalikeen/storm-marathon) which contains Docker images and Marathon recipes to launch a Storm Docker cluster on Apache Mesos through Marathon. Let me know your thoughts. Regards Obaid From: Tomas Barton [mailto:[email protected]] Sent: Saturday, December 20, 2014 2:23 AM To: Salikeen, Obaid Subject: Re: How to launch Storm topology on Apache Marathon Hi Obaid, I remember having problems with configuration being loaded from storm.yml that is bundled in storm jar. Nimbus hostname should be initialized to IP address, where storm-mesos framework is running (which is not necessarily same host as marathon machine). Anyway by the time you're launching storm executors you know the ipaddress - it's the address where framework is running, so you can pass it with env variable $NIMBUS_HOSTNAME Regards, Tomas On 13 December 2014 at 00:06, Obaid Salikeen <[email protected]<mailto:[email protected]>> wrote: Thanks a lot for your previous replies. I have made a lot of progress, and found an issue which can be a possible bug. Basically the problem is with the hardcoded nimbus.host (set to localhost if not specified) field in storm.yaml file. Marathon decides where to run Nimbus (for example Node A) and right after the launch of Nimbus over Marathon, the nimbus.host static entry becomes stale (it was pointing to node B and Marathon launched it to A). All the Supervisors do reach Nimbus through Zookeeper (i.e topologies are launched initially) however the code and storm.yaml files are copied from Nimbus to the Supervisors (following Storm workflow) before launching. So the Supervisors get the invalid/stale nimbus.host value and loose the connection since there is no Nimbus running on that machine. The Executer tar (tgz) file does have the latest nimbus.host entry however that storm.yaml is overwritten by the configuration provided by the Nimbus as the initial initialization step. Possible Solution: I want to make this field nimbus.host dynamic for Nimbus i.e it should be initialized by the $HOSTNAME of the machine where the Marathon launched it. It would be great if you could share your opinion and a possible workaround. After this exercise I am planning to write down my findings/tutorial. Thanks a lot Obaid From: Tomas Barton [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, December 09, 2014 2:27 PM To: Salikeen, Obaid Cc: [email protected]<mailto:[email protected]> Subject: Re: How to launch Storm topology on Apache Marathon Nimbus IP address should be stored in ZooKeeper as soon as Nimbus is launched by Marathon. Marathon could be used for launching Nimbus (log and ui services). However launching storm topologies should be negotiated by Storm-Mesos(Nimbus) framework (not Marathon). On 9 December 2014 at 22:04, Obaid Salikeen <[email protected]<mailto:[email protected]>> wrote: Thanks, I have previously deployed a few topologies on Storm cluster and Apache Mesos cluster (using Storm-Mesos project), however I want to know explicitly that if I launched Nimbus through Apache Marathon then how do I launch my topology Jar file on Nimbus (since there is no local machine installation of Nimbus, it was directly launched through Marathon and Marathon decides which machine it should run Nimbus on)? I have currently Mesos cluster with 10 nodes, and Marathon on top. Previously I deployed my Storm topology on this cluster using Storm-Mesos framework however now I want to use Marathon to launch nimbus and my tasks . Thanks Obaid From: Tomas Barton [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, December 09, 2014 11:17 AM To: Salikeen, Obaid Cc: [email protected]<mailto:[email protected]> Subject: Re: How to launch Storm topology on Apache Marathon Marathon is a meta-framework, it makes sense to run Storm on Marathon when you need to have Storm in a HA mode. Storm comes up with a command line client: http://storm.apache.org/documentation/Command-line-client.html For submitting topology you can use `storm jar ...` command. There's a configuration file storm.yaml where you can setup various settings, anyway the steps should be: 1) setup ZooKeeper quorum (for testing 1 instance should be enough 2) distribute storm binary on a cluster 3) update storm.yaml accordingly 4) start nimbus (resp. storm-mesos framework - could be done via Marathon, for testing purposes starting from console is also fine) 5) submit topology jar Nimbus address should be fetched from ZooKeeper, or you could hardcode it into storm.yaml. On 9 December 2014 at 19:08, Obaid Salikeen <[email protected]<mailto:[email protected]>> wrote: Thanks a lot Tomas, I was actually trying out Apache Marathon. I am trying to run storm-mesos framework over Marathon. So far I managed to run Nimbus and UI by running following two commands through Marathon UI, however - UI dosent know where to find Nimbus. - Secondly I don’t know how to deploy my Storm topology jar on running instance of Nimbus on Apache Marathon: Run Numbus on Marathon: Command: ./storm-mesos-0.9/bin/storm-mesos nimbus URI: http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz Run storm UI on Marathon: Command: ./storm-mesos-0.9/bin/storm ui URI: http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz It would be great if you could let me know how to deploy Storm topology over Marathon. Thanks a lot, Obaid PS [cid:[email protected]] From: Tomas Barton [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, December 09, 2014 2:15 AM To: user Subject: Re: How to launch Storm topology on Apache Marathon Hi Obaid, you'll need one instance of Nimbus (storm coordinator) which could be running as Mesos framework, have a look here: https://github.com/deric/storm-mesos Nimbus could be started via marathon, just use something like: /usr/local/bin/storm-mesos nimbus Then on Mesos slaves will be launched tasks requested by Nimbus, you don't have to start Supervisors on each slave. You could use either some binary package for distributing Storm jars or it could be copied before launching tasks via Mesos. If you are using Storm DRPC you would need to start DRPC daemon on each node. Also for accessing logs from Storm UI there is a log service that should be running on each node if you would like to use this feature. Regards, Tomas On 9 December 2014 at 03:44, Obaid Salikeen <[email protected]<mailto:[email protected]>> wrote: Hi, Currently, I couldn’t find any tutorial or steps regarding how to launch storm-mesos (Storm topology) framework on Apache Marathon. It would be great if you guys could give me any reference or hints over how to launch Storm topology over Marathon. - Do I need to first install Storm on every single machine on my Mesos cluster? - What is the recommended way to launch Storm topology over Marathon? Thanks a lot, Obaid
