Thanks, I have previously deployed a few topologies on Storm cluster and Apache Mesos cluster (using Storm-Mesos project), however I want to know explicitly that if I launched Nimbus through Apache Marathon then how do I launch my topology Jar file on Nimbus (since there is no local machine installation of Nimbus, it was directly launched through Marathon and Marathon decides which machine it should run Nimbus on)? I have currently Mesos cluster with 10 nodes, and Marathon on top. Previously I deployed my Storm topology on this cluster using Storm-Mesos framework however now I want to use Marathon to launch nimbus and my tasks .
Thanks Obaid From: Tomas Barton [mailto:[email protected]] Sent: Tuesday, December 09, 2014 11:17 AM To: Salikeen, Obaid Cc: [email protected] Subject: Re: How to launch Storm topology on Apache Marathon Marathon is a meta-framework, it makes sense to run Storm on Marathon when you need to have Storm in a HA mode. Storm comes up with a command line client: http://storm.apache.org/documentation/Command-line-client.html For submitting topology you can use `storm jar ...` command. There's a configuration file storm.yaml where you can setup various settings, anyway the steps should be: 1) setup ZooKeeper quorum (for testing 1 instance should be enough 2) distribute storm binary on a cluster 3) update storm.yaml accordingly 4) start nimbus (resp. storm-mesos framework - could be done via Marathon, for testing purposes starting from console is also fine) 5) submit topology jar Nimbus address should be fetched from ZooKeeper, or you could hardcode it into storm.yaml. On 9 December 2014 at 19:08, Obaid Salikeen <[email protected]<mailto:[email protected]>> wrote: Thanks a lot Tomas, I was actually trying out Apache Marathon. I am trying to run storm-mesos framework over Marathon. So far I managed to run Nimbus and UI by running following two commands through Marathon UI, however - UI dosent know where to find Nimbus. - Secondly I don’t know how to deploy my Storm topology jar on running instance of Nimbus on Apache Marathon: Run Numbus on Marathon: Command: ./storm-mesos-0.9/bin/storm-mesos nimbus URI: http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz Run storm UI on Marathon: Command: ./storm-mesos-0.9/bin/storm ui URI: http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz It would be great if you could let me know how to deploy Storm topology over Marathon. Thanks a lot, Obaid PS [cid:[email protected]] From: Tomas Barton [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, December 09, 2014 2:15 AM To: user Subject: Re: How to launch Storm topology on Apache Marathon Hi Obaid, you'll need one instance of Nimbus (storm coordinator) which could be running as Mesos framework, have a look here: https://github.com/deric/storm-mesos Nimbus could be started via marathon, just use something like: /usr/local/bin/storm-mesos nimbus Then on Mesos slaves will be launched tasks requested by Nimbus, you don't have to start Supervisors on each slave. You could use either some binary package for distributing Storm jars or it could be copied before launching tasks via Mesos. If you are using Storm DRPC you would need to start DRPC daemon on each node. Also for accessing logs from Storm UI there is a log service that should be running on each node if you would like to use this feature. Regards, Tomas On 9 December 2014 at 03:44, Obaid Salikeen <[email protected]<mailto:[email protected]>> wrote: Hi, Currently, I couldn’t find any tutorial or steps regarding how to launch storm-mesos (Storm topology) framework on Apache Marathon. It would be great if you guys could give me any reference or hints over how to launch Storm topology over Marathon. - Do I need to first install Storm on every single machine on my Mesos cluster? - What is the recommended way to launch Storm topology over Marathon? Thanks a lot, Obaid
