I'm getting kind of confused by the storm.yaml file. Should I be using the default.yaml and just modify the zookeeper and nimbus ip, or should I use a bran new storm.yaml?
My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on the nimbus machine. what should the storm.yaml look like on my worker and nimbus machine? Will the storm.yaml be the same on my worker and nimbus machine? I am not trying to do anything fancy, I am just trying to get a very basic cluster up and running. -Thanks, Stephen On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <[email protected] > wrote: > All Thanks so much for your help. I cannot tell you how much I > appreciate it. I'm going to try out your suggestions and keep banging my > head again the wall : D. I've spent an enormous amount of time trying to > get this to work. I'll let you know what happens after I try to implement > your suggestions. It would be really cool if someone had a tutorial that > detailed this part. (I'll make it myself if I ever get this to work!) It > seems like trying to get a two-machine cluster setup on AWS would be a VERY > common use-case. I've read and watched everything I can on the topic and > nothing got it working for me! > > On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt < > [email protected]> wrote: > >> The worker connects to the thrift port and not the ui port. You need to >> open port 6627 or whatever is the value being set in storm.yaml using >> property “nimbus.thrift.port”. >> >> Based on the configuration that you have pointed so far it seems your >> nimbus host has nimbus,ui,supervisor working because you actually have >> zookeeper running locally on that host. As Harsha pointed out you need to >> change it to a value that is the public ip instead of loopback interface. >> >> Thanks >> Parth >> >> >> On Sep 8, 2014, at 3:42 PM, Harsha <[email protected]> wrote: >> >> storm.zookeeper.servers: >> - "127.0.0.1" >> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback >> interface , instead either use your public ip or 0.0.0.0*) >> storm.local.dir: /tmp/storm ( I* recommend this to move to a different >> folder probably /home/storm, /tmp/storm will get deleted if your machine is >> restarted)* >> >> make sure you zookeeper is also listening in 0.0.0.0 or public ip not >> 127.0.0.1. >> >> "No, I cannot ping my host which has a public ip address of 54.68.149.181" >> you are not able to reach this ip form worker node but able to access the >> UI using it? >> -Harsha >> >> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote: >> >> Harsha, >> >> The storm.yaml on the host machine looks like this: >> >> storm.zookeeper.servers: >> - "127.0.0.1" >> >> >> nimbus.host: "127.0.0.1" >> >> storm.local.dir: /tmp/storm >> >> >> The storm.yaml on the worker machine looks like this: >> >> storm.zookeeper.servers: >> - "54.68.149.181" >> >> >> nimbus.host: "54.68.149.181" >> >> storm.local.dir: /tmp/storm >> >> No, I cannot ping my host which has a public ip address of 54.68.149.181 >> although I can connect to the UI web page when it is hosted. I don't know >> how I would go about connecting to zookeeper on the nimbus host. >> -Thanks, Stephen >> >> >> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <[email protected]> wrote: >> >> >> There aren't any errors in worker machine supervisor logs. Are you using >> the same storm.yaml for both the machines and also are you able to ping >> your nimbus host or connect to zookeeper on nimbus host. >> -Harsha >> >> >> >> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote: >> >> Harsha, >> >> Thanks so much for getting back with me. I will check the logs, but I >> don't seem to get any error messages. I have a nimbus AWS machine with >> zookeeper on it and a worker AWS machine. >> >> On the nimbus machine I start the zookeeper and then I run: >> >> bin/storm nimbus & >> bin/storm supervisor & >> bin/storm ui >> >> On the worker machine I run: >> bin/storm supervisor >> >> When I go to the UI page, I only see 1 supervisor (the one on the nimbus >> machine). So apparently, the worker machine isn't "registering" with the >> nimbus machine. >> >> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <[email protected]> wrote: >> >> >> Hi Stephen, >> What are the issues you are seeing. >> "How do worker machines "know" how to connect to nimbus? Is it in the >> storm configuration file" >> >> Yes. make sure you the supervisor(worker) , nimbus nodes are able to >> connect to your zookeeper cluster. >> Check your logs under storm_inst/logs/ for any errors when you try to >> start nimbus or supervisors. >> If you are installing it manually try following these steps if you are >> not already done. >> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/ >> -Harsha >> >> >> >> >> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote: >> >> All, >> >> I would greatly appreciate any help that anyone would afford. I've been >> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2 >> machines. So far, I haven't been able to get a cluster built. I can get a >> supervisor and nimbus to run on a single machine, but I can't figure out >> how to get another worker to connect to nimbus. How do worker machines >> "know" how to connect to nimbus? Is it in the storm configuration file? >> I've gone through many tutorials and the official documentation, but this >> point doesn't seem to be covered anywhere in sufficient detail for a new >> guy like me. >> >> Some of you may be tempted to point me toward storm-deploy, but I spent >> four days trying to get that to work until I gave up. I'm having Issue #58 >> on github. Following the instructions exactly and other tutorials on a bran >> new AWS machine fails. So I gave up on storm-deploy and decided to try and >> setup a cluster manually. Thanks in advance to anyone willing to offer me >> any inputs you can! >> >> >> >> >> >> >> >> >> >> >> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> > >
