All, I implemented the suggestions given by Parh and Harsha. I am now using the default.yaml but I changed the storm.zookeeper.servers to the nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives the following error: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
You should be able to see the error it gives by going to the web page yourself at: http://54.68.149.181:8080. I am only using this account to test and see if I can even get storm to work, so these machines are only for testing. Perhaps someone could tell me what the storm.yaml file should look like for this setup? -Thanks, Stephne On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <[email protected] > wrote: > I'm getting kind of confused by the storm.yaml file. Should I be using the > default.yaml and just modify the zookeeper and nimbus ip, or should I use a > bran new storm.yaml? > > My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on > the nimbus machine. what should the storm.yaml look like on my worker and > nimbus machine? Will the storm.yaml be the same on my worker and nimbus > machine? I am not trying to do anything fancy, I am just trying to get a > very basic cluster up and running. > > -Thanks, Stephen > > On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell < > [email protected]> wrote: > >> All Thanks so much for your help. I cannot tell you how much I >> appreciate it. I'm going to try out your suggestions and keep banging my >> head again the wall : D. I've spent an enormous amount of time trying to >> get this to work. I'll let you know what happens after I try to implement >> your suggestions. It would be really cool if someone had a tutorial that >> detailed this part. (I'll make it myself if I ever get this to work!) It >> seems like trying to get a two-machine cluster setup on AWS would be a VERY >> common use-case. I've read and watched everything I can on the topic and >> nothing got it working for me! >> >> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt < >> [email protected]> wrote: >> >>> The worker connects to the thrift port and not the ui port. You need to >>> open port 6627 or whatever is the value being set in storm.yaml using >>> property “nimbus.thrift.port”. >>> >>> Based on the configuration that you have pointed so far it seems your >>> nimbus host has nimbus,ui,supervisor working because you actually have >>> zookeeper running locally on that host. As Harsha pointed out you need to >>> change it to a value that is the public ip instead of loopback interface. >>> >>> Thanks >>> Parth >>> >>> >>> On Sep 8, 2014, at 3:42 PM, Harsha <[email protected]> wrote: >>> >>> storm.zookeeper.servers: >>> - "127.0.0.1" >>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback >>> interface , instead either use your public ip or 0.0.0.0*) >>> storm.local.dir: /tmp/storm ( I* recommend this to move to a different >>> folder probably /home/storm, /tmp/storm will get deleted if your machine is >>> restarted)* >>> >>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not >>> 127.0.0.1. >>> >>> "No, I cannot ping my host which has a public ip address of >>> 54.68.149.181" >>> you are not able to reach this ip form worker node but able to access >>> the UI using it? >>> -Harsha >>> >>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote: >>> >>> Harsha, >>> >>> The storm.yaml on the host machine looks like this: >>> >>> storm.zookeeper.servers: >>> - "127.0.0.1" >>> >>> >>> nimbus.host: "127.0.0.1" >>> >>> storm.local.dir: /tmp/storm >>> >>> >>> The storm.yaml on the worker machine looks like this: >>> >>> storm.zookeeper.servers: >>> - "54.68.149.181" >>> >>> >>> nimbus.host: "54.68.149.181" >>> >>> storm.local.dir: /tmp/storm >>> >>> No, I cannot ping my host which has a public ip address of 54.68.149.181 >>> although I can connect to the UI web page when it is hosted. I don't know >>> how I would go about connecting to zookeeper on the nimbus host. >>> -Thanks, Stephen >>> >>> >>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <[email protected]> wrote: >>> >>> >>> There aren't any errors in worker machine supervisor logs. Are you using >>> the same storm.yaml for both the machines and also are you able to ping >>> your nimbus host or connect to zookeeper on nimbus host. >>> -Harsha >>> >>> >>> >>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote: >>> >>> Harsha, >>> >>> Thanks so much for getting back with me. I will check the logs, but I >>> don't seem to get any error messages. I have a nimbus AWS machine with >>> zookeeper on it and a worker AWS machine. >>> >>> On the nimbus machine I start the zookeeper and then I run: >>> >>> bin/storm nimbus & >>> bin/storm supervisor & >>> bin/storm ui >>> >>> On the worker machine I run: >>> bin/storm supervisor >>> >>> When I go to the UI page, I only see 1 supervisor (the one on the nimbus >>> machine). So apparently, the worker machine isn't "registering" with the >>> nimbus machine. >>> >>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <[email protected]> wrote: >>> >>> >>> Hi Stephen, >>> What are the issues you are seeing. >>> "How do worker machines "know" how to connect to nimbus? Is it in the >>> storm configuration file" >>> >>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to >>> connect to your zookeeper cluster. >>> Check your logs under storm_inst/logs/ for any errors when you try to >>> start nimbus or supervisors. >>> If you are installing it manually try following these steps if you are >>> not already done. >>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/ >>> -Harsha >>> >>> >>> >>> >>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote: >>> >>> All, >>> >>> I would greatly appreciate any help that anyone would afford. I've been >>> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2 >>> machines. So far, I haven't been able to get a cluster built. I can get a >>> supervisor and nimbus to run on a single machine, but I can't figure out >>> how to get another worker to connect to nimbus. How do worker machines >>> "know" how to connect to nimbus? Is it in the storm configuration file? >>> I've gone through many tutorials and the official documentation, but this >>> point doesn't seem to be covered anywhere in sufficient detail for a new >>> guy like me. >>> >>> Some of you may be tempted to point me toward storm-deploy, but I >>> spent four days trying to get that to work until I gave up. I'm having >>> Issue #58 on github. Following the instructions exactly and other tutorials >>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided >>> to try and setup a cluster manually. Thanks in advance to anyone willing to >>> offer me any inputs you can! >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >>> >> >> >
