Stephen,
I am not able to reach that IP. But you shouldn't
modify the default.yaml just change storm.yaml under conf
"Will the storm.yaml be the same on my worker and nimbus
machine?"
it should be the same on both machines. Make sure your
zookeeper also running on that ip. And check for logs under you
storm installation it should be under logs dir.
-Harsha
On Mon, Sep 8, 2014, at 05:23 PM, Stephen Hartzell wrote:
All,
I implemented the suggestions given by Parh and Harsha. I am
now using the default.yaml but I changed the
storm.zookeeper.servers to the nimbus machine's ip address:
54.68.149.181. I also changed the nimbus.host to 54.68.149.181.
I also opened up port 6627. Now, the UI web page gives the
following
error: org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused
You should be able to see the error it gives by going to the
web page yourself at: [1]http://54.68.149.181:8080. I am only
using this account to test and see if I can even get storm to
work, so these machines are only for testing. Perhaps someone
could tell me what the storm.yaml file should look like for
this setup?
-Thanks, Stephne
On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell
<[2][email protected]> wrote:
I'm getting kind of confused by the storm.yaml file. Should I
be using the default.yaml and just modify the zookeeper and
nimbus ip, or should I use a bran new storm.yaml?
My nimbus machine has the ip address: 54.68.149.181.My
zookeeper is on the nimbus machine. what should the storm.yaml
look like on my worker and nimbus machine? Will the storm.yaml
be the same on my worker and nimbus machine? I am not trying to
do anything fancy, I am just trying to get a very basic cluster
up and running.
-Thanks, Stephen
On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell
<[3][email protected]> wrote:
All Thanks so much for your help. I cannot tell you how much
I appreciate it. I'm going to try out your suggestions and keep
banging my head again the wall : D. I've spent an enormous
amount of time trying to get this to work. I'll let you know
what happens after I try to implement your suggestions. It
would be really cool if someone had a tutorial that detailed
this part. (I'll make it myself if I ever get this to work!) It
seems like trying to get a two-machine cluster setup on AWS
would be a VERY common use-case. I've read and watched
everything I can on the topic and nothing got it working for
me!
On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt
<[4][email protected]> wrote:
The worker connects to the thrift port and not the ui port. You
need to open port 6627 or whatever is the value being set in
storm.yaml using property “nimbus.thrift.port”.
Based on the configuration that you have pointed so far it
seems your nimbus host has nimbus,ui,supervisor working because
you actually have zookeeper running locally on that host. As
Harsha pointed out you need to change it to a value that is the
public ip instead of loopback interface.
Thanks
Parth
On Sep 8, 2014, at 3:42 PM, Harsha <[5][email protected]> wrote:
storm.zookeeper.servers:
- "127.0.0.1"
nimbus.host: "127.0.0.1" ( 127.0.0.1 causes to bind a loopback
interface , instead either use your public ip or 0.0.0.0)
storm.local.dir: /tmp/storm ( I recommend this to move to a
different folder probably /home/storm, /tmp/storm will get
deleted if your machine is restarted)
make sure you zookeeper is also listening in 0.0.0.0 or public
ip not 127.0.0.1.
"No, I cannot ping my host which has a public ip address of
54.68.149.181"
you are not able to reach this ip form worker node but able to
access the UI using it?
-Harsha
On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
Harsha,
The storm.yaml on the host machine looks like this:
storm.zookeeper.servers:
- "127.0.0.1"
nimbus.host: "127.0.0.1"
storm.local.dir: /tmp/storm
The storm.yaml on the worker machine looks like this:
storm.zookeeper.servers:
- "54.68.149.181"
nimbus.host: "54.68.149.181"
storm.local.dir: /tmp/storm
No, I cannot ping my host which has a public ip address of
54.68.149.181 although I can connect to the UI web page when it
is hosted. I don't know how I would go about connecting to
zookeeper on the nimbus host.
-Thanks, Stephen
On Mon, Sep 8, 2014 at 6:28 PM, Harsha <[6][email protected]>
wrote:
There aren't any errors in worker machine supervisor logs. Are
you using the same storm.yaml for both the machines and also
are you able to ping your nimbus host or connect to zookeeper
on nimbus host.
-Harsha
On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
Harsha,
Thanks so much for getting back with me. I will check the
logs, but I don't seem to get any error messages. I have a
nimbus AWS machine with zookeeper on it and a worker AWS
machine.
On the nimbus machine I start the zookeeper and then I run:
bin/storm nimbus &
bin/storm supervisor &
bin/storm ui
On the worker machine I run:
bin/storm supervisor
When I go to the UI page, I only see 1 supervisor (the one on
the nimbus machine). So apparently, the worker machine isn't
"registering" with the nimbus machine.
On Mon, Sep 8, 2014 at 6:16 PM, Harsha <[7][email protected]>
wrote:
Hi Stephen,
What are the issues you are seeing.
"How do worker machines "know" how to connect to nimbus? Is it
in the storm configuration file"
Yes. make sure you the supervisor(worker) , nimbus nodes are
able to connect to your zookeeper cluster.
Check your logs under storm_inst/logs/ for any errors when you
try to start nimbus or supervisors.
If you are installing it manually try following these steps if
you are not already done.
[8]http://www.michael-noll.com/tutorials/running-multi-node-sto
rm-cluster/
-Harsha
On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
All,
I would greatly appreciate any help that anyone would afford.
I've been trying to setup a storm cluster on AWS for a few
weeks now on centOS EC2 machines. So far, I haven't been able
to get a cluster built. I can get a supervisor and nimbus to
run on a single machine, but I can't figure out how to get
another worker to connect to nimbus. How do worker machines
"know" how to connect to nimbus? Is it in the storm
configuration file? I've gone through many tutorials and the
official documentation, but this point doesn't seem to be
covered anywhere in sufficient detail for a new guy like me.
Some of you may be tempted to point me toward storm-deploy,
but I spent four days trying to get that to work until I gave
up. I'm having Issue #58 on github. Following the instructions
exactly and other tutorials on a bran new AWS machine fails. So
I gave up on storm-deploy and decided to try and setup a
cluster manually. Thanks in advance to anyone willing to offer
me any inputs you can!
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual
or entity to which it is addressed and may contain information
that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the
intended recipient, you are hereby notified that any printing,
copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately and delete it from your system. Thank You.
References
1. http://54.68.149.181:8080/
2. mailto:[email protected]
3. mailto:[email protected]
4. mailto:[email protected]
5. mailto:[email protected]
6. mailto:[email protected]
7. mailto:[email protected]
8. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
