My apologies, I run "bin/storm supervisor &" at the end. That was a bad copy and paste.
On Tue, Sep 9, 2014 at 8:10 AM, Vikas Agarwal <[email protected]> wrote: > >>*I launch another basic EC2 CentOS machine and do the exact same > commands except that I don't install zookeeper and I only run "bin/storm > nimbus &" at the end*<< > > Why would you run nimbus on two machines? > > Yes, by default Hortonworks provides one supervisor node par machine, so I > am trying to add new EC2 machine with supervisor role. And it worked upto > some extent but still not fully functional. I am still testing its linear > scalability. > > > On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell < > [email protected]> wrote: > >> Vikas, >> >> >> I've tried to use the HortonWorks distribution, but that only provides >> one supervisor and nimbus on one virtual machine. I'm excited to hear that >> you have storm working on AWS EC2 machines because that is exactly what I >> am trying to do! Right now we're still in the development stage, so all we >> are trying to do is to have one worker machine connect to one nimbus >> machine. So far we haven't got this work. >> >> Although it might be lengthy, let me go ahead and post the commands I'm >> using to setup the nimbus machine. >> >> >> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to TCP >> connections (it's public IP is 54.68.149.181)* >> >> sudo yum update >> >> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel >> >> *# Install zeromq* >> >> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz >> >> tar –zxvf zeromq-2.1.7.tar.gz >> >> cd zeromq-2.1.7.tar.gz >> >> ./configure >> >> make >> >> sudo make install >> >> cd ~ >> >> *# Install jzmq* >> >> sudo yum install git java-devel libtool >> >> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *# >> JDK required for jzmq in the configuration stage* >> >> cd $JAVA_HOME >> >> sudo ln –s include include *# >> JDK include directory has headers required for jzmq* >> >> cd ~ >> >> git clone https://github.com/nathanmarz/jzmq.git >> >> cd jzmq/src/ >> >> CLASSPATH=.:./.:$CLASSPATH >> >> touch classdist_noinst.stamp >> >> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java >> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java >> org/zeromq/ZMQStreamer.java >> >> cd ~/jzmq >> >> ./autogen.sh >> >> ./configure >> >> make >> >> sudo make install >> >> cd ~ >> >> *# Download zookeeper* >> >> wget >> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz >> >> tar -zxvf zookeeper-3.4.6.tar.gz >> >> cd zookeeper-3.4.6 >> >> vi conf/zoo.cfg >> >> >> tickTime=2000 >> >> dataDir=/tmp/zookeeper >> >> clientPort=2181 >> >> >> mkdir /tmp/zookeeper >> bin/zkServer.sh start >> bin/zkCli.sh -server 54.68.149.181:2181 >> >> >> *# Download storm and modify configuration file (changes over >> default.yaml shown in bold)* >> >> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip >> storm-0.8.2.zip >> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip> >> >> cd storm-0.8.2.zip >> >> vi conf/storm.yaml >> >> >> # Licensed to the Apache Software Foundation (ASF) under one >>> >>> # or more contributor license agreements. See the NOTICE file >>> >>> # distributed with this work for additional information >>> >>> # regarding copyright ownership. The ASF licenses this file >>> >>> # to you under the Apache License, Version 2.0 (the >>> >>> # "License"); you may not use this file except in compliance >>> >>> # with the License. You may obtain a copy of the License at >>> >>> # >>> >>> # http:# www.apache.org/licenses/LICENSE-2.0 >>> >>> # >>> >>> # Unless required by applicable law or agreed to in writing, software >>> >>> # distributed under the License is distributed on an "AS IS" BASIS, >>> >>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >>>> implied. >>> >>> # See the License for the specific language governing permissions and >>> >>> # limitations under the License. >>> >>> >>>> >>>> ########### These all have default values as shown >>> >>> ########### Additional configuration goes into storm.yaml >>> >>> >>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib: >>>> */home/ec2-user*" >>> >>> >>>> ### storm.* configs are general configurations >>> >>> # the local dir is where jars are kept >>> >>> storm.local.dir: "storm-local" >>> >>> storm.zookeeper.servers: >>> >>> - "*54.68.149.181*" >>> >>> storm.zookeeper.port: 2181 >>> >>> storm.zookeeper.root: "/storm" >>> >>> storm.zookeeper.session.timeout: 20000 >>> >>> storm.zookeeper.connection.timeout: 15000 >>> >>> storm.zookeeper.retry.times: 5 >>> >>> storm.zookeeper.retry.interval: 1000 >>> >>> storm.zookeeper.retry.intervalceiling.millis: 30000 >>> >>> storm.cluster.mode: "distributed" # can be distributed or local >>> >>> storm.local.mode.zmq: false >>> >>> storm.thrift.transport: >>>> "backtype.storm.security.auth.SimpleTransportPlugin" >>> >>> storm.messaging.transport: "backtype.storm.messaging.zmq" >>> >>> >>>> ### nimbus.* configs are for the master >>> >>> nimbus.host: "*54.68.149.181*" >>> >>> nimbus.thrift.port: 6627 >>> >>> nimbus.childopts: "-Xmx1024m" >>> >>> nimbus.task.timeout.secs: 30 >>> >>> nimbus.supervisor.timeout.secs: 60 >>> >>> nimbus.monitor.freq.secs: 10 >>> >>> nimbus.cleanup.inbox.freq.secs: 600 >>> >>> nimbus.inbox.jar.expiration.secs: 3600 >>> >>> nimbus.task.launch.secs: 120 >>> >>> nimbus.reassign: true >>> >>> nimbus.file.copy.expiration.secs: 600 >>> >>> nimbus.topology.validator: >>>> "backtype.storm.nimbus.DefaultTopologyValidator" >>> >>> >>>> ### ui.* configs are for the master >>> >>> ui.port: 8080 >>> >>> ui.childopts: "-Xmx768m" >>> >>> >>>> logviewer.port: 8000 >>> >>> logviewer.childopts: "-Xmx128m" >>> >>> logviewer.appender.name: "A1" >>> >>> >>>> >>>> drpc.port: 3772 >>> >>> drpc.worker.threads: 64 >>> >>> drpc.queue.size: 128 >>> >>> drpc.invocations.port: 3773 >>> >>> drpc.request.timeout.secs: 600 >>> >>> drpc.childopts: "-Xmx768m" >>> >>> >>>> transactional.zookeeper.root: "/transactional" >>> >>> transactional.zookeeper.servers: null >>> >>> transactional.zookeeper.port: null >>> >>> >>>> ### supervisor.* configs are for node supervisors >>> >>> # Define the amount of workers that can be run on this machine. Each >>>> worker is assigned a port to use for communication >>> >>> supervisor.slots.ports: >>> >>> - 6700 >>> >>> - 6701 >>> >>> - 6702 >>> >>> - 6703 >>> >>> supervisor.childopts: "-Xmx256m" >>> >>> #how long supervisor will wait to ensure that a worker process is started >>> >>> supervisor.worker.start.timeout.secs: 120 >>> >>> #how long between heartbeats until supervisor considers that worker dead >>>> and tries to restart it >>> >>> supervisor.worker.timeout.secs: 30 >>> >>> #how frequently the supervisor checks on the status of the processes >>>> it's monitoring and restarts if necessary >>> >>> supervisor.monitor.frequency.secs: 3 >>> >>> #how frequently the supervisor heartbeats to the cluster state (for >>>> nimbus) >>> >>> supervisor.heartbeat.frequency.secs: 5 >>> >>> supervisor.enable: true >>> >>> >>>> ### worker.* configs are for task workers >>> >>> worker.childopts: "-Xmx768m" >>> >>> worker.heartbeat.frequency.secs: 1 >>> >>> >>>> task.heartbeat.frequency.secs: 3 >>> >>> task.refresh.poll.secs: 10 >>> >>> >>>> zmq.threads: 1 >>> >>> zmq.linger.millis: 5000 >>> >>> zmq.hwm: 0 >>> >>> >>>> >>>> storm.messaging.netty.server_worker_threads: 1 >>> >>> storm.messaging.netty.client_worker_threads: 1 >>> >>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer >>> >>> storm.messaging.netty.max_retries: 30 >>> >>> storm.messaging.netty.max_wait_ms: 1000 >>> >>> storm.messaging.netty.min_wait_ms: 100 >>> >>> >>>> ### topology.* configs are for specific executing storms >>> >>> topology.enable.message.timeouts: true >>> >>> topology.debug: false >>> >>> topology.optimize: true >>> >>> topology.workers: 1 >>> >>> topology.acker.executors: null >>> >>> topology.tasks: null >>> >>> # maximum amount of time a message has to complete before it's >>>> considered failed >>> >>> topology.message.timeout.secs: 30 >>> >>> topology.skip.missing.kryo.registrations: false >>> >>> topology.max.task.parallelism: null >>> >>> topology.max.spout.pending: null >>> >>> topology.state.synchronization.timeout.secs: 60 >>> >>> topology.stats.sample.rate: 0.05 >>> >>> topology.builtin.metrics.bucket.size.secs: 60 >>> >>> topology.fall.back.on.java.serialization: true >>> >>> topology.worker.childopts: null >>> >>> topology.executor.receive.buffer.size: 1024 #batched >>> >>> topology.executor.send.buffer.size: 1024 #individual messages >>> >>> topology.receiver.buffer.size: 8 # setting it too high causes a lot of >>>> problems (heartbeat thread gets starved, throughput plummets) >>> >>> topology.transfer.buffer.size: 1024 # batched >>> >>> topology.tick.tuple.freq.secs: null >>> >>> topology.worker.shared.thread.pool.size: 4 >>> >>> topology.disruptor.wait.strategy: >>>> "com.lmax.disruptor.BlockingWaitStrategy" >>> >>> topology.spout.wait.strategy: >>>> "backtype.storm.spout.SleepSpoutWaitStrategy" >>> >>> topology.sleep.spout.wait.strategy.time.ms: 1 >>> >>> topology.error.throttle.interval.secs: 10 >>> >>> topology.max.error.report.per.interval: 5 >>> >>> topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory" >>> >>> topology.tuple.serializer: >>>> "backtype.storm.serialization.types.ListDelegateSerializer" >>> >>> topology.trident.batch.emit.interval.millis: 500 >>> >>> >>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper" >>> >>> >> >> bin/storm nimbus & >> >> bin/storm supervisor & >> >> bin/storm ui & >> >> *I launch another basic EC2 CentOS machine and do the exact same commands >> except that I don't install zookeeper and I only run "bin/storm nimbus &" >> at the end* >> >> Any thoughts would be greatly appreciated. Thanks so much for all of your >> help. I'm sure someone else has done this before! >> >> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <[email protected]> >> wrote: >> >>> Although, implementing the Storm cluster manually would be really nice >>> learning, I would suggest using HortonWorks distribution which comes with >>> Storm as OOTB solution and you can configure everything from Ambari UI. We >>> are using Storm on Amazon EC2 machine, though it is right now in beta >>> stage. We are going to move to production in coming 2-3 months. >>> >>> >>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell < >>> [email protected]> wrote: >>> >>>> All, >>>> >>>> I implemented the suggestions given by Parh and Harsha. I am now >>>> using the default.yaml but I changed the storm.zookeeper.servers to the >>>> nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host >>>> to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives >>>> the following error: org.apache.thrift7.transport.TTransportException: >>>> java.net.ConnectException: Connection refused >>>> >>>> You should be able to see the error it gives by going to the web page >>>> yourself at: http://54.68.149.181:8080. I am only using this account >>>> to test and see if I can even get storm to work, so these machines are only >>>> for testing. Perhaps someone could tell me what the storm.yaml file should >>>> look like for this setup? >>>> >>>> -Thanks, Stephne >>>> >>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell < >>>> [email protected]> wrote: >>>> >>>>> I'm getting kind of confused by the storm.yaml file. Should I be using >>>>> the default.yaml and just modify the zookeeper and nimbus ip, or should I >>>>> use a bran new storm.yaml? >>>>> >>>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is >>>>> on the nimbus machine. what should the storm.yaml look like on my worker >>>>> and nimbus machine? Will the storm.yaml be the same on my worker and >>>>> nimbus >>>>> machine? I am not trying to do anything fancy, I am just trying to get a >>>>> very basic cluster up and running. >>>>> >>>>> -Thanks, Stephen >>>>> >>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell < >>>>> [email protected]> wrote: >>>>> >>>>>> All Thanks so much for your help. I cannot tell you how much I >>>>>> appreciate it. I'm going to try out your suggestions and keep banging my >>>>>> head again the wall : D. I've spent an enormous amount of time trying to >>>>>> get this to work. I'll let you know what happens after I try to implement >>>>>> your suggestions. It would be really cool if someone had a tutorial that >>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It >>>>>> seems like trying to get a two-machine cluster setup on AWS would be a >>>>>> VERY >>>>>> common use-case. I've read and watched everything I can on the topic and >>>>>> nothing got it working for me! >>>>>> >>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> The worker connects to the thrift port and not the ui port. You need >>>>>>> to open port 6627 or whatever is the value being set in storm.yaml using >>>>>>> property “nimbus.thrift.port”. >>>>>>> >>>>>>> Based on the configuration that you have pointed so far it seems >>>>>>> your nimbus host has nimbus,ui,supervisor working because you actually >>>>>>> have >>>>>>> zookeeper running locally on that host. As Harsha pointed out you need >>>>>>> to >>>>>>> change it to a value that is the public ip instead of loopback >>>>>>> interface. >>>>>>> >>>>>>> Thanks >>>>>>> Parth >>>>>>> >>>>>>> >>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <[email protected]> wrote: >>>>>>> >>>>>>> storm.zookeeper.servers: >>>>>>> - "127.0.0.1" >>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback >>>>>>> interface , instead either use your public ip or 0.0.0.0*) >>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a >>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if >>>>>>> your >>>>>>> machine is restarted)* >>>>>>> >>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip >>>>>>> not 127.0.0.1. >>>>>>> >>>>>>> "No, I cannot ping my host which has a public ip address of >>>>>>> 54.68.149.181" >>>>>>> you are not able to reach this ip form worker node but able to >>>>>>> access the UI using it? >>>>>>> -Harsha >>>>>>> >>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote: >>>>>>> >>>>>>> Harsha, >>>>>>> >>>>>>> The storm.yaml on the host machine looks like this: >>>>>>> >>>>>>> storm.zookeeper.servers: >>>>>>> - "127.0.0.1" >>>>>>> >>>>>>> >>>>>>> nimbus.host: "127.0.0.1" >>>>>>> >>>>>>> storm.local.dir: /tmp/storm >>>>>>> >>>>>>> >>>>>>> The storm.yaml on the worker machine looks like this: >>>>>>> >>>>>>> storm.zookeeper.servers: >>>>>>> - "54.68.149.181" >>>>>>> >>>>>>> >>>>>>> nimbus.host: "54.68.149.181" >>>>>>> >>>>>>> storm.local.dir: /tmp/storm >>>>>>> >>>>>>> No, I cannot ping my host which has a public ip address of >>>>>>> 54.68.149.181 although I can connect to the UI web page when it is >>>>>>> hosted. >>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus >>>>>>> host. >>>>>>> -Thanks, Stephen >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <[email protected]> wrote: >>>>>>> >>>>>>> >>>>>>> There aren't any errors in worker machine supervisor logs. Are you >>>>>>> using the same storm.yaml for both the machines and also are you able to >>>>>>> ping your nimbus host or connect to zookeeper on nimbus host. >>>>>>> -Harsha >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote: >>>>>>> >>>>>>> Harsha, >>>>>>> >>>>>>> Thanks so much for getting back with me. I will check the logs, >>>>>>> but I don't seem to get any error messages. I have a nimbus AWS machine >>>>>>> with zookeeper on it and a worker AWS machine. >>>>>>> >>>>>>> On the nimbus machine I start the zookeeper and then I run: >>>>>>> >>>>>>> bin/storm nimbus & >>>>>>> bin/storm supervisor & >>>>>>> bin/storm ui >>>>>>> >>>>>>> On the worker machine I run: >>>>>>> bin/storm supervisor >>>>>>> >>>>>>> When I go to the UI page, I only see 1 supervisor (the one on the >>>>>>> nimbus machine). So apparently, the worker machine isn't "registering" >>>>>>> with >>>>>>> the nimbus machine. >>>>>>> >>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <[email protected]> wrote: >>>>>>> >>>>>>> >>>>>>> Hi Stephen, >>>>>>> What are the issues you are seeing. >>>>>>> "How do worker machines "know" how to connect to nimbus? Is it in >>>>>>> the storm configuration file" >>>>>>> >>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able >>>>>>> to connect to your zookeeper cluster. >>>>>>> Check your logs under storm_inst/logs/ for any errors when you try >>>>>>> to start nimbus or supervisors. >>>>>>> If you are installing it manually try following these steps if you >>>>>>> are not already done. >>>>>>> >>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/ >>>>>>> -Harsha >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote: >>>>>>> >>>>>>> All, >>>>>>> >>>>>>> I would greatly appreciate any help that anyone would afford. I've >>>>>>> been trying to setup a storm cluster on AWS for a few weeks now on >>>>>>> centOS >>>>>>> EC2 machines. So far, I haven't been able to get a cluster built. I can >>>>>>> get >>>>>>> a supervisor and nimbus to run on a single machine, but I can't figure >>>>>>> out >>>>>>> how to get another worker to connect to nimbus. How do worker machines >>>>>>> "know" how to connect to nimbus? Is it in the storm configuration file? >>>>>>> I've gone through many tutorials and the official documentation, but >>>>>>> this >>>>>>> point doesn't seem to be covered anywhere in sufficient detail for a new >>>>>>> guy like me. >>>>>>> >>>>>>> Some of you may be tempted to point me toward storm-deploy, but I >>>>>>> spent four days trying to get that to work until I gave up. I'm having >>>>>>> Issue #58 on github. Following the instructions exactly and other >>>>>>> tutorials >>>>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and >>>>>>> decided >>>>>>> to try and setup a cluster manually. Thanks in advance to anyone >>>>>>> willing to >>>>>>> offer me any inputs you can! >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> CONFIDENTIALITY NOTICE >>>>>>> NOTICE: This message is intended for the use of the individual or >>>>>>> entity to which it is addressed and may contain information that is >>>>>>> confidential, privileged and exempt from disclosure under applicable >>>>>>> law. >>>>>>> If the reader of this message is not the intended recipient, you are >>>>>>> hereby >>>>>>> notified that any printing, copying, dissemination, distribution, >>>>>>> disclosure or forwarding of this communication is strictly prohibited. >>>>>>> If >>>>>>> you have received this communication in error, please contact the sender >>>>>>> immediately and delete it from your system. Thank You. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> Vikas Agarwal >>> 91 – 9928301411 >>> >>> InfoObjects, Inc. >>> Execution Matters >>> http://www.infoobjects.com >>> 2041 Mission College Boulevard, #280 >>> Santa Clara, CA 95054 >>> +1 (408) 988-2000 Work >>> +1 (408) 716-2726 Fax >>> >>> >> > > > -- > Regards, > Vikas Agarwal > 91 – 9928301411 > > InfoObjects, Inc. > Execution Matters > http://www.infoobjects.com > 2041 Mission College Boulevard, #280 > Santa Clara, CA 95054 > +1 (408) 988-2000 Work > +1 (408) 716-2726 Fax > >
