We are running in single server . We have everything in same server . Would remote work only if we have multiple machines in place ?
From: Vikas Agarwal [mailto:[email protected]] Sent: 09 September 2014 12:57 To: [email protected] Subject: Re: Running Topology on Remote Machine I stuck there for 2 weeks. :) Basically, I tried to debug every command that storm processes are trying to run. You can manually run the worker java command on console to check what error it is throwing. In our case there were some problems, few of them are: 1) slf4j jar conflicts 2) somehow wrong value of hostname in storm worker's configuration 3) something wrong in bolt's prepare method or spout's open method (may be waiting for some connection outside of storm) 4) ports being blocked by firewall (i added few ports in Amazon EC2 security groups to make it runnable) On Tue, Sep 9, 2014 at 12:48 PM, Pavan Jakati G <[email protected]> wrote: How did you fix it Vikas . Can you help us here . We are stuck here since a week . L From: Vikas Agarwal [mailto:[email protected]] Sent: 09 September 2014 12:39 To: [email protected] Subject: Re: Running Topology on Remote Machine We faced similar errors. May be it can help http://mail-archives.apache.org/mod_mbox/incubator-storm-user/201408.mbox/browser On Tue, Sep 9, 2014 at 11:02 AM, Pavan Jakati G <[email protected]> wrote: Worker log 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt __system:(-1) 2014-09-08 07:19:09 b.s.d.executor [INFO] Loading executor SOIMbolt:[1 1] 2014-09-08 07:19:09 b.s.d.executor [INFO] Loaded executor tasks SOIMbolt:[1 1] 2014-09-08 07:19:09 b.s.d.executor [INFO] Finished loading executor SOIMbolt:[1 1] 2014-09-08 07:19:09 b.s.d.executor [INFO] Preparing bolt SOIMbolt:(1) 2014-09-08 07:19:09 b.s.d.worker [INFO] Launching receive-thread for 51fb6b55-0522-437c-a18e-72212192db0f:6703 2014-09-08 07:19:09 b.s.d.executor [INFO] Prepared bolt SOIMbolt:(1) 2014-09-08 07:19:09 b.s.m.n.Server [INFO] Create Netty Server Netty-server-localhost-6703, buffer_size: 5242880, maxWorkers: 1 2014-09-08 07:19:09 b.s.m.loader [INFO] Starting receive-thread: [stormId: PaVan-1-1410160741, port: 6703, thread-id: 0 ] 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker has topology config {"storm.id" "PaVan-1-1410160741", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "drpc.servers" ["192.168.10.7"], "storm.messaging.netty.flush.check.interval.ms" 10, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "storm-local", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "192.168.10.7", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["192.168.10.7"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.kryo.decorators" (), "topology.name" "PaVan", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-Xmx768m", "supervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 30, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "nimbus.thrift.max_buffer_size" 1048576, "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-Xmx256m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "topology.multilang.serializer" "backtype.storm.multilang.JsonSerializer", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "topology.worker.receiver.thread.count" 1, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "storm.cluster.mode" "distributed", "topology.max.task.parallelism" nil, "storm.messaging.netty.transfer.batch.size" 262144} 2014-09-08 07:19:09 b.s.d.worker [INFO] Worker 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 for storm PaVan-1-1410160741 on 51fb6b55-0522-437c-a18e-72212192db0f:6703 has finished loading Supervisor Log: b/json-simple-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.6.jar:/usr/local/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/usr/local/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/usr/local/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/usr/local/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/usr/local/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-1-1410160741/stormjar.jar' 'backtype.storm.daemon.worker' 'PaVan-1-1410160741' '51fb6b55-0522-437c-a18e-72212192db0f' '6703' '2c12526f-d588-4dcc-99b9-b26e8c1d1d70' 2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:02 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:03 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:04 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:05 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:06 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:07 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started 2014-09-08 07:19:08 b.s.d.supervisor [INFO] 2c12526f-d588-4dcc-99b9-b26e8c1d1d70 still hasn't started From: Raghavendra Nandagopal [mailto:[email protected]] Sent: 09 September 2014 10:59 To: [email protected] Subject: RE: Running Topology on Remote Machine Hi Sridhar, Can you check the worker logs and see if there are any exceptions? Thanks, Raghav On Sep 8, 2014 10:25 PM, "Sridhar G" <[email protected]> wrote: Hi Dhaval, We have tried with the below config but no luck . Anything else to add to it ? Regards Sridhar Gadi -----Original Message----- From: Dhaval Modi [mailto:[email protected]] Sent: Tue 9/9/2014 9:45 AM To: user Cc: Pavan Jakati G Subject: Re: Running Topology on Remote Machine Hi Sridhar, Did you try commenting " conf.setMaxTaskParallelism(10);" ? Regards, Dhaval Modi [email protected] On 9 September 2014 09:13, Sridhar G <[email protected]> wrote: > > > > > Hi All, > > > > We are stuck with running our topology on Remote Machine. We have be able > to run our topology on local machine which emits tuples. But when we try to > run it on Remote Machine it does not emit tuples. Whereas the Formation of > Topology is shown on STORM UI under TOPOLOGY SUMMARY. TOPOLOGY STATS , > SPOUTS and BOLTS are visible with ZERO emits. > > > > The Code which we have used for Running Topology on REMOTE Machine is: > > > > LIBRARIES IMPORTS ARE - > > > > import backtype.storm.Config; > > import backtype.storm.LocalCluster; > > import backtype.storm.topology.TopologyBuilder; > > import backtype.storm.utils.Utils; > > > > > > import org.json.simple.JSONValue; > > import backtype.storm.StormSubmitter; > > import backtype.storm.generated.AlreadyAliveException; > > import backtype.storm.generated.Nimbus.Client; > > import backtype.storm.topology.TopologyBuilder; > > import backtype.storm.utils.NimbusClient; > > import java.util.Map; > > > ------------------------------------------------------------------------------------------------------------------------- > > *TOPOLOGY PROGRAM -* > > > > TopologyBuilder builder = new TopologyBuilder(); > > builder.setSpout("POCspout", new POCSpout(),3); > > builder.setBolt("POCbolt", new > POCBolt(),3).shuffleGrouping("POCspout"); > > > > Config conf = new Config(); > > conf.put(Config.NIMBUS_HOST, "192.xxx.xx.x"); > > conf.setDebug(true); > > conf.setNumWorkers(1); > > conf.setMaxSpoutPending(5000); > > conf.setNumAckers(1); > > conf.setMaxTaskParallelism(10); > > Map storm_conf = Utils.readStormConfig(); > > > > storm_conf.put("nimbus.host", "192.xxx.xx.x"); > > Client client = > NimbusClient.getConfiguredClient(storm_conf).getClient(); > > > > String inputJar = > "/root/SOIM_PaVan/target/SOIMTopology-1.0-SNAPSHOT.jar"; > > NimbusClient nimbus = new NimbusClient(storm_conf, > "192.xxx.xx.x",6627); > > String uploadedJarLocation = > StormSubmitter.submitJar(storm_conf,inputJar); > > > > try { > > String jsonConf = JSONValue.toJSONString(storm_conf); > > > > nimbus.getClient().submitTopology("TOPOLOGYPOC",uploadedJarLocation,jsonConf,builder.createTopology()); > > } > > catch (AlreadyAliveException ae) > > { > > ae.printStackTrace(); > > } > > > > > > > > *For any Correction in code let us know.* > > > > > > Regards, > > Sridhar > > *Believe in the impossible and remove the improbable* > > The information contained in this transmission may contain privileged and > confidential information of Microland Limited, including information > protected by privacy laws. It is intended only for the use of Microland > Limited. If you are not the intended recipient, you are hereby notified > that any review, dissemination, distribution, or duplication of this > communication is strictly prohibited. If you are not the intended > recipient, please contact the sender by reply email and destroy all copies > of the original message. > The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 <tel:%2B1%20%28408%29%20988-2000> Work +1 (408) 716-2726 <tel:%2B1%20%28408%29%20716-2726> Fax -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
