I really don't think you can have any expectation of this doing anything without somehow shoving work into it. Look at all the things being done in the "local cluster" block versus the "remote" block of your code.
Would be nice if someone who has ever used DRPC could respond. The docs seem to be lacking in this critical area of getting it to work with a remote cluster. i.e., the 3 steps definitely don't seem to be sufficient. - Erik On Wed, Jan 27, 2016 at 3:16 PM, researcher cs <[email protected]> wrote: > thanks but Agian i imported this project from github that it's supposed > work well , really i'm on this problem for months without fixing it !! i > sent coder message but he didn't reply with this problem . don't know who > can i ask but thanks for your time > > On Thu, Jan 28, 2016 at 1:12 AM, Erik Weathers < > [email protected]> wrote: > > > hey Sam, > > > > Again, I've never used the DRPC feature. But it naively looks like you > > aren't using it correctly. > > Note that the working case for you is with a LocalCluster & LocalDRPC, > and > > you are explicitly invoking the execute(): > > String result = drpc.execute(TOPOLOGY_NAME, tweetJson); > > > > In the remote case you aren't doing anything except submitting the > > topology. I assume you need to invoke drpc.execute *somewhere*... Here's > > what I think is the same question as you are posing: > > > > - http://stackoverflow.com/a/26440260/318428 > > > > - Erik > > > > On Wed, Jan 27, 2016 at 3:04 PM, researcher cs < > [email protected]> > > wrote: > > > > > also got zeros after open storm ui except executed column > > > > > > > > > > > > > > > On Thu, Jan 28, 2016 at 12:43 AM, researcher cs < > > [email protected] > > > > wrote: > > > > > >> i atteached what i got when i submitted topology and this happened > also > > >> when i submitted > storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar > > >> storm.starter.BasicDRPCTopology basic-drpc > > >> > > >> this storm.yaml > > >> storm.zookeeper.servers: > > >> - "localhost" > > >> # - "server2" > > >> nimbus.host: "localhost" > > >> > > >> storm.local.dir: "/var/storm" > > >> supervisor.slots.ports: > > >> - 7660 > > >> - 7659 > > >> - 7658 > > >> - 7657 > > >> supervisor.childopts: "-Djava.net.preferIPv4Stack=true" > > >> nimbus.childopts: "-Djava.net.preferIPv4Stack=true" > > >> worker.childopts: "-Djava.net.preferIPv4Stack=true" > > >> # topology.message.timeout.secs: 30 > > >> # topology.workers: 1 > > >> topology.stats.sample.rate: 1.0 > > >> topology.acker.executors: 1 > > >> # topology.executor.receive.buffer.size: 16384 > > >> # topology.executor.send.buffer.size: 16384 > > >> # topology.transfer.buffer.size: 32 > > >> # topology.receiver.buffer.size: 8 > > >> # > > >> # ##### These may optionally be filled in: > > >> # > > >> ## List of custom serializations > > >> # topology.kryo.register: > > >> # - org.mycompany.MyType > > >> # - org.mycompany.MyType2: org.mycompany.MyType2Serializer > > >> # > > >> ## List of custom kryo decorators > > >> # topology.kryo.decorators: > > >> # - org.mycompany.MyDecorator > > >> # > > >> ## Locations of the drpc servers > > >> # drpc.servers: > > >> # - "localhost" > > >> # - "server2" > > >> > > >> ## Metrics Consumers > > >> # topology.metrics.consumer.register: > > >> # - class: "backtype.storm.metrics.LoggingMetricsConsumer" > > >> # parallelism.hint: 1 > > >> # - class: "org.mycompany.MyMetricsConsumer" > > >> # parallelism.hint: 1 > > >> # argument: > > >> # - endpoint: "metrics-collector.mycompany.org" > > >> storm.messaging.transport: "backtype.storm.messaging.netty.Context" > > >> storm.messaging.netty.server_worker_threads: 1 > > >> storm.messaging.netty.client_worker_threads: 1 > > >> storm.messaging.netty.buffer_size: 5242880 > > >> storm.messaging.netty.max_retries: 100 > > >> storm.messaging.netty.max_wait_ms: 1000 > > >> storm.messaging.netty.min_wait_ms: 100 > > >> > > >> this etc/hosts > > >> 127.0.0.1 localhost > > >> 127.0.1.1 user-Lenovo-G50-70 > > >> ##The following lines are desirable for IPv6 capable hosts## > > >> ::1 ip6-localhost ip6-loopback > > >> fe00::0 ip6-localnet > > >> ff00::0 ip6-mcastprefix > > >> ff02::1 ip6-allnodes > > >> ff02::2 ip6-allrouters > > >> > > >> i set drpc in code > > >> private static Config createTopologyConfiguration(Properties prop, > > >> boolean localMode) { > > >> Config conf = new Config(); > > >> List<String> dprcServers = new ArrayList<String>(); > > >> dprcServers.add("localhost"); > > >> > > >> conf.put(Config.DRPC_SERVERS, dprcServers); > > >> conf.put(Config.DRPC_PORT, 3772); > > >> if (!localMode) > > >> conf.put(Config.STORM_CLUSTER_MODE, new > > >> String("distributed")); > > >> > > >> } > > >> public static void main(String[] args) throws Exception { > > >> if (args == null || args.length == 0) { > > >> Config conf = createTopologyConfiguration(prop, true); > > >> LocalDRPC drpc = new LocalDRPC(); > > >> LocalCluster cluster = new LocalCluster(); > > >> > > >> cluster.submitTopology(TOPOLOGY_NAME, conf, > > >> buildTopology(drpc)); > > >> > > >> Thread.sleep(2000); // give it some time to setup > > >> > > >> BufferedReader br = new BufferedReader(new > > >> FileReader(queryFile)); > > >> String tweetJson; > > >> fos.write("Start: ".getBytes()); > > >> > > >> fos.write(String.valueOf(System.currentTimeMillis()).getBytes()); > > >> byte[] newLine = "\n".getBytes(); > > >> int times = 0; > > >> // emit tweets into topology > > >> while ((tweetJson = br.readLine()) != null) { > > >> > > >> String result = drpc.execute(TOPOLOGY_NAME, > tweetJson); > > >> > > >> Status s = null; > > >> try { > > >> s = DataObjectFactory.createStatus(tweetJson); > > >> result = s.getId() + "\t" + s.getText() + "\t" + > > >> result; > > >> } catch (TwitterException e) { > > >> LOG.error(e.toString()); > > >> } > > >> > > >> fos.write(result.getBytes()); > > >> fos.write(newLine); > > >> > > >> // times++; > > >> // if (times == 1000) > > >> // break; > > >> } > > >> fos.write(newLine); > > >> fos.write("Finish: ".getBytes()); > > >> > > >> fos.write(String.valueOf(System.currentTimeMillis()).getBytes()); > > >> > > >> fos.flush(); > > >> fos.close(); > > >> br.close(); > > >> drpc.shutdown(); > > >> cluster.shutdown(); > > >> } else { > > >> // distributed mode > > >> Config conf = createTopologyConfiguration(prop,true); > > >> LocalDRPC drpc = null; > > >> StormSubmitter.submitTopology(args[0], conf, > > >> buildTopology(drpc)); > > >> > > >> } > > >> > > >> } > > >> > > >> On Wed, Jan 27, 2016 at 11:14 PM, Erik Weathers < > > >> [email protected]> wrote: > > >> > > >>> Please put more effort into describing the issue. "It doesn't work" > is > > >>> unfortunately not enough info for anyone to provide help. > > >>> e.g., post links to some code you are trying to run, and the configs > of > > >>> the > > >>> storm components that you are running. > > >>> > > >>> - Erik > > >>> > > >>> On Wed, Jan 27, 2016 at 4:59 AM, sam mohel <[email protected]> > > wrote: > > >>> > > >>> > I wrote the actual problem in My first message drpc server not > > working > > >>> I > > >>> > hot zeros in the columns of storm ui like emitted and transferred , > > >>> result > > >>> > file is empty > > >>> > > > >>> > On Wednesday, January 27, 2016, Erik Weathers > > >>> > <[email protected]> > > >>> > wrote: > > >>> > > > >>> > > Your mail client is wrapping the log lines prematurely, I have a > > >>> really > > >>> > > really hard time reading wrapped lines, I'd look into fixing that > > if > > >>> I > > >>> > were > > >>> > > you. Here they are unwrapped: > > >>> > > > > >>> > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] > Starting > > >>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client > > >>> connection, > > >>> > > connectString=localhost:2181 sessionTimeout=20000 > > >>> > > watcher=com.netflix.curator.ConnectionState@2fa423d2 > > >>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket > > >>> connection to > > >>> > > server localhost/127.0.1.1:2181. Will not attempt to > authenticate > > >>> using > > >>> > > SASL (unknown error) > > >>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection > > >>> established > > >>> > > to localhost/127.0.1.1:2181, initiating session > > >>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment > > >>> > complete > > >>> > > on server localhost/127.0.1.1:2181, sessionid = > 0x152804f3a3a0002, > > >>> > > negotiated timeout = 20000 > > >>> > > 2016-01-27 01:41:00 b.s.zookeeper [INFO] Zookeeper state update: > > >>> > > :connected:none > > >>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Session: > > 0x152804f3a3a0002 > > >>> > > closed > > >>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] EventThread shut down > > >>> > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] > Starting > > >>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client > > >>> connection, > > >>> > > connectString=localhost:2181/storm sessionTimeout=20000 > > >>> > > watcher=com.netflix.curator.ConnectionState@2ee8b0bf > > >>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket > > >>> connection to > > >>> > > server user-Lenovo-G50-70/127.0.0.1:2181. Will not attempt to > > >>> > authenticate > > >>> > > using SASL (unknown error) > > >>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection > > >>> established > > >>> > > to user-Lenovo-G50-70/127.0.0.1:2181, initiating session > > >>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment > > >>> > complete > > >>> > > on server user-Lenovo-G50-70/127.0.0.1:2181, sessionid = > > >>> > > 0x152804f3a3a0003, > > >>> > > negotiated timeout = 20000 > > >>> > > > > >>> > > None of those indicate a problem, they look pretty standard to > me. > > >>> > > > > >>> > > Please spend a bit more time zeroing in on what the actual > problem > > >>> is so > > >>> > > that the members of the list(s) can provide help. > > >>> > > > > >>> > > - Erik > > >>> > > > > >>> > > On Tue, Jan 26, 2016 at 10:15 PM, researcher cs < > > >>> > [email protected] > > >>> > > <javascript:;>> > > >>> > > wrote: > > >>> > > > > >>> > > > yes i tried in local and worked well > > >>> > > > and about /etc/hosts . i'm feeling that this file has a mistake > > , i > > >>> > made > > >>> > > > alot of changes in this file and didn't remember what was > default > > >>> > > > > > >>> > > > In nimbus log file when it connected zookeeper i got > > >>> > > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] > > Starting > > >>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client > > >>> > connection, > > >>> > > > connectString=localhost:2181 sessionTimeout=20000 > > >>> > > > watcher=com.netflix.curator.ConnectionState@2fa423d2 > > >>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket > > >>> connection > > >>> > to > > >>> > > > server localhost/127.0.1.1:2181. Will not attempt to > > authenticate > > >>> > using > > >>> > > > SASL (unknown error) > > >>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection > > >>> > established > > >>> > > > to localhost/127.0.1.1:2181, initiating session > > >>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session > establishment > > >>> > > complete > > >>> > > > on server localhost/127.0.1.1:2181, sessionid = > > 0x152804f3a3a0002, > > >>> > > > negotiated timeout = 20000 > > >>> > > > 2016-01-27 01:41:00 b.s.zookeeper [INFO] Zookeeper state > update: > > >>> > > > :connected:none > > >>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Session: > > >>> 0x152804f3a3a0002 > > >>> > > > closed > > >>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] EventThread shut > down > > >>> > > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] > > Starting > > >>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client > > >>> > connection, > > >>> > > > connectString=localhost:2181/storm sessionTimeout=20000 > > >>> > > > watcher=com.netflix.curator.ConnectionState@2ee8b0bf > > >>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket > > >>> connection > > >>> > to > > >>> > > > server user-Lenovo-G50-70/127.0.0.1:2181. Will not attempt to > > >>> > > authenticate > > >>> > > > using SASL (unknown error) > > >>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection > > >>> > established > > >>> > > > to user-Lenovo-G50-70/127.0.0.1:2181, initiating session > > >>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session > establishment > > >>> > > complete > > >>> > > > on server user-Lenovo-G50-70/127.0.0.1:2181, sessionid = > > >>> > > > 0x152804f3a3a0003, > > >>> > > > negotiated timeout = 20000 > > >>> > > > > > >>> > > > and i set for drpc.server : localhost > > >>> > > > storm.zookeeper.server : localhost > > >>> > > > nimbus.host : localhost > > >>> > > > > > >>> > > > in my /etc/hosts > > >>> > > > 127.0.0.1 user-Lenovo-G50-70 localhost > > >>> > > > 127.0.1.1 localhost > > >>> > > > > > >>> > > > is that right ? > > >>> > > > > > >>> > > > > > >>> > > > On Wed, Jan 27, 2016 at 6:12 AM, Erik Weathers < > > >>> > > > [email protected]> wrote: > > >>> > > > > > >>> > > > > You said: "except the statement of drpc server trying to > > connect" > > >>> > > > > > > >>> > > > > Maybe you are confused about what "b.s.d.drpc [INFO] Starting > > >>> > > Distributed > > >>> > > > > RPC servers..." implies? > > >>> > > > > That is just saying that the server is being started. It's > a > > >>> > server, > > >>> > > > not > > >>> > > > > a client, so it's basic operation is *not* to connect to some > > >>> other > > >>> > > > thing. > > >>> > > > > It's up and waiting for you to tell it to do stuff. > > >>> > > > > > > >>> > > > > Have you gotten the Local-mode version of DRPC working? > > >>> > > > > > > >>> > > > > - > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > > https://storm.apache.org/documentation/Distributed-RPC.html#local-mode-drpc > > >>> > > > > > > >>> > > > > Not sure what you're asking with regards to DNS and > /etc/hosts, > > >>> those > > >>> > > > seem > > >>> > > > > unrelated to your basic issue. > > >>> > > > > > > >>> > > > > - Erik > > >>> > > > > > > >>> > > > > On Tue, Jan 26, 2016 at 6:58 PM, researcher cs < > > >>> > > > [email protected] <javascript:;>> > > >>> > > > > wrote: > > >>> > > > > > > >>> > > > > > There is no error except the statement of drpc server > trying > > to > > >>> > > connect > > >>> > > > > . I > > >>> > > > > > guess I have problem with Dns . if you have any idea about > > this > > >>> > > pleaes > > >>> > > > > help > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > I want to submit topology with single machine > > >>> > > > > > Only on my laptop without any other devices as a first step > > so > > >>> > > > > > What this file /etc/hosts should contains ? > > >>> > > > > > As I set in drpc.server : localhost > > >>> > > > > > Storm.zookeeper. server: localhost > > >>> > > > > > Nimbus.host:localhost > > >>> > > > > > > > >>> > > > > > As this file contains 127.0.1.1 and 127.0.0.1 and my IP > > >>> address ? > > >>> > > > > > > > >>> > > > > > What should I use and what should I hash it to not using > it ? > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > On Wednesday, January 27, 2016, Erik Weathers > > >>> > > > > > <[email protected]> > > >>> > > > > > wrote: > > >>> > > > > > > > >>> > > > > > > What does the client code that is supposed to make the > DRPC > > >>> > > > connection > > >>> > > > > > > telling you? i.e., you should see some exception or log > > >>> about > > >>> > not > > >>> > > > > > > establishing the connection, right? > > >>> > > > > > > > > >>> > > > > > > Alternatively, perhaps the connections aren't persistent > > and > > >>> > > there's > > >>> > > > no > > >>> > > > > > > actual problem? > > >>> > > > > > > > > >>> > > > > > > - Erik > > >>> > > > > > > > > >>> > > > > > > On Tue, Jan 26, 2016 at 4:55 PM, researcher cs < > > >>> > > > > > [email protected] <javascript:;> > > >>> > > > > > > <javascript:;>> > > >>> > > > > > > wrote: > > >>> > > > > > > > > >>> > > > > > > > thanks for replying , i read the documentation before > , > > i > > >>> > > imported > > >>> > > > > > > project > > >>> > > > > > > > supposed to work well but not working with me > > >>> > > > > > > > i checked port by lsof -i gave me all ports i connected > > it > > >>> for > > >>> > > > storm > > >>> > > > > > > > java 10675 root 20u IPv4 98126 0t0 TCP > > >>> *:52022 > > >>> > > > > > (LISTEN) > > >>> > > > > > > > java 10675 root 26u IPv4 98131 0t0 TCP > > >>> *:2181 > > >>> > > > > (LISTEN) > > >>> > > > > > > > java 10675 root 27u IPv4 101944 0t0 TCP > > >>> > > > > > > > localhost:2181->user-Lenovo-G50-70:38150 (ESTABLISHED) > > >>> > > > > > > > java 10675 root 29u IPv4 98974 0t0 TCP > > >>> > > > > > > > user-Lenovo-G50-70:2181->user-Lenovo-G50-70:50526 > > >>> (ESTABLISHED) > > >>> > > > > > > > java 10675 root 30u IPv4 99105 0t0 TCP > > >>> > > > > > > > localhost:2181->user-Lenovo-G50-70:38165 (ESTABLISHED) > > >>> > > > > > > > java 10715 root 90u IPv4 98953 0t0 TCP > > >>> > > > > > > > user-Lenovo-G50-70:38150->localhost:2181 (ESTABLISHED) > > >>> > > > > > > > java 10715 root 91u IPv4 98245 0t0 TCP > > >>> *:6627 > > >>> > > > > (LISTEN) > > >>> > > > > > > > java 10792 root 90u IPv4 99973 0t0 TCP > > >>> > > > > > > > user-Lenovo-G50-70:50526->user-Lenovo-G50-70:2181 > > >>> (ESTABLISHED) > > >>> > > > > > > > java 10864 root 82u IPv4 102425 0t0 TCP > > >>> *:3772 > > >>> > > > > (LISTEN) > > >>> > > > > > > > java 10864 root 84u IPv4 102429 0t0 TCP > > >>> *:3773 > > >>> > > > > (LISTEN) > > >>> > > > > > > > java 10864 root 92u IPv4 102197 0t0 TCP > > >>> > > > > > > > user-Lenovo-G50-70:3773->user-Lenovo-G50-70:50825 > > >>> (ESTABLISHED) > > >>> > > > > > > > java 10928 root 81u IPv4 102070 0t0 TCP > > >>> > > *:http-alt > > >>> > > > > > > (LISTEN) > > >>> > > > > > > > java 11087 root 81u IPv4 100091 0t0 TCP > > >>> > > > > > > > user-Lenovo-G50-70:50825->user-Lenovo-G50-70:3773 > > >>> (ESTABLISHED) > > >>> > > > > > > > java 11087 root 91u IPv4 102196 0t0 TCP > > >>> > > > > > > > user-Lenovo-G50-70:38165->localhost:2181 (ESTABLISHED) > > >>> > > > > > > > java 11087 root 94u IPv4 102561 0t0 TCP > > >>> *:7660 > > >>> > > > > (LISTEN) > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > here you can see that 3772 not established > > >>> > > > > > > > > > >>> > > > > > > > On Wed, Jan 27, 2016 at 2:47 AM, Erik Weathers < > > >>> > > > > > > > [email protected]> wrote: > > >>> > > > > > > > > > >>> > > > > > > > > hey, > > >>> > > > > > > > > > > >>> > > > > > > > > The DRPC server is up and listening on port 3772. > Why > > >>> do > > >>> > you > > >>> > > > > expect > > >>> > > > > > > > > established connections? > > >>> > > > > > > > > > > >>> > > > > > > > > I'm not familiar with using Storm's DRPC feature, but > > I'm > > >>> > sure > > >>> > > > you > > >>> > > > > > need > > >>> > > > > > > > to > > >>> > > > > > > > > write code that interacts with the DRPC server, and > > >>> you've > > >>> > made > > >>> > > > no > > >>> > > > > > > > mention > > >>> > > > > > > > > of doing so in your email. I'd start here: > > >>> > > > > > > > > > > >>> > > > > > > > > - > > >>> > > https://storm.apache.org/documentation/Distributed-RPC.html > > >>> > > > > > > > > > > >>> > > > > > > > > - Erik > > >>> > > > > > > > > > > >>> > > > > > > > > On Tue, Jan 26, 2016 at 4:29 PM, researcher cs < > > >>> > > > > > > > [email protected] <javascript:;> > <javascript:;>> > > >>> > > > > > > > > wrote: > > >>> > > > > > > > > > > >>> > > > > > > > > > I set in the code > > >>> > > > > > > > > > conf.put(Config.DRPC_SERVERS, dprcServers); > > >>> > > > > > > > > > conf.put(Config.DRPC_PORT, 3772); > > >>> > > > > > > > > > but when i submit topolgoy i found at the end of > the > > >>> file > > >>> > > > > > > > > > b.s.d.drpc [INFO] Starting Distributed RPC > > servers... > > >>> > > > > > > > > > > > >>> > > > > > > > > > i checked port 3772 by > > >>> > > > > > > > > > > > >>> > > > > > > > > > sudo netstat -ap | grep 3772 > > >>> > > > > > > > > > > > >>> > > > > > > > > > i got > > >>> > > > > > > > > > > > >>> > > > > > > > > > tcp 0 0 *:3772 *:* LISTEN > > >>> 10864/java > > >>> > > > > > > > > > unix 3 [ ] STREAM CONNECTED > > 13772 > > >>> > > > > > > > > > 587/dbus-daemon /var/run/dbus/system_bus_socket > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > why it's not established ? > > >>> > > > > > > > > > > > >>> > > > > > > > > > can i find help ? > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> > > >> > > > > > >
