hey Sam,
Again, I've never used the DRPC feature. But it naively looks like you
aren't using it correctly.
Note that the working case for you is with a LocalCluster & LocalDRPC, and
you are explicitly invoking the execute():
String result = drpc.execute(TOPOLOGY_NAME, tweetJson);
In the remote case you aren't doing anything except submitting the
topology. I assume you need to invoke drpc.execute *somewhere*... Here's
what I think is the same question as you are posing:
- http://stackoverflow.com/a/26440260/318428
- Erik
On Wed, Jan 27, 2016 at 3:04 PM, researcher cs <[email protected]>
wrote:
> also got zeros after open storm ui except executed column
>
>
>
>
> On Thu, Jan 28, 2016 at 12:43 AM, researcher cs <[email protected]
> > wrote:
>
>> i atteached what i got when i submitted topology and this happened also
>> when i submitted storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>> storm.starter.BasicDRPCTopology basic-drpc
>>
>> this storm.yaml
>> storm.zookeeper.servers:
>> - "localhost"
>> # - "server2"
>> nimbus.host: "localhost"
>>
>> storm.local.dir: "/var/storm"
>> supervisor.slots.ports:
>> - 7660
>> - 7659
>> - 7658
>> - 7657
>> supervisor.childopts: "-Djava.net.preferIPv4Stack=true"
>> nimbus.childopts: "-Djava.net.preferIPv4Stack=true"
>> worker.childopts: "-Djava.net.preferIPv4Stack=true"
>> # topology.message.timeout.secs: 30
>> # topology.workers: 1
>> topology.stats.sample.rate: 1.0
>> topology.acker.executors: 1
>> # topology.executor.receive.buffer.size: 16384
>> # topology.executor.send.buffer.size: 16384
>> # topology.transfer.buffer.size: 32
>> # topology.receiver.buffer.size: 8
>> #
>> # ##### These may optionally be filled in:
>> #
>> ## List of custom serializations
>> # topology.kryo.register:
>> # - org.mycompany.MyType
>> # - org.mycompany.MyType2: org.mycompany.MyType2Serializer
>> #
>> ## List of custom kryo decorators
>> # topology.kryo.decorators:
>> # - org.mycompany.MyDecorator
>> #
>> ## Locations of the drpc servers
>> # drpc.servers:
>> # - "localhost"
>> # - "server2"
>>
>> ## Metrics Consumers
>> # topology.metrics.consumer.register:
>> # - class: "backtype.storm.metrics.LoggingMetricsConsumer"
>> # parallelism.hint: 1
>> # - class: "org.mycompany.MyMetricsConsumer"
>> # parallelism.hint: 1
>> # argument:
>> # - endpoint: "metrics-collector.mycompany.org"
>> storm.messaging.transport: "backtype.storm.messaging.netty.Context"
>> storm.messaging.netty.server_worker_threads: 1
>> storm.messaging.netty.client_worker_threads: 1
>> storm.messaging.netty.buffer_size: 5242880
>> storm.messaging.netty.max_retries: 100
>> storm.messaging.netty.max_wait_ms: 1000
>> storm.messaging.netty.min_wait_ms: 100
>>
>> this etc/hosts
>> 127.0.0.1 localhost
>> 127.0.1.1 user-Lenovo-G50-70
>> ##The following lines are desirable for IPv6 capable hosts##
>> ::1 ip6-localhost ip6-loopback
>> fe00::0 ip6-localnet
>> ff00::0 ip6-mcastprefix
>> ff02::1 ip6-allnodes
>> ff02::2 ip6-allrouters
>>
>> i set drpc in code
>> private static Config createTopologyConfiguration(Properties prop,
>> boolean localMode) {
>> Config conf = new Config();
>> List<String> dprcServers = new ArrayList<String>();
>> dprcServers.add("localhost");
>>
>> conf.put(Config.DRPC_SERVERS, dprcServers);
>> conf.put(Config.DRPC_PORT, 3772);
>> if (!localMode)
>> conf.put(Config.STORM_CLUSTER_MODE, new
>> String("distributed"));
>>
>> }
>> public static void main(String[] args) throws Exception {
>> if (args == null || args.length == 0) {
>> Config conf = createTopologyConfiguration(prop, true);
>> LocalDRPC drpc = new LocalDRPC();
>> LocalCluster cluster = new LocalCluster();
>>
>> cluster.submitTopology(TOPOLOGY_NAME, conf,
>> buildTopology(drpc));
>>
>> Thread.sleep(2000); // give it some time to setup
>>
>> BufferedReader br = new BufferedReader(new
>> FileReader(queryFile));
>> String tweetJson;
>> fos.write("Start: ".getBytes());
>>
>> fos.write(String.valueOf(System.currentTimeMillis()).getBytes());
>> byte[] newLine = "\n".getBytes();
>> int times = 0;
>> // emit tweets into topology
>> while ((tweetJson = br.readLine()) != null) {
>>
>> String result = drpc.execute(TOPOLOGY_NAME, tweetJson);
>>
>> Status s = null;
>> try {
>> s = DataObjectFactory.createStatus(tweetJson);
>> result = s.getId() + "\t" + s.getText() + "\t" +
>> result;
>> } catch (TwitterException e) {
>> LOG.error(e.toString());
>> }
>>
>> fos.write(result.getBytes());
>> fos.write(newLine);
>>
>> // times++;
>> // if (times == 1000)
>> // break;
>> }
>> fos.write(newLine);
>> fos.write("Finish: ".getBytes());
>>
>> fos.write(String.valueOf(System.currentTimeMillis()).getBytes());
>>
>> fos.flush();
>> fos.close();
>> br.close();
>> drpc.shutdown();
>> cluster.shutdown();
>> } else {
>> // distributed mode
>> Config conf = createTopologyConfiguration(prop,true);
>> LocalDRPC drpc = null;
>> StormSubmitter.submitTopology(args[0], conf,
>> buildTopology(drpc));
>>
>> }
>>
>> }
>>
>> On Wed, Jan 27, 2016 at 11:14 PM, Erik Weathers <
>> [email protected]> wrote:
>>
>>> Please put more effort into describing the issue. "It doesn't work" is
>>> unfortunately not enough info for anyone to provide help.
>>> e.g., post links to some code you are trying to run, and the configs of
>>> the
>>> storm components that you are running.
>>>
>>> - Erik
>>>
>>> On Wed, Jan 27, 2016 at 4:59 AM, sam mohel <[email protected]> wrote:
>>>
>>> > I wrote the actual problem in My first message drpc server not working
>>> I
>>> > hot zeros in the columns of storm ui like emitted and transferred ,
>>> result
>>> > file is empty
>>> >
>>> > On Wednesday, January 27, 2016, Erik Weathers
>>> > <[email protected]>
>>> > wrote:
>>> >
>>> > > Your mail client is wrapping the log lines prematurely, I have a
>>> really
>>> > > really hard time reading wrapped lines, I'd look into fixing that if
>>> I
>>> > were
>>> > > you. Here they are unwrapped:
>>> > >
>>> > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>>> connection,
>>> > > connectString=localhost:2181 sessionTimeout=20000
>>> > > watcher=com.netflix.curator.ConnectionState@2fa423d2
>>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket
>>> connection to
>>> > > server localhost/127.0.1.1:2181. Will not attempt to authenticate
>>> using
>>> > > SASL (unknown error)
>>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>>> established
>>> > > to localhost/127.0.1.1:2181, initiating session
>>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>>> > complete
>>> > > on server localhost/127.0.1.1:2181, sessionid = 0x152804f3a3a0002,
>>> > > negotiated timeout = 20000
>>> > > 2016-01-27 01:41:00 b.s.zookeeper [INFO] Zookeeper state update:
>>> > > :connected:none
>>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Session: 0x152804f3a3a0002
>>> > > closed
>>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] EventThread shut down
>>> > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>>> connection,
>>> > > connectString=localhost:2181/storm sessionTimeout=20000
>>> > > watcher=com.netflix.curator.ConnectionState@2ee8b0bf
>>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket
>>> connection to
>>> > > server user-Lenovo-G50-70/127.0.0.1:2181. Will not attempt to
>>> > authenticate
>>> > > using SASL (unknown error)
>>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>>> established
>>> > > to user-Lenovo-G50-70/127.0.0.1:2181, initiating session
>>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>>> > complete
>>> > > on server user-Lenovo-G50-70/127.0.0.1:2181, sessionid =
>>> > > 0x152804f3a3a0003,
>>> > > negotiated timeout = 20000
>>> > >
>>> > > None of those indicate a problem, they look pretty standard to me.
>>> > >
>>> > > Please spend a bit more time zeroing in on what the actual problem
>>> is so
>>> > > that the members of the list(s) can provide help.
>>> > >
>>> > > - Erik
>>> > >
>>> > > On Tue, Jan 26, 2016 at 10:15 PM, researcher cs <
>>> > [email protected]
>>> > > <javascript:;>>
>>> > > wrote:
>>> > >
>>> > > > yes i tried in local and worked well
>>> > > > and about /etc/hosts . i'm feeling that this file has a mistake , i
>>> > made
>>> > > > alot of changes in this file and didn't remember what was default
>>> > > >
>>> > > > In nimbus log file when it connected zookeeper i got
>>> > > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>>> > connection,
>>> > > > connectString=localhost:2181 sessionTimeout=20000
>>> > > > watcher=com.netflix.curator.ConnectionState@2fa423d2
>>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket
>>> connection
>>> > to
>>> > > > server localhost/127.0.1.1:2181. Will not attempt to authenticate
>>> > using
>>> > > > SASL (unknown error)
>>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>>> > established
>>> > > > to localhost/127.0.1.1:2181, initiating session
>>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>>> > > complete
>>> > > > on server localhost/127.0.1.1:2181, sessionid = 0x152804f3a3a0002,
>>> > > > negotiated timeout = 20000
>>> > > > 2016-01-27 01:41:00 b.s.zookeeper [INFO] Zookeeper state update:
>>> > > > :connected:none
>>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Session:
>>> 0x152804f3a3a0002
>>> > > > closed
>>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] EventThread shut down
>>> > > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>>> > connection,
>>> > > > connectString=localhost:2181/storm sessionTimeout=20000
>>> > > > watcher=com.netflix.curator.ConnectionState@2ee8b0bf
>>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket
>>> connection
>>> > to
>>> > > > server user-Lenovo-G50-70/127.0.0.1:2181. Will not attempt to
>>> > > authenticate
>>> > > > using SASL (unknown error)
>>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>>> > established
>>> > > > to user-Lenovo-G50-70/127.0.0.1:2181, initiating session
>>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>>> > > complete
>>> > > > on server user-Lenovo-G50-70/127.0.0.1:2181, sessionid =
>>> > > > 0x152804f3a3a0003,
>>> > > > negotiated timeout = 20000
>>> > > >
>>> > > > and i set for drpc.server : localhost
>>> > > > storm.zookeeper.server : localhost
>>> > > > nimbus.host : localhost
>>> > > >
>>> > > > in my /etc/hosts
>>> > > > 127.0.0.1 user-Lenovo-G50-70 localhost
>>> > > > 127.0.1.1 localhost
>>> > > >
>>> > > > is that right ?
>>> > > >
>>> > > >
>>> > > > On Wed, Jan 27, 2016 at 6:12 AM, Erik Weathers <
>>> > > > [email protected]> wrote:
>>> > > >
>>> > > > > You said: "except the statement of drpc server trying to connect"
>>> > > > >
>>> > > > > Maybe you are confused about what "b.s.d.drpc [INFO] Starting
>>> > > Distributed
>>> > > > > RPC servers..." implies?
>>> > > > > That is just saying that the server is being started. It's a
>>> > server,
>>> > > > not
>>> > > > > a client, so it's basic operation is *not* to connect to some
>>> other
>>> > > > thing.
>>> > > > > It's up and waiting for you to tell it to do stuff.
>>> > > > >
>>> > > > > Have you gotten the Local-mode version of DRPC working?
>>> > > > >
>>> > > > > -
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://storm.apache.org/documentation/Distributed-RPC.html#local-mode-drpc
>>> > > > >
>>> > > > > Not sure what you're asking with regards to DNS and /etc/hosts,
>>> those
>>> > > > seem
>>> > > > > unrelated to your basic issue.
>>> > > > >
>>> > > > > - Erik
>>> > > > >
>>> > > > > On Tue, Jan 26, 2016 at 6:58 PM, researcher cs <
>>> > > > [email protected] <javascript:;>>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > There is no error except the statement of drpc server trying to
>>> > > connect
>>> > > > > . I
>>> > > > > > guess I have problem with Dns . if you have any idea about this
>>> > > pleaes
>>> > > > > help
>>> > > > > >
>>> > > > > >
>>> > > > > > I want to submit topology with single machine
>>> > > > > > Only on my laptop without any other devices as a first step so
>>> > > > > > What this file /etc/hosts should contains ?
>>> > > > > > As I set in drpc.server : localhost
>>> > > > > > Storm.zookeeper. server: localhost
>>> > > > > > Nimbus.host:localhost
>>> > > > > >
>>> > > > > > As this file contains 127.0.1.1 and 127.0.0.1 and my IP
>>> address ?
>>> > > > > >
>>> > > > > > What should I use and what should I hash it to not using it ?
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > On Wednesday, January 27, 2016, Erik Weathers
>>> > > > > > <[email protected]>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > What does the client code that is supposed to make the DRPC
>>> > > > connection
>>> > > > > > > telling you? i.e., you should see some exception or log
>>> about
>>> > not
>>> > > > > > > establishing the connection, right?
>>> > > > > > >
>>> > > > > > > Alternatively, perhaps the connections aren't persistent and
>>> > > there's
>>> > > > no
>>> > > > > > > actual problem?
>>> > > > > > >
>>> > > > > > > - Erik
>>> > > > > > >
>>> > > > > > > On Tue, Jan 26, 2016 at 4:55 PM, researcher cs <
>>> > > > > > [email protected] <javascript:;>
>>> > > > > > > <javascript:;>>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > thanks for replying , i read the documentation before , i
>>> > > imported
>>> > > > > > > project
>>> > > > > > > > supposed to work well but not working with me
>>> > > > > > > > i checked port by lsof -i gave me all ports i connected it
>>> for
>>> > > > storm
>>> > > > > > > > java 10675 root 20u IPv4 98126 0t0 TCP
>>> *:52022
>>> > > > > > (LISTEN)
>>> > > > > > > > java 10675 root 26u IPv4 98131 0t0 TCP
>>> *:2181
>>> > > > > (LISTEN)
>>> > > > > > > > java 10675 root 27u IPv4 101944 0t0 TCP
>>> > > > > > > > localhost:2181->user-Lenovo-G50-70:38150 (ESTABLISHED)
>>> > > > > > > > java 10675 root 29u IPv4 98974 0t0 TCP
>>> > > > > > > > user-Lenovo-G50-70:2181->user-Lenovo-G50-70:50526
>>> (ESTABLISHED)
>>> > > > > > > > java 10675 root 30u IPv4 99105 0t0 TCP
>>> > > > > > > > localhost:2181->user-Lenovo-G50-70:38165 (ESTABLISHED)
>>> > > > > > > > java 10715 root 90u IPv4 98953 0t0 TCP
>>> > > > > > > > user-Lenovo-G50-70:38150->localhost:2181 (ESTABLISHED)
>>> > > > > > > > java 10715 root 91u IPv4 98245 0t0 TCP
>>> *:6627
>>> > > > > (LISTEN)
>>> > > > > > > > java 10792 root 90u IPv4 99973 0t0 TCP
>>> > > > > > > > user-Lenovo-G50-70:50526->user-Lenovo-G50-70:2181
>>> (ESTABLISHED)
>>> > > > > > > > java 10864 root 82u IPv4 102425 0t0 TCP
>>> *:3772
>>> > > > > (LISTEN)
>>> > > > > > > > java 10864 root 84u IPv4 102429 0t0 TCP
>>> *:3773
>>> > > > > (LISTEN)
>>> > > > > > > > java 10864 root 92u IPv4 102197 0t0 TCP
>>> > > > > > > > user-Lenovo-G50-70:3773->user-Lenovo-G50-70:50825
>>> (ESTABLISHED)
>>> > > > > > > > java 10928 root 81u IPv4 102070 0t0 TCP
>>> > > *:http-alt
>>> > > > > > > (LISTEN)
>>> > > > > > > > java 11087 root 81u IPv4 100091 0t0 TCP
>>> > > > > > > > user-Lenovo-G50-70:50825->user-Lenovo-G50-70:3773
>>> (ESTABLISHED)
>>> > > > > > > > java 11087 root 91u IPv4 102196 0t0 TCP
>>> > > > > > > > user-Lenovo-G50-70:38165->localhost:2181 (ESTABLISHED)
>>> > > > > > > > java 11087 root 94u IPv4 102561 0t0 TCP
>>> *:7660
>>> > > > > (LISTEN)
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > here you can see that 3772 not established
>>> > > > > > > >
>>> > > > > > > > On Wed, Jan 27, 2016 at 2:47 AM, Erik Weathers <
>>> > > > > > > > [email protected]> wrote:
>>> > > > > > > >
>>> > > > > > > > > hey,
>>> > > > > > > > >
>>> > > > > > > > > The DRPC server is up and listening on port 3772. Why
>>> do
>>> > you
>>> > > > > expect
>>> > > > > > > > > established connections?
>>> > > > > > > > >
>>> > > > > > > > > I'm not familiar with using Storm's DRPC feature, but I'm
>>> > sure
>>> > > > you
>>> > > > > > need
>>> > > > > > > > to
>>> > > > > > > > > write code that interacts with the DRPC server, and
>>> you've
>>> > made
>>> > > > no
>>> > > > > > > > mention
>>> > > > > > > > > of doing so in your email. I'd start here:
>>> > > > > > > > >
>>> > > > > > > > > -
>>> > > https://storm.apache.org/documentation/Distributed-RPC.html
>>> > > > > > > > >
>>> > > > > > > > > - Erik
>>> > > > > > > > >
>>> > > > > > > > > On Tue, Jan 26, 2016 at 4:29 PM, researcher cs <
>>> > > > > > > > [email protected] <javascript:;> <javascript:;>>
>>> > > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > I set in the code
>>> > > > > > > > > > conf.put(Config.DRPC_SERVERS, dprcServers);
>>> > > > > > > > > > conf.put(Config.DRPC_PORT, 3772);
>>> > > > > > > > > > but when i submit topolgoy i found at the end of the
>>> file
>>> > > > > > > > > > b.s.d.drpc [INFO] Starting Distributed RPC servers...
>>> > > > > > > > > >
>>> > > > > > > > > > i checked port 3772 by
>>> > > > > > > > > >
>>> > > > > > > > > > sudo netstat -ap | grep 3772
>>> > > > > > > > > >
>>> > > > > > > > > > i got
>>> > > > > > > > > >
>>> > > > > > > > > > tcp 0 0 *:3772 *:* LISTEN
>>> 10864/java
>>> > > > > > > > > > unix 3 [ ] STREAM CONNECTED 13772
>>> > > > > > > > > > 587/dbus-daemon /var/run/dbus/system_bus_socket
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > why it's not established ?
>>> > > > > > > > > >
>>> > > > > > > > > > can i find help ?
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
