also got zeros after open storm ui except executed column
On Thu, Jan 28, 2016 at 12:43 AM, researcher cs <prog.researc...@gmail.com>
wrote:
> i atteached what i got when i submitted topology and this happened also
> when i submitted storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> storm.starter.BasicDRPCTopology basic-drpc
>
> this storm.yaml
> storm.zookeeper.servers:
> - "localhost"
> # - "server2"
> nimbus.host: "localhost"
>
> storm.local.dir: "/var/storm"
> supervisor.slots.ports:
> - 7660
> - 7659
> - 7658
> - 7657
> supervisor.childopts: "-Djava.net.preferIPv4Stack=true"
> nimbus.childopts: "-Djava.net.preferIPv4Stack=true"
> worker.childopts: "-Djava.net.preferIPv4Stack=true"
> # topology.message.timeout.secs: 30
> # topology.workers: 1
> topology.stats.sample.rate: 1.0
> topology.acker.executors: 1
> # topology.executor.receive.buffer.size: 16384
> # topology.executor.send.buffer.size: 16384
> # topology.transfer.buffer.size: 32
> # topology.receiver.buffer.size: 8
> #
> # ##### These may optionally be filled in:
> #
> ## List of custom serializations
> # topology.kryo.register:
> # - org.mycompany.MyType
> # - org.mycompany.MyType2: org.mycompany.MyType2Serializer
> #
> ## List of custom kryo decorators
> # topology.kryo.decorators:
> # - org.mycompany.MyDecorator
> #
> ## Locations of the drpc servers
> # drpc.servers:
> # - "localhost"
> # - "server2"
>
> ## Metrics Consumers
> # topology.metrics.consumer.register:
> # - class: "backtype.storm.metrics.LoggingMetricsConsumer"
> # parallelism.hint: 1
> # - class: "org.mycompany.MyMetricsConsumer"
> # parallelism.hint: 1
> # argument:
> # - endpoint: "metrics-collector.mycompany.org"
> storm.messaging.transport: "backtype.storm.messaging.netty.Context"
> storm.messaging.netty.server_worker_threads: 1
> storm.messaging.netty.client_worker_threads: 1
> storm.messaging.netty.buffer_size: 5242880
> storm.messaging.netty.max_retries: 100
> storm.messaging.netty.max_wait_ms: 1000
> storm.messaging.netty.min_wait_ms: 100
>
> this etc/hosts
> 127.0.0.1 localhost
> 127.0.1.1 user-Lenovo-G50-70
> ##The following lines are desirable for IPv6 capable hosts##
> ::1 ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
>
> i set drpc in code
> private static Config createTopologyConfiguration(Properties prop,
> boolean localMode) {
> Config conf = new Config();
> List<String> dprcServers = new ArrayList<String>();
> dprcServers.add("localhost");
>
> conf.put(Config.DRPC_SERVERS, dprcServers);
> conf.put(Config.DRPC_PORT, 3772);
> if (!localMode)
> conf.put(Config.STORM_CLUSTER_MODE, new String("distributed"));
>
> }
> public static void main(String[] args) throws Exception {
> if (args == null || args.length == 0) {
> Config conf = createTopologyConfiguration(prop, true);
> LocalDRPC drpc = new LocalDRPC();
> LocalCluster cluster = new LocalCluster();
>
> cluster.submitTopology(TOPOLOGY_NAME, conf,
> buildTopology(drpc));
>
> Thread.sleep(2000); // give it some time to setup
>
> BufferedReader br = new BufferedReader(new
> FileReader(queryFile));
> String tweetJson;
> fos.write("Start: ".getBytes());
>
> fos.write(String.valueOf(System.currentTimeMillis()).getBytes());
> byte[] newLine = "\n".getBytes();
> int times = 0;
> // emit tweets into topology
> while ((tweetJson = br.readLine()) != null) {
>
> String result = drpc.execute(TOPOLOGY_NAME, tweetJson);
>
> Status s = null;
> try {
> s = DataObjectFactory.createStatus(tweetJson);
> result = s.getId() + "\t" + s.getText() + "\t" +
> result;
> } catch (TwitterException e) {
> LOG.error(e.toString());
> }
>
> fos.write(result.getBytes());
> fos.write(newLine);
>
> // times++;
> // if (times == 1000)
> // break;
> }
> fos.write(newLine);
> fos.write("Finish: ".getBytes());
>
> fos.write(String.valueOf(System.currentTimeMillis()).getBytes());
>
> fos.flush();
> fos.close();
> br.close();
> drpc.shutdown();
> cluster.shutdown();
> } else {
> // distributed mode
> Config conf = createTopologyConfiguration(prop,true);
> LocalDRPC drpc = null;
> StormSubmitter.submitTopology(args[0], conf,
> buildTopology(drpc));
>
> }
>
> }
>
> On Wed, Jan 27, 2016 at 11:14 PM, Erik Weathers <
> eweath...@groupon.com.invalid> wrote:
>
>> Please put more effort into describing the issue. "It doesn't work" is
>> unfortunately not enough info for anyone to provide help.
>> e.g., post links to some code you are trying to run, and the configs of
>> the
>> storm components that you are running.
>>
>> - Erik
>>
>> On Wed, Jan 27, 2016 at 4:59 AM, sam mohel <sammoh...@gmail.com> wrote:
>>
>> > I wrote the actual problem in My first message drpc server not working I
>> > hot zeros in the columns of storm ui like emitted and transferred ,
>> result
>> > file is empty
>> >
>> > On Wednesday, January 27, 2016, Erik Weathers
>> > <eweath...@groupon.com.invalid>
>> > wrote:
>> >
>> > > Your mail client is wrapping the log lines prematurely, I have a
>> really
>> > > really hard time reading wrapped lines, I'd look into fixing that if I
>> > were
>> > > you. Here they are unwrapped:
>> > >
>> > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>> connection,
>> > > connectString=localhost:2181 sessionTimeout=20000
>> > > watcher=com.netflix.curator.ConnectionState@2fa423d2
>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket connection
>> to
>> > > server localhost/127.0.1.1:2181. Will not attempt to authenticate
>> using
>> > > SASL (unknown error)
>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>> established
>> > > to localhost/127.0.1.1:2181, initiating session
>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>> > complete
>> > > on server localhost/127.0.1.1:2181, sessionid = 0x152804f3a3a0002,
>> > > negotiated timeout = 20000
>> > > 2016-01-27 01:41:00 b.s.zookeeper [INFO] Zookeeper state update:
>> > > :connected:none
>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Session: 0x152804f3a3a0002
>> > > closed
>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] EventThread shut down
>> > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>> > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>> connection,
>> > > connectString=localhost:2181/storm sessionTimeout=20000
>> > > watcher=com.netflix.curator.ConnectionState@2ee8b0bf
>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket connection
>> to
>> > > server user-Lenovo-G50-70/127.0.0.1:2181. Will not attempt to
>> > authenticate
>> > > using SASL (unknown error)
>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>> established
>> > > to user-Lenovo-G50-70/127.0.0.1:2181, initiating session
>> > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>> > complete
>> > > on server user-Lenovo-G50-70/127.0.0.1:2181, sessionid =
>> > > 0x152804f3a3a0003,
>> > > negotiated timeout = 20000
>> > >
>> > > None of those indicate a problem, they look pretty standard to me.
>> > >
>> > > Please spend a bit more time zeroing in on what the actual problem is
>> so
>> > > that the members of the list(s) can provide help.
>> > >
>> > > - Erik
>> > >
>> > > On Tue, Jan 26, 2016 at 10:15 PM, researcher cs <
>> > prog.researc...@gmail.com
>> > > <javascript:;>>
>> > > wrote:
>> > >
>> > > > yes i tried in local and worked well
>> > > > and about /etc/hosts . i'm feeling that this file has a mistake , i
>> > made
>> > > > alot of changes in this file and didn't remember what was default
>> > > >
>> > > > In nimbus log file when it connected zookeeper i got
>> > > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>> > connection,
>> > > > connectString=localhost:2181 sessionTimeout=20000
>> > > > watcher=com.netflix.curator.ConnectionState@2fa423d2
>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket
>> connection
>> > to
>> > > > server localhost/127.0.1.1:2181. Will not attempt to authenticate
>> > using
>> > > > SASL (unknown error)
>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>> > established
>> > > > to localhost/127.0.1.1:2181, initiating session
>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>> > > complete
>> > > > on server localhost/127.0.1.1:2181, sessionid = 0x152804f3a3a0002,
>> > > > negotiated timeout = 20000
>> > > > 2016-01-27 01:41:00 b.s.zookeeper [INFO] Zookeeper state update:
>> > > > :connected:none
>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Session:
>> 0x152804f3a3a0002
>> > > > closed
>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] EventThread shut down
>> > > > 2016-01-27 01:41:00 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>> > > > 2016-01-27 01:41:00 o.a.z.ZooKeeper [INFO] Initiating client
>> > connection,
>> > > > connectString=localhost:2181/storm sessionTimeout=20000
>> > > > watcher=com.netflix.curator.ConnectionState@2ee8b0bf
>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Opening socket
>> connection
>> > to
>> > > > server user-Lenovo-G50-70/127.0.0.1:2181. Will not attempt to
>> > > authenticate
>> > > > using SASL (unknown error)
>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Socket connection
>> > established
>> > > > to user-Lenovo-G50-70/127.0.0.1:2181, initiating session
>> > > > 2016-01-27 01:41:00 o.a.z.ClientCnxn [INFO] Session establishment
>> > > complete
>> > > > on server user-Lenovo-G50-70/127.0.0.1:2181, sessionid =
>> > > > 0x152804f3a3a0003,
>> > > > negotiated timeout = 20000
>> > > >
>> > > > and i set for drpc.server : localhost
>> > > > storm.zookeeper.server : localhost
>> > > > nimbus.host : localhost
>> > > >
>> > > > in my /etc/hosts
>> > > > 127.0.0.1 user-Lenovo-G50-70 localhost
>> > > > 127.0.1.1 localhost
>> > > >
>> > > > is that right ?
>> > > >
>> > > >
>> > > > On Wed, Jan 27, 2016 at 6:12 AM, Erik Weathers <
>> > > > eweath...@groupon.com.invalid> wrote:
>> > > >
>> > > > > You said: "except the statement of drpc server trying to connect"
>> > > > >
>> > > > > Maybe you are confused about what "b.s.d.drpc [INFO] Starting
>> > > Distributed
>> > > > > RPC servers..." implies?
>> > > > > That is just saying that the server is being started. It's a
>> > server,
>> > > > not
>> > > > > a client, so it's basic operation is *not* to connect to some
>> other
>> > > > thing.
>> > > > > It's up and waiting for you to tell it to do stuff.
>> > > > >
>> > > > > Have you gotten the Local-mode version of DRPC working?
>> > > > >
>> > > > > -
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://storm.apache.org/documentation/Distributed-RPC.html#local-mode-drpc
>> > > > >
>> > > > > Not sure what you're asking with regards to DNS and /etc/hosts,
>> those
>> > > > seem
>> > > > > unrelated to your basic issue.
>> > > > >
>> > > > > - Erik
>> > > > >
>> > > > > On Tue, Jan 26, 2016 at 6:58 PM, researcher cs <
>> > > > prog.researc...@gmail.com <javascript:;>>
>> > > > > wrote:
>> > > > >
>> > > > > > There is no error except the statement of drpc server trying to
>> > > connect
>> > > > > . I
>> > > > > > guess I have problem with Dns . if you have any idea about this
>> > > pleaes
>> > > > > help
>> > > > > >
>> > > > > >
>> > > > > > I want to submit topology with single machine
>> > > > > > Only on my laptop without any other devices as a first step so
>> > > > > > What this file /etc/hosts should contains ?
>> > > > > > As I set in drpc.server : localhost
>> > > > > > Storm.zookeeper. server: localhost
>> > > > > > Nimbus.host:localhost
>> > > > > >
>> > > > > > As this file contains 127.0.1.1 and 127.0.0.1 and my IP address
>> ?
>> > > > > >
>> > > > > > What should I use and what should I hash it to not using it ?
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Wednesday, January 27, 2016, Erik Weathers
>> > > > > > <eweath...@groupon.com.invalid>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > What does the client code that is supposed to make the DRPC
>> > > > connection
>> > > > > > > telling you? i.e., you should see some exception or log about
>> > not
>> > > > > > > establishing the connection, right?
>> > > > > > >
>> > > > > > > Alternatively, perhaps the connections aren't persistent and
>> > > there's
>> > > > no
>> > > > > > > actual problem?
>> > > > > > >
>> > > > > > > - Erik
>> > > > > > >
>> > > > > > > On Tue, Jan 26, 2016 at 4:55 PM, researcher cs <
>> > > > > > prog.researc...@gmail.com <javascript:;>
>> > > > > > > <javascript:;>>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > thanks for replying , i read the documentation before , i
>> > > imported
>> > > > > > > project
>> > > > > > > > supposed to work well but not working with me
>> > > > > > > > i checked port by lsof -i gave me all ports i connected it
>> for
>> > > > storm
>> > > > > > > > java 10675 root 20u IPv4 98126 0t0 TCP
>> *:52022
>> > > > > > (LISTEN)
>> > > > > > > > java 10675 root 26u IPv4 98131 0t0 TCP
>> *:2181
>> > > > > (LISTEN)
>> > > > > > > > java 10675 root 27u IPv4 101944 0t0 TCP
>> > > > > > > > localhost:2181->user-Lenovo-G50-70:38150 (ESTABLISHED)
>> > > > > > > > java 10675 root 29u IPv4 98974 0t0 TCP
>> > > > > > > > user-Lenovo-G50-70:2181->user-Lenovo-G50-70:50526
>> (ESTABLISHED)
>> > > > > > > > java 10675 root 30u IPv4 99105 0t0 TCP
>> > > > > > > > localhost:2181->user-Lenovo-G50-70:38165 (ESTABLISHED)
>> > > > > > > > java 10715 root 90u IPv4 98953 0t0 TCP
>> > > > > > > > user-Lenovo-G50-70:38150->localhost:2181 (ESTABLISHED)
>> > > > > > > > java 10715 root 91u IPv4 98245 0t0 TCP
>> *:6627
>> > > > > (LISTEN)
>> > > > > > > > java 10792 root 90u IPv4 99973 0t0 TCP
>> > > > > > > > user-Lenovo-G50-70:50526->user-Lenovo-G50-70:2181
>> (ESTABLISHED)
>> > > > > > > > java 10864 root 82u IPv4 102425 0t0 TCP
>> *:3772
>> > > > > (LISTEN)
>> > > > > > > > java 10864 root 84u IPv4 102429 0t0 TCP
>> *:3773
>> > > > > (LISTEN)
>> > > > > > > > java 10864 root 92u IPv4 102197 0t0 TCP
>> > > > > > > > user-Lenovo-G50-70:3773->user-Lenovo-G50-70:50825
>> (ESTABLISHED)
>> > > > > > > > java 10928 root 81u IPv4 102070 0t0 TCP
>> > > *:http-alt
>> > > > > > > (LISTEN)
>> > > > > > > > java 11087 root 81u IPv4 100091 0t0 TCP
>> > > > > > > > user-Lenovo-G50-70:50825->user-Lenovo-G50-70:3773
>> (ESTABLISHED)
>> > > > > > > > java 11087 root 91u IPv4 102196 0t0 TCP
>> > > > > > > > user-Lenovo-G50-70:38165->localhost:2181 (ESTABLISHED)
>> > > > > > > > java 11087 root 94u IPv4 102561 0t0 TCP
>> *:7660
>> > > > > (LISTEN)
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > here you can see that 3772 not established
>> > > > > > > >
>> > > > > > > > On Wed, Jan 27, 2016 at 2:47 AM, Erik Weathers <
>> > > > > > > > eweath...@groupon.com.invalid> wrote:
>> > > > > > > >
>> > > > > > > > > hey,
>> > > > > > > > >
>> > > > > > > > > The DRPC server is up and listening on port 3772. Why do
>> > you
>> > > > > expect
>> > > > > > > > > established connections?
>> > > > > > > > >
>> > > > > > > > > I'm not familiar with using Storm's DRPC feature, but I'm
>> > sure
>> > > > you
>> > > > > > need
>> > > > > > > > to
>> > > > > > > > > write code that interacts with the DRPC server, and you've
>> > made
>> > > > no
>> > > > > > > > mention
>> > > > > > > > > of doing so in your email. I'd start here:
>> > > > > > > > >
>> > > > > > > > > -
>> > > https://storm.apache.org/documentation/Distributed-RPC.html
>> > > > > > > > >
>> > > > > > > > > - Erik
>> > > > > > > > >
>> > > > > > > > > On Tue, Jan 26, 2016 at 4:29 PM, researcher cs <
>> > > > > > > > prog.researc...@gmail.com <javascript:;> <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I set in the code
>> > > > > > > > > > conf.put(Config.DRPC_SERVERS, dprcServers);
>> > > > > > > > > > conf.put(Config.DRPC_PORT, 3772);
>> > > > > > > > > > but when i submit topolgoy i found at the end of the
>> file
>> > > > > > > > > > b.s.d.drpc [INFO] Starting Distributed RPC servers...
>> > > > > > > > > >
>> > > > > > > > > > i checked port 3772 by
>> > > > > > > > > >
>> > > > > > > > > > sudo netstat -ap | grep 3772
>> > > > > > > > > >
>> > > > > > > > > > i got
>> > > > > > > > > >
>> > > > > > > > > > tcp 0 0 *:3772 *:* LISTEN
>> 10864/java
>> > > > > > > > > > unix 3 [ ] STREAM CONNECTED 13772
>> > > > > > > > > > 587/dbus-daemon /var/run/dbus/system_bus_socket
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > why it's not established ?
>> > > > > > > > > >
>> > > > > > > > > > can i find help ?
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
