is that right command ps aux |grep 2621 user 2621 7.8 2.7 3444276 108056 pts/12 Sl+ 02:42 12:01 java -server -Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2 -Dlog4j.configuration=storm.log.properties -cp /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/commons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm-0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/log4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/storm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simple-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/storm-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/jline-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/lib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/storm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/conf:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/stormjar.jar backtype.storm.daemon.worker fsd-1-1449794574 1953edf9-8f3e-4bd8-870a-05370272b097 6703 ff8603ef-5fb9-47c7-90d0-01813ace4188 user 3573 0.0 0.0 16016 2256 pts/28 S+ 05:16 0:00 grep --color=auto 2621
On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <[email protected]> wrote: > please , how can i check it ? > > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) <[email protected]> > wrote: > >> Okay. Please check which application this process belongs to, and whether >> is it OK to kill it. >> >> Regards >> Basti >> -----Original Message----- >> From: sam mohel [mailto:[email protected]] >> Sent: Friday, December 11, 2015 11:06 AM >> To: [email protected] >> Subject: Re: Problem with storm since 4 months >> >> i used >> >> netstat -ntape >> >> and got >> Code: >> >> tcp 0 0 0.0.0.0:6703 0.0.0.0:* >> LISTEN 1000 22776 2621/java >> >> then used this command with PID of PORT = 2621 >> Code: >> >> lsof -P -w -n -p 2621 -a -i >> >> and got >> Code: >> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >> java 2621 user 76u IPv4 18906 0t0 TCP >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED) >> java 2621 user 83u IPv4 22773 0t0 TCP >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED) >> java 2621 user 86u IPv4 22776 0t0 TCP *:6703 (LISTEN) >> >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) <[email protected]> >> wrote: >> >> > Hi Sam, >> > >> > Have you tried "netstat" and "lsof" commands recommended by Erik? What >> > is the result? >> > We need to find which process bound this port, then kill the process >> > to release this port. >> > >> > Regards >> > Basti >> > >> > -----Original Message----- >> > From: sam mohel [mailto:[email protected]] >> > Sent: Friday, December 11, 2015 8:15 AM >> > To: [email protected] >> > Subject: Re: Problem with storm since 4 months >> > >> > > now i converted the change >> > >> > but problem still >> > >> > > i mean in the link you sent , he said "the ephermal port range is >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can probably >> > extend it to run from 16k to 64k." >> > >> > how can i do that ? >> > >> > > how can i check if there is anothe dump worker working , >> > or >> > is there another way to kill any worker if it is dump or not >> "completely" >> > and start from scratch >> > >> > thanks for your patience and your time >> > >> > >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < >> > [email protected]> wrote: >> > >> > > responses inline. >> > > >> > > >Ah, so, you shouldn't have touched the "local port range" setting >> > > >since >> > > you >> > > don't fully understand ephemeral ports and TCP yet! :-) >> > > >> > > I'll give a brief synopsis: >> > > >> > > Say you're making a connection from a client to a server that >> > > listens on port 80. The client needs to have a port of its own to >> > > receive the response packets from the server. The port that is >> > > allocated for it by the TCP stack is a "random" port, which is >> > > called an "ephemeral" port in >> > this context. >> > > So with your original default config (32768 61000) the port would >> be >> > > chosen from an available port on the machine that lies within that >> > > range of >> > > 32768->61000. That's good, it wouldn't conflict with the default >> > > 32768->TCP ports >> > > being listened to by the Storm Worker processes (67xx). So 1. isn't >> > > your problem. But now you've made it possible for it to become a >> > > problem, since now the range of ports that can be given as an >> > > ephemeral port overlaps with the default Storm Worker ports >> > > (1024->65535 includes 67xx). So you should revert that config change. >> > > >> > > Thanks for your replying , now i reverted the change >> > >> > > >> > > >Sorry, I don't understand what you are asking. What thing is "16" >> > > >that >> > > you >> > > are trying to extend? >> > > >> > > i mean in the link you sent , he adviced to extend from 16 to 64 >> > > >You can kill topologies from the Nimbus UI (web page). Or with the >> > > bin/storm command. >> > > >> > > >> > > >The commands are *solely* intended to figure out what is conflicting. >> > > Your >> > > logs claim that there is something holding onto 67xx which prevents >> the >> > > Storm Worker from launching. So if that is happening you should >> > > *immediately* try to figure out what is actually holding onto the >> > > port and preventing your Storm Worker from launching. >> > > >> > > >> > > >Not sure what you mean here. >> > > >> > > Please note that Storm often suffers from "cascading" failures, >> > > where there are a lot of exceptions and errors that aren't actually >> > > the root cause of the problem. Often you need to spend time and >> > > effort looking at lots of logs and tracing back to the real root >> cause. >> > > >> > > >> > > >> > > >> > > > >> > > > Really , Thanks for your time >> > > > >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < >> > > > [email protected]> wrote: >> > > > >> > > > > Regarding Basti's suggestion (1.) that your host's configured >> > > > > ephemeral ports might be conflicting with the storm worker >> > > > > ports, here's how you >> > > > can >> > > > > check your "local port range" setting: >> > > > > >> > > > > >> > > > > >> > > > >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how >> > > -m any-ephemeral-ports-are-left-available >> > > > > >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range >> > > > > >> > > > > >> > > > > It's possible that there is a zombie worker process holding onto >> > > > > port >> > > > 6703. >> > > > > I would try to identify the process like so: >> > > > > >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN | >> > > > > grep -w >> > > > 6703 >> > > > > >> > > > > Alternatively you can try a global lsof search: >> > > > > >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703 >> > > > > >> > > > > - Erik >> > > > > >> > > > > >> > > > > >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) < >> > > [email protected]> >> > > > > wrote: >> > > > > >> > > > > > Hi Sam, >> > > > > > >> > > > > > You can try to find which process has bound this port by >> > > > > > "netstat >> > > -anp" >> > > > > > first. >> > > > > > >> > > > > > Generally, there are following cases for the binding error. >> > > > > > 1. "local port range" is not set to exclude the port range >> > > > > > used in >> > > > Storm. >> > > > > > 2. The previous worker was not killed correctly. >> > > > > > 3. There is bug of assignment in some scenarios. Same port was >> > > assigned >> > > > > to >> > > > > > two workers. >> > > > > > >> > > > > > Regards >> > > > > > Basti >> > > > > > >> > > > > > -----Original Message----- >> > > > > > From: sam mohel [mailto:[email protected]] >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM >> > > > > > To: [email protected] >> > > > > > Subject: Re: Problem with storm since 4 months >> > > > > > >> > > > > > i tried to use storm-0.9.5 but problem changed with >> > > > > > >> > > > > > cannot bind port 6703 i think it's same problem >> > > > > > >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <[email protected]> wrote: >> > > > > > >> > > > > > > Sam, >> > > > > > > you might be using very old version of storm since >> > > > > > > its >> > > > > showing >> > > > > > > ZeroMQ. Can you try using newer version storm >> > > > > > > without >> > > zero >> > > > > mq. >> > > > > > > -Harsha >> > > > > > > >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote: >> > > > > > > > I have this problem since 4months when I submitted >> > > > > > > > topology I got this in the worker log file [ERROR] Async >> loop died! >> > > > > > org.zeromq.ZMQException: >> > > > > > > > Address already in use(0x62) at >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at >> > > > > > > > >> > > > > > > >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__16 >> > > > 29 >> > > > .i >> > > > > > > nvoke(loader.clj:26) >> > > > > > > > at >> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:375 >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at >> > > > java.lang.Thread.run(Unknown >> > > > > > > > Source) >> > > > > > > > >> > > > > > > > when i tried to connect port 6703 and 6702 >> > > > > > > > >> > > > > > > > And supervisor log file hadn't still start >> > > > > > > > >> > > > > > > > >> > > > > > > > I searched everywhere but cannot find any solution I hope >> > > > > > > > you can >> > > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> >> >
