[ 
https://issues.apache.org/jira/browse/STORM-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457901#comment-16457901
 ] 

Jorge Moraleda edited comment on STORM-3039 at 4/29/18 3:49 AM:
----------------------------------------------------------------

I have observed this problem in 1.2.1 and I have found a way a scenario that 
makes it relatively reproducible by including a bolt that throws consistently 
in its _prepare_ method so that the topology is continuously being restarted.

In that situation, when the topology is killed manually (e.g. via the UI)) and 
resubmitted, there is a significant probability that the new topology will fail 
to start with the error shown in the original report.

The new topology will try to restart every two minutes. Eventually the new 
topology will bind to a new port (typically 6702) but the process can be 
repeated with more and more ports being full and the new topologies taking 
longer and longer to start.

Restarting the affected supervisor solves the problem.


was (Author: jmoraleda):
I have observed this problem in 1.2.1 and I have found a way a scenario that 
makes it relatively reproducible by including a bolt that throws consistently 
in its _prepare_ method so that the topology is continuously being restarted. 

In that situation, when the topology is killed manually (e.g. via the UI)) and 
resubmitted, there is a significant probability that the new topology will fail 
to start with the error shown in the original report.

The new topology will try to restart every two minutes. Eventually the new 
topology will bind to a new port (typically 6702) but the process can be 
repeated with more and more ports being full and the new topologies taking 
longer and longer to start.

> Ports of killed topologies remain in TIME_WAIT state preventing to start new 
> topology
> -------------------------------------------------------------------------------------
>
>                 Key: STORM-3039
>                 URL: https://issues.apache.org/jira/browse/STORM-3039
>             Project: Apache Storm
>          Issue Type: Improvement
>    Affects Versions: 1.1.2, 1.2.1
>            Reporter: Gergely Hajós
>            Assignee: Gergely Hajós
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> When topology is killed the slot ports (supervisor.slots.ports) remain in 
> TIME_WAIT state. In that case new topology can not be started, because 
> workers throw the following error:
> {code:java}
> 2018-04-20 08:37:08.742 o.a.s.d.worker main [ERROR] Error on initialization 
> of server mk-worker
> org.apache.storm.shade.org.jboss.netty.channel.ChannelException: Failed to 
> bind to: 0.0.0.0/0.0.0.0:6700
>  at 
> org.apache.storm.shade.org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at org.apache.storm.messaging.netty.Server.<init>(Server.java:101) 
> ~[storm-core-1.2.1.jar:1.2.1]
>  at org.apache.storm.messaging.netty.Context.bind(Context.java:67) 
> ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> org.apache.storm.daemon.worker$worker_data$fn__10395.invoke(worker.clj:285) 
> ~[storm-core-1.2.1.jar:1.2.1]
>  at org.apache.storm.util$assoc_apply_self.invoke(util.clj:931) 
> ~[storm-core-1.2.1.jar:1.2.1]
>  at org.apache.storm.daemon.worker$worker_data.invoke(worker.clj:282) 
> ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> org.apache.storm.daemon.worker$fn__10693$exec_fn__3301__auto__$reify__10695.run(worker.clj:626)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
>  at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_161]
>  at 
> org.apache.storm.daemon.worker$fn__10693$exec_fn__3301__auto____10694.invoke(worker.clj:624)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
>  at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
>  at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
>  at 
> org.apache.storm.daemon.worker$fn__10693$mk_worker__10784.doInvoke(worker.clj:598)
>  [storm-core-1.2.1.jar:1.2.1]
>  at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
>  at org.apache.storm.daemon.worker$_main.invoke(worker.clj:787) 
> [storm-core-1.2.1.jar:1.2.1]
>  at clojure.lang.AFn.applyToHelper(AFn.java:165) [clojure-1.7.0.jar:?]
>  at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?]
>  at org.apache.storm.daemon.worker.main(Unknown Source) 
> [storm-core-1.2.1.jar:1.2.1]
> Caused by: java.net.BindException: Address already in use
>  at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_161]
>  at sun.nio.ch.Net.bind(Net.java:433) ~[?:1.8.0_161]
>  at sun.nio.ch.Net.bind(Net.java:425) ~[?:1.8.0_161]
>  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
> ~[?:1.8.0_161]
>  at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
> ~[?:1.8.0_161]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>  ~[storm-core-1.2.1.jar:1.2.1]
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  ~[?:1.8.0_161]
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  ~[?:1.8.0_161]
>  at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_161]
> {code}
>  
> This exception occurs often when topologies stopped and started automatically.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to