Hi Storm users,
I encountered strange behavior of my storm cluster. When one of Nimbuses is
down (the second one is working correctly) the healthy Nimbus seems to work
ok (getting leadership if it was a follower etc.) however Rest API and
probably Nimbus doesn't work correctly. Getting response from Rest API
takes a couple of seconds (e.g. http://localhost:8080/api/v1/nimbus/summary
).
Nimbus logs looks ok, however ui log is like that (this logs appears in
loop until the second Nimbus machine is up):
A lot of: 018-07-19 13:13:55.580
o.a.s.u.StormBoundedExponentialBackoffRetry [WARN] WILL SLEEP FOR 2001ms
(NOT MAX)
and:
2018-07-19 13:13:55.620 o.a.s.u.NimbusClient [WARN] Ignoring exception
while trying to get leader nimbus info from 10.0.5.2. will retry with a
different seed host.
java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.storm.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at
org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:112)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:73)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:136)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:92)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:66)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.ui.core$all_topologies_summary.invoke(core.clj:509)
~[storm-core-1.2.2.jar:1.2.2]
at org.apache.storm.ui.core$fn__8445.invoke(core.clj:1144)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.compojure.core$make_route$fn__3195.invoke(core.clj:100)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.compojure.core$if_route$fn__3183.invoke(core.clj:46)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.compojure.core$if_method$fn__3176.invoke(core.clj:31)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.compojure.core$routing$fn__3201.invoke(core.clj:113)
~[storm-core-1.2.2.jar:1.2.2]
at clojure.core$some.invoke(core.clj:2570) ~[clojure-1.7.0.jar:?]
at
org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113)
~[storm-core-1.2.2.jar:1.2.2]
at clojure.lang.RestFn.applyTo(RestFn.java:139)
~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:632) ~[clojure-1.7.0.jar:?]
at
org.apache.storm.shade.compojure.core$routes$fn__3205.invoke(core.clj:118)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__7765.invoke(cors.clj:149)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__7712.invoke(json.clj:56)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__4416.invoke(multipart_params.clj:118)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__7241.invoke(reload.clj:22)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.ui.helpers$requests_middleware$fn__4669.invoke(helpers.clj:52)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.ui.core$catch_errors$fn__8632.invoke(core.clj:1428)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__4336.invoke(keyword_params.clj:35)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__4379.invoke(nested_params.clj:84)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.params$wrap_params$fn__4308.invoke(params.clj:64)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__4416.invoke(multipart_params.clj:118)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__4631.invoke(flash.clj:35)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.session$wrap_session$fn__4617.invoke(session.clj:98)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.middleware.session$wrap_session$fn__4617.invoke(session.clj:98)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.util.servlet$make_service_method$fn__4166.invoke(servlet.clj:127)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.util.servlet$servlet$fn__4170.invoke(servlet.clj:136)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown
Source) ~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
~[storm-core-1.2.2.jar:1.2.2]
at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.7.0.jar:?]
at
org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__4762.invoke(helpers.clj:192)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown
Source) ~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
~[storm-core-1.2.2.jar:1.2.2]
Caused by: java.lang.RuntimeException:
org.apache.storm.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at
org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:104)
~[storm-core-1.2.2.jar:1.2.2]
... 64 more
Caused by: org.apache.storm.thrift.transport.TTransportException:
java.net.ConnectException: Connection refused (Connection refused)
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:105)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:104)
~[storm-core-1.2.2.jar:1.2.2]
... 64 more
Caused by: java.net.ConnectException: Connection refused (Connection
refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
~[?:1.8.0_171]
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
~[?:1.8.0_171]
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
~[?:1.8.0_171]
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
~[?:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
~[?:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_171]
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:105)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
~[storm-core-1.2.2.jar:1.2.2]
at
org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:104)
~[storm-core-1.2.2.jar:1.2.2]
... 64 more
2018-07-19 13:13:55.624 o.a.s.u.NimbusClient [INFO] Found leader nimbus :
10.0.5.2:6627
Topology and supervisors works correctly all the time.
My storm config:
Storm version (1.0.1, tested also with 1.2.2), 3 zookeepers, 2 Nimbuses, 2
Supervisors.
Default ports configuration.
I also noticed that if nimbus seeds are ordered it seems to work ok. E.g.
let's have 10.0.5.2 and 10.0.5.3 seeds. On Nimbus 10.0.5.2 seeds look like
["10.0.5.2", "10.0.5.3"], and on 10.0.5.3 ["10.0.5.3", "10.0.5.2"]. When
such config is applied I don't get this logs with connection refused and
everything looks good. However depending on order is awkward.
Can you help me with finding the solution?
Best regards,
Luk.
