> How can I check whether bolts are dying? You can have a look at the worker log file at 15.50.53.52 <http://15.50.53.52:6703/>.
There were some issues around the Netty Client that got fixed recently (HEAD of 0.9.x-branch). With the change, it also logs the cause of connection error that wasn't logged before. So IMO there is a very slim chance that this fix will fix it (I don't know why it would though), and if it doesn't, at least you may get to see the cause of connection error (albeit it may not be that informative). So in conclusion... if you found bolts dying, you should fix that. If not, you could try the HEAD of 0.9.x-branch and see if it fixes it / if you see new information in the log. On Mon, Jul 20, 2015 at 2:28 PM, 张炜 <[email protected]> wrote: > Thanks for the reply. How can I check whether bolts are dying? > The scenario happened every time when I started my topology, before kafka > messages are sent out. I don't understand why "Bolts at the destination > dies". > > I hope this can be fixed, because this occurred every time. > > On Mon, Jul 20, 2015 at 6:31 PM Enno Shioji <[email protected]> wrote: > >> It can happen when the bolts at the destination dies. Do you see logs >> indicating that bolts are dying on 15.50.53.52:6703? >> >> If it doesn't happen frequently, I wouldn't worry too much as there are >> mechanisms to cope with bolt deaths. If there is some systemic problem >> (like a skew blowing up the memory and killing bolts again and again), then >> you probably want to fix that. >> >> >> On Mon, Jul 20, 2015 at 11:21 AM, 张炜 <[email protected]> wrote: >> >>> Hi all, >>> I did see the Giving up information, as shown below. >>> >>> May I know why this will happen and how to solve this? >>> I have several bolts with parallelism 50, most of them are successful >>> established, but there are a few like this. >>> >>> >>> 2015-07-20 10:07:01.084 STDIO [ERROR] Jul 20, 2015 10:07:01 AM >>> org.apache.storm.guava.util.concurrent.ExecutionList executeListener >>> SEVERE: RuntimeException while executing runnable >>> org.apache.storm.guava.util.concurrent.Futures$4@1aa49e9a with executor >>> org.apache.storm.guava.util.concurrent.MoreExecutors$SameThreadExecutorService@2da89e20 >>> java.lang.RuntimeException: Failed to connect to >>> Netty-Client-c9t07982.itcs.****.com/15.50.53.52:6703 >>> at backtype.storm.messaging.netty.Client.connect(Client.java:300) >>> at backtype.storm.messaging.netty.Client.access$1100(Client.java:66) >>> at >>> backtype.storm.messaging.netty.Client$2.reconnectAgain(Client.java:289) >>> at backtype.storm.messaging.netty.Client$2.onSuccess(Client.java:275) >>> at backtype.storm.messaging.netty.Client$2.onSuccess(Client.java:267) >>> at >>> org.apache.storm.guava.util.concurrent.Futures$4.run(Futures.java:1181) >>> at >>> org.apache.storm.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) >>> at >>> org.apache.storm.guava.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) >>> at >>> org.apache.storm.guava.util.concurrent.ExecutionList.execute(ExecutionList.java:145) >>> at >>> org.apache.storm.guava.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91) >>> at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:380) >>> at java.util.concurrent.FutureTask.set(FutureTask.java:229) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:270) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: java.lang.RuntimeException: Giving up to connect to >>> Netty-Client-c9t07982.itcs.****.com/15.50.53.52:6703 after 12 failed >>> attempts >>> at backtype.storm.messaging.netty.Client.connect(Client.java:295) >>> ... 19 more >>> >>> >>> Regards, >>> Sai >>> >>> >>> On Thu, Jul 2, 2015 at 8:53 AM 임정택 <[email protected]> wrote: >>> >>>> Hi. >>>> >>>> Seems like these are logs which occurs in progress of connecting. >>>> (We may want to lower its log level cause it doesn't mean we give up to >>>> connect, just attempting.) >>>> >>>> Your worker should print either "Giving up to connect to ", or "connection >>>> established to ". >>>> >>>> Former means that worker gave up connecting to other worker, and that >>>> worker kills itself. >>>> If you received this log, you may want to check that these are able to >>>> connect each other. >>>> >>>> Latter means worker succeed to connect other worker, so it is safe. >>>> >>>> Hope this helps. >>>> >>>> Thanks, >>>> Jungtaek Lim (HeartSaVioR) >>>> >>>> >>>> >>>> 2015-07-02 8:59 GMT+09:00 张炜 <[email protected]>: >>>> >>>>> Hi all, >>>>> I met strange problems and below are some typical log output. >>>>> Could you please help me know where the problem is? Or could anyone >>>>> guide me which areas that I can start to investigate and where to find >>>>> related materials? >>>>> >>>>> These things look very internal to me that I don't know where to start. >>>>> Thank you very much! >>>>> >>>>> Regards, >>>>> Sai >>>>> >>>>> 2015-07-01T01:42:53.290+0000 b.s.d.executor [INFO] Shutting down executor >>>>> mycheck:[97 97] >>>>> 2015-07-01T01:42:53.290+0000 b.s.util [INFO] Async loop interrupted! >>>>> 2015-07-01T01:42:53.291+0000 b.s.util [INFO] Async loop interrupted! >>>>> 2015-07-01T01:42:53.291+0000 b.s.d.executor [INFO] Shut down executor >>>>> mycheck:[97 97] >>>>> >>>>> >>>>> 2015-07-01T01:42:52.868+0000 b.s.m.n.Client [INFO] connection attempt 1 >>>>> to Netty-Client-localhost/127.0.0.1:6702 scheduled to run in 0 ms >>>>> 2015-07-01T01:42:52.875+0000 b.s.m.n.Client [INFO] connection established >>>>> to Netty-Client-localhost/127.0.0.1:6702 >>>>> 2015-07-01T01:42:52.889+0000 b.s.m.loader [INFO] Shutting down >>>>> receiving-thread: [Topology-2-1435680709, 6702] >>>>> 2015-07-01T01:42:52.891+0000 b.s.m.n.Client [INFO] closing Netty Client >>>>> Netty-Client-localhost/127.0.0.1:6702 >>>>> 2015-07-01T01:42:52.891+0000 b.s.m.n.Client [INFO] waiting up to 600000 >>>>> ms to send 1 pending messages to Netty-Client-localhost/127.0.0.1:6702 >>>>> 2015-07-01T15:49:30.089+0000 o.a.s.c.r.ExponentialBackoffRetry [WARN] >>>>> maxRetries too large (300). Pinning to 29 >>>>> 2015-07-01T15:49:30.132+0000 o.a.s.c.r.ExponentialBackoffRetry [WARN] >>>>> maxRetries too large (300). Pinning to 29 >>>>> 2015-07-01T15:49:30.148+0000 o.a.s.c.r.ExponentialBackoffRetry [WARN] >>>>> maxRetries too large (300). Pinning to 29 >>>>> 2015-07-01T15:49:30.150+0000 o.a.s.c.r.ExponentialBackoffRetry [WARN] >>>>> maxRetries too large (300). Pinning to 29 >>>>> 2015-07-01T15:49:30.151+0000 o.a.s.c.r.ExponentialBackoffRetry [WARN] >>>>> maxRetries too large (300). Pinning to 29 >>>>> 2015-07-01T15:49:30.268+0000 b.s.m.n.Client [ERROR] connection attempt 1 >>>>> to Netty-Client- failed: java.lang.RuntimeException: Returned channel was >>>>> actually not established >>>>> 2015-07-01T15:49:30.291+0000 b.s.m.n.Client [ERROR] connection attempt 1 >>>>> to Netty-Client-6703 failed: java.lang.RuntimeException: Returned channel >>>>> was actually not established >>>>> 2015-07-01T15:49:30.379+0000 b.s.m.n.Client [ERROR] connection attempt 2 >>>>> to Netty-Client- failed: java.lang.RuntimeException: Returned channel was >>>>> actually not established >>>>> 2015-07-01T15:49:30.886+0000 b.s.m.n.Client [ERROR] connection attempt 2 >>>>> to Netty-Client-/15.50.46.234:6703 failed: java.lang.RuntimeException: >>>>> Returned channel was actually not established >>>>> 2015-07-01T15:49:30.912+0000 b.s.m.n.Client [ERROR] connection attempt 3 >>>>> to Netty-Client-:6702 failed: java.lang.RuntimeException: Returned >>>>> channel was actually not established >>>>> 2015-07-01T15:49:31.010+0000 b.s.m.n.Client [ERROR] connection attempt 3 >>>>> to Netty-Client-:6703 failed: java.lang.RuntimeException: Returned >>>>> channel was actually not established >>>>> 2015-07-01T15:49:31.047+0000 b.s.m.n.Client [ERROR] connection attempt 4 >>>>> to Netty-Client-:6702 failed: java.lang.RuntimeException: Returned >>>>> channel was actually not established >>>>> >>>>> >>>> >>>> >>>> -- >>>> Name : 임 정택 >>>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >>>> Twitter : http://twitter.com/heartsavior >>>> LinkedIn : http://www.linkedin.com/in/heartsavior >>>> >>> >>
