I found  workers in cluster always restart themselves and the executors up
time is never more than 2 minutes.
In every restart loop, the workers run well after restart, where it execute
and emit tuples normally. Then when the uptime is near 2 minutes, all
statistic data is reset to zero including the up time.

Here is the error log before every restart behavior:

2016-09-27 17:10:38.831 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm08/132.228.28.151:6734
2016-09-27 17:10:38.831 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm08/132.228.28.151:6731
2016-09-27 17:10:38.831 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm08/132.228.28.151:6731
2016-09-27 17:10:38.832 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm08/132.228.28.151:6738
2016-09-27 17:10:38.832 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm08/132.228.28.151:6738
2016-09-27 17:10:38.833 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm08/132.228.28.151:6741
2016-09-27 17:10:38.833 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm08/132.228.28.151:6741
2016-09-27 17:10:38.833 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm08/132.228.28.151:6737
2016-09-27 17:10:38.833 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm08/132.228.28.151:6737
2016-09-27 17:10:38.834 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm08/132.228.28.151:6740
2016-09-27 17:10:38.834 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm08/132.228.28.151:6740
2016-09-27 17:10:38.834 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm08/132.228.28.151:6751
2016-09-27 17:10:38.834 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm08/132.228.28.151:6751
2016-09-27 17:10:40.894 o.a.s.d.worker [INFO] Shutting down worker
DPI-10-1474620300 e0060b80-1f9b-4ea3-8ccf-fec15a2293d9 6737
2016-09-27 17:10:40.895 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm07/132.228.28.150:6748
2016-09-27 17:10:40.895 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm07/132.228.28.150:6748
2016-09-27 17:10:40.895 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm02/132.228.28.137:6737
2016-09-27 17:10:40.896 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm02/132.228.28.137:6737
2016-09-27 17:10:40.896 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm07/132.228.28.150:6733
2016-09-27 17:10:40.896 o.a.s.m.n.Client [INFO] waiting up to 600000 ms to
send 0 pending messages to Netty-Client-storm07/132.228.28.150:6733
2016-09-27 17:10:40.896 o.a.s.m.n.Client [INFO] closing Netty Client
Netty-Client-storm07/132.228.28.150:6752


Regard,
Junfeng Chen

Reply via email to