Hi,
I am not sure what could cause this, but I am observing this behaviour
consistently. Everything was working fine till we added a feature in our
deployment script to automatically kill a topology if deployed (with wait
only for 1 sec to kill topology, *killOptions.set_wait_secs(1)*) and then
deploy the topology again waiting 30 sec before deploying.
The issue is after a while (ranging from 1 hour to 6+ hours), only the one
of our topology gets undeployed automatically, I don't know what is causing
this to getting undeployed? The other topology is still running fine.
I do not see anything wrong in the storm nimbus log and supervisor logs.
Only thing I see is a messaging getting logged that toplogy is getting
undeployed:
Does anyone has any insight as to what is causing this? We add this auto
deploy feature in our script, which waits for having nimbus up and certain
number of supervisors running, if so then it undeploys topology and deploy
them again.
Nimbus log:
*CURRENTLY DEPLOYED TOPOLOGY IS KILLED with killOptions.set_wait_secs(1):*
*12 Jul 2014 01:16:19,152 INFO [pool-4-thread-13]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Delaying event :remove for
5 secs for event-system-topology-5-140512770812 Jul 2014 01:16:19,175 INFO
[pool-4-thread-13] (clojure.tools.logging$eval1$fn__7.invoke:0) - Updated
event-system-topology-5-1405127708 with status {:type :killed,
:kill-time-secs 5}12 Jul 2014 01:16:24,607 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Killing topology:
event-system-topology-5-1405127708*
12 Jul 2014 01:16:26,652 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Cleaning up
event-system-topology-5-1405127708
12 Jul 2014 01:17:38,223 INFO [pool-4-thread-28]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Uploading file from client
to /var/lib/storm/nimbus/inbox/storm
jar-713ab63d-93e3-4040-a05e-49faed71614b.jar
12 Jul 2014 01:17:38,719 INFO [pool-4-thread-60]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Finished uploading file
from client: /var/lib/storm/nimbus/inbo
x/stormjar-713ab63d-93e3-4040-a05e-49faed71614b.jar
*TOPOLOGY IS SUBMITTED AGAIN:*
12 Jul 2014 01:17:38,737 INFO [pool-4-thread-53]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
*Received topology submission for event-system-topology with con*f {"
storm.id" "event-system-topology-6-1405127858", "nimbus.host" "localhost",
"eventClientRetryInterval" "2000", "topology.acker.executors" 0, "
topology.kryo.de
corators" (), "topology.name" "event-system-topology",
"tCollectorHostNamePortList" "localhost:4242",
"tCollectorMetricReportingIntervalInSeconds" "60", "eventCl
ientMaxQueueSize" "10000", "topology.kryo.register" nil, "topology.workers"
5, "nimbus.thrift.port" 6627, "eventCodePrefix" "050", "eventClientTimeout"
"3000", "
topology.max.task.parallelism" nil, "eventEndpointUri" "
http://localhost:8086/events"}
12 Jul 2014 01:17:38,789 INFO [pool-4-thread-53]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Activating
event-system-topology: event-system-topology-6-14051
27858
12 Jul 2014 01:17:38,851 INFO [pool-4-thread-53]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Available slots:
(["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700])
12 Jul 2014 01:17:38,864 INFO [pool-4-thread-53]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Setting new assignment for
topology id event-system-topology-6-1405127858:
#backtype.storm.daemon.common.Assignment{:master-code-dir
"/var/lib/storm/nimbus/stormdist/event-system-topology-6-1405127858",
:node->host {"3bdc4df5-145b-4929-97e2-82a8e08d6951" "
storm3.r6.03c.snaa.gaikai.org", "4088a57f-883c-4be9-9609-b412dc00b6a4" "
storm2.r6.03c.snaa.gaikai.org"}, :executor->node+port {[2 2]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [34 34]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [66 66]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [98 98]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [3 3]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [35 35]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [67 67]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [99 99]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [4 4]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [36 36]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [68 68]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [100 100]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [5 5]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [37 37]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [69 69]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [101 101]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [6 6]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [38 38]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [70 70]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [102 102]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [7 7]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [39 39]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [71 71]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [103 103]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [8 8]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [40 40]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [72 72]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [104 104]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [9 9]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [41 41]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [73 73]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [105 105]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [10 10]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [42 42]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [74 74]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [106 106]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [11 11]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [43 43]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [75 75]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [107 107]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [12 12]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [44 44]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [76 76]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [108 108]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [13 13]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [45 45]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [77 77]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [109 109]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [14 14]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [46 46]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [78 78]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [110 110]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [15 15]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [47 47]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [79 79]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [111 111]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [16 16]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [48 48]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [80 80]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [112 112]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [17 17]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [49 49]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [81 81]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [113 113]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [18 18]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [50 50]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [82 82]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [114 114]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [19 19]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [51 51]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [83 83]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [115 115]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [20 20]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [52 52]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [84 84]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [116 116]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [21 21]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [53 53]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [85 85]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [117 117]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [22 22]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [54 54]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [86 86]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [118 118]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [23 23]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [55 55]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [87 87]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [119 119]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [24 24]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [56 56]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [88 88]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [120 120]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [25 25]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [57 57]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [89 89]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [121 121]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [26 26]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [58 58]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [90 90]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [122 122]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [27 27]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [59 59]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [91 91]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [123 123]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [28 28]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [60 60]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [92 92]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [124 124]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [29 29]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [61 61]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [93 93]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [125 125]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [30 30]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [62 62]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [94 94]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6701], [31 31]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [63 63]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [95 95]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6700], [32 32]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [64 64]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [96 96]
["4088a57f-883c-4be9-9609-b412dc00b6a4" 6700], [1 1]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [33 33]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [65 65]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701], [97 97]
["3bdc4df5-145b-4929-97e2-82a8e08d6951" 6701]}, :executor->start-time-secs
{[2 2] 1405127858, [34 34] 1405127858, [66 66] 1405127858, [98 98]
1405127858, [3 3] 1405127858, [35 35] 1405127858, [67 67] 1405127858, [99
99] 1405127858, [4 4] 1405127858, [36 36] 1405127858, [68 68] 1405127858,
[100 100] 1405127858, [5 5] 1405127858, [37 37] 1405127858, [69 69]
1405127858, [101 101] 1405127858, [6 6] 1405127858, [38 38] 1405127858, [70
70] 1405127858, [102 102] 1405127858, [7 7] 1405127858, [39 39] 1405127858,
[71 71] 1405127858, [103 103] 1405127858, [8 8] 1405127858, [40 40]
1405127858, [72 72] 1405127858, [104 104] 1405127858, [9 9] 1405127858, [41
41] 1405127858, [73 73] 1405127858, [105 105] 1405127858, [10 10]
1405127858, [42 42] 1405127858, [74 74] 1405127858, [106 106] 1405127858,
[11 11] 1405127858, [43 43] 1405127858, [75 75] 1405127858, [107 107]
1405127858, [12 12] 1405127858, [44 44] 1405127858, [76 76] 1405127858,
[108 108] 1405127858, [13 13] 1405127858, [45 45] 1405127858, [77 77]
1405127858, [109 109] 1405127858, [14 14] 1405127858, [46 46] 1405127858,
[78 78] 1405127858, [110 110] 1405127858, [15 15] 1405127858, [47 47]
1405127858, [79 79] 1405127858, [111 111] 1405127858, [16 16] 1405127858,
[48 48] 1405127858, [80 80] 1405127858, [112 112] 1405127858, [17 17]
1405127858, [49 49] 1405127858, [81 81] 1405127858, [113 113] 1405127858,
[18 18] 1405127858, [50 50] 1405127858, [82 82] 1405127858, [114 114]
1405127858, [19 19] 1405127858, [51 51] 1405127858, [83 83] 1405127858,
[115 115] 1405127858, [20 20] 1405127858, [52 52] 1405127858, [84 84]
1405127858, [116 116] 1405127858, [21 21] 1405127858, [53 53] 1405127858,
[85 85] 1405127858, [117 117] 1405127858, [22 22] 1405127858, [54 54]
1405127858, [86 86] 1405127858, [118 118] 1405127858, [23 23] 1405127858,
[55 55] 1405127858, [87 87] 1405127858, [119 119] 1405127858, [24 24]
1405127858, [56 56] 1405127858, [88 88] 1405127858, [120 120] 1405127858,
[25 25] 1405127858, [57 57] 1405127858, [89 89] 1405127858, [121 121]
1405127858, [26 26] 1405127858, [58 58] 1405127858, [90 90] 1405127858,
[122 122] 1405127858, [27 27] 1405127858, [59 59] 1405127858, [91 91]
1405127858, [123 123] 1405127858, [28 28] 1405127858, [60 60] 1405127858,
[92 92] 1405127858, [124 124] 1405127858, [29 29] 1405127858, [61 61]
1405127858, [93 93] 1405127858, [125 125] 1405127858, [30 30] 1405127858,
[62 62] 1405127858, [94 94] 1405127858, [31 31] 1405127858, [63 63]
1405127858, [95 95] 1405127858, [32 32] 1405127858, [64 64] 1405127858, [96
96] 1405127858, [1 1] 1405127858, [33 33] 1405127858, [65 65] 1405127858,
[97 97] 1405127858}}
12 Jul 2014 02:03:36,949 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Cleaning inbox ... deleted:
stormjar-6bc462ca-1d60-41d0-9d7a-acd91a9e9b0f.jar
12 Jul 2014 02:23:36,358 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Cleaning inbox ... deleted:
stormjar-713ab63d-93e3-4040-a05e-49faed71614b.jar
12 Jul 2014 02:23:36,364 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Cleaning inbox ... deleted:
stormjar-cd0a6364-3679-48f5-b57b-56263634cb28.jar
12 Jul 2014 07:20:10,425 INFO [pool-4-thread-36]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Delaying event :remove for
1 secs for event-system-topology-6-1405127858
12 Jul 2014 07:20:10,431 INFO [pool-4-thread-36]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Updated
event-system-topology-6-1405127858 with status {:type :killed,
:kill-time-secs 1}
*TOPOLOGY IS GETTING KILLED AUTOMATICALLY, NOT SURE WHY?*
*12 Jul 2014 07:20:11,420 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Killing topology:
event-system-topology-6-1405127858*
12 Jul 2014 07:20:13,462 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Cleaning up
event-system-topology-6-1405127858
Supervisor log:
*GOT NEW TOPOLOGY JAR FOR DEPLOYMENT:12 Jul 2014 01:15:08,339 INFO
[Thread-2] (clojure.tools.logging$eval1$fn__7.invoke:0) - Downloading code
for storm id event-system-topology-5-1405127708 from
/var/lib/storm/nimbus/stormdist/event-system-topology-5-1405127708*
12 Jul 2014 01:15:10,252 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Could not extract resources
from /var/lib/storm/s
upervisor/tmp/63e4d168-0e83-4135-8ee1-87cc54fb01b3/stormjar.jar
12 Jul 2014 01:15:10,253 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Finished downloading code
for storm id event-syst
em-topology-5-1405127708 from
/var/lib/storm/nimbus/stormdist/event-system-topology-5-1405127708
12 Jul 2014 01:15:10,260 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Launching worker with
assignment #backtype.storm.
daemon.supervisor.LocalAssignment{:storm-id
"event-system-topology-5-1405127708", :executors ([3 3] [35 35] [67 67] [99
99] [7 7] [39 39] [
71 71] [103 103] [11 11] [43 43] [75 75] [107 107] [15 15] [47 47] [79 79]
[111 111] [19 19] [51 51] [83 83] [115 115] [23 23] [55 55] [87
87] [119 119] [27 27] [59 59] [91 91] [123 123] [31 31] [63 63] [95 95])}
for this supervisor 3bdc4df5-145b-4929-97e2-82a8e08d6951 on port
6700 with id 5c2f90e7-d550-47b2-8ff3-9f7cb27f472d
12 Jul 2014 01:15:10,260 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) - *Launching worker* with
command: java -server -Dco
m.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.access.file=/etc/storm/jmx.access
-Dcom.sun.management.jmxremot
e.password.file=/etc/storm/jmx.passwd
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=16700 -Djava.library.pa
th=/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker-6700.log
-Dstorm.home=/usr/share/storm -Dlog4j.configuration=storm.log.prop
erties -cp
/usr/share/storm/storm-0.8.2.jar:/usr/share/storm/lib/commons-exec-1.1.jar:/usr/share/storm/lib/ring-core-1.1.5.jar:/usr/share/s
torm/lib/kryo-2.17.jar:/usr/share/storm/lib/asm-4.0.jar:/usr/share/storm/lib/math.numeric-tower-0.0.1.jar:/usr/share/storm/lib/commons-io-2
.jar:/usr/share/storm/lib/carbonite-1.5.0.jar:/usr/share/storm/lib/log4j-1.2.16.jar:/usr/share/storm/lib/clout-1.0.1.jar:/usr/share/storm/l
ib/slf4j-log4j12-1.5.8.jar:/usr/share/storm/lib/log4j-cee.jar:/usr/share/storm/lib/guava-14.jar:/usr/share/storm/lib/commons-fileupload-1.2
.1.jar:/usr/share/storm/lib/commons-lang-2.5.jar:/usr/share/storm/lib/tools.macro-0.1.0.jar:/usr/share/storm/lib/jgrapht-0.8.3.jar:/usr/sha
re/storm/lib/zookeeper-3.3.3.jar:/usr/share/storm/lib/jetty-6.1.26.jar:/usr/share/storm/lib/json-simple-1.1.jar:/usr/share/storm/lib/jackso
n-annotations.jar:/usr/share/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/share/storm/lib/jackson-databind.jar:/usr/share/storm/lib/curator
-client-1.0.1.jar:/usr/share/storm/lib/commons-codec-1.4.jar:/usr/share/storm/lib/clojure-1.4.0.jar:/usr/share/storm/lib/joda-time-2.0.jar:
/usr/share/storm/lib/jetty-util-6.1.26.jar:/usr/share/storm/lib/objenesis-1.2.jar:/usr/share/storm/lib/tools.cli-0.2.2.jar:/usr/share/storm
/lib/httpcore-4.1.jar:/usr/share/storm/lib/clj-time-0.4.1.jar:/usr/share/storm/lib/minlog-1.2.jar:/usr/share/storm/lib/tools.logging-0.2.3.
jar:/usr/share/storm/lib/curator-framework-1.0.1.jar:/usr/share/storm/lib/hiccup-0.3.6.jar:/usr/share/storm/lib/slf4j-api-1.5.8.jar:/usr/sh
are/storm/lib/httpclient-4.1.1.jar:/usr/share/storm/lib/core.incubator-0.1.0.jar:/usr/share/storm/lib/jackson-core.jar:/usr/share/storm/lib
/junit-3.8.1.jar:/usr/share/storm/lib/disruptor-2.10.1.jar:/usr/share/storm/lib/servlet-api-2.5-20081211.jar:/usr/share/storm/lib/commons-l
ogging-1.1.1.jar:/usr/share/storm/lib/jzmq-2.1.0.jar:/usr/share/storm/lib/libthrift7-0.7.0.jar:/usr/share/storm/lib/log4j.jar:/usr/share/st
orm/lib/jline-0.9.94.jar:/usr/share/storm/lib/compojure-1.1.3.jar:/usr/share/storm/lib/snakeyaml-1.9.jar:/usr/share/storm/lib/ring-servlet-
0.3.11.jar:/usr/share/storm/lib/reflectasm-1.07-shaded.jar:/usr/share/storm/lib/servlet-api-2.5.jar:/usr/share/storm/log4j:/usr/share/storm
/conf:/var/lib/storm/supervisor/stormdist/event-system-topology-5-1405127708/stormjar.jar
backtype.storm.daemon.worker event-system-topolog
y-5-1405127708 3bdc4df5-145b-4929-97e2-82a8e08d6951 6700
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d
12 Jul 2014 01:15:10,263 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Launching worker with
assignment #backtype.storm.
daemon.supervisor.LocalAssignment{:storm-id
"event-system-topology-5-1405127708", :executors ([5 5] [37 37] [69 69]
[101 101] [9 9] [41 41]
[73 73] [105 105] [13 13] [45 45] [77 77] [109 109] [17 17] [49 49] [81
81] [113 113] [21 21] [53 53] [85 85] [117 117] [25 25] [57 57] [8
9 89] [121 121] [29 29] [61 61] [93 93] [125 125] [1 1] [33 33] [65 65] [97
97])} for this supervisor 3bdc4df5-145b-4929-97e2-82a8e08d6951
on port 6701 with id 0e995cd5-41c8-4072-8b72-1da2cce0a274
12 Jul 2014 01:15:10,263 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Launching worker with
command: java -server -Dco
m.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.access.file=/etc/storm/jmx.access
-Dcom.sun.management.jmxremot
e.password.file=/etc/storm/jmx.passwd
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=16701 -Djava.library.pa
th=/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker-6701.log
-Dstorm.home=/usr/share/storm -Dlog4j.configuration=storm.log.prop
erties -cp
/usr/share/storm/storm-0.8.2.jar:/usr/share/storm/lib/commons-exec-1.1.jar:/usr/share/storm/lib/ring-core-1.1.5.jar:/usr/share/s
torm/lib/kryo-2.17.jar:/usr/share/storm/lib/asm-4.0.jar:/usr/share/storm/lib/math.numeric-tower-0.0.1.jar:/usr/share/storm/lib/commons-io-2
.jar:/usr/share/storm/lib/carbonite-1.5.0.jar:/usr/share/storm/lib/log4j-1.2.16.jar:/usr/share/storm/lib/clout-1.0.1.jar:/usr/share/storm/l
ib/slf4j-log4j12-1.5.8.jar:/usr/share/storm/lib/log4j-cee.jar:/usr/share/storm/lib/guava-14.jar:/usr/share/storm/lib/commons-fileupload-1.2
.1.jar:/usr/share/storm/lib/commons-lang-2.5.jar:/usr/share/storm/lib/tools.macro-0.1.0.jar:/usr/share/storm/lib/jgrapht-0.8.3.jar:/usr/sha
re/storm/lib/zookeeper-3.3.3.jar:/usr/share/storm/lib/jetty-6.1.26.jar:/usr/share/storm/lib/json-simple-1.1.jar:/usr/share/storm/lib/jackso
n-annotations.jar:/usr/share/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/share/storm/lib/jackson-databind.jar:/usr/share/storm/lib/curator
-client-1.0.1.jar:/usr/share/storm/lib/commons-codec-1.4.jar:/usr/share/storm/lib/clojure-1.4.0.jar:/usr/share/storm/lib/joda-time-2.0.jar:
/usr/share/storm/lib/jetty-util-6.1.26.jar:/usr/share/storm/lib/objenesis-1.2.jar:/usr/share/storm/lib/tools.cli-0.2.2.jar:/usr/share/storm
/lib/httpcore-4.1.jar:/usr/share/storm/lib/clj-time-0.4.1.jar:/usr/share/storm/lib/minlog-1.2.jar:/usr/share/storm/lib/tools.logging-0.2.3.
jar:/usr/share/storm/lib/curator-framework-1.0.1.jar:/usr/share/storm/lib/hiccup-0.3.6.jar:/usr/share/storm/lib/slf4j-api-1.5.8.jar:/usr/sh
are/storm/lib/httpclient-4.1.1.jar:/usr/share/storm/lib/core.incubator-0.1.0.jar:/usr/share/storm/lib/jackson-core.jar:/usr/share/storm/lib/junit-3.8.1.jar:/usr/share/storm/lib/disruptor-2.10.1.jar:/usr/share/storm/lib/servlet-api-2.5-20081211.jar:/usr/share/storm/lib/commons-logging-1.1.1.jar:/usr/share/storm/lib/jzmq-2.1.0.jar:/usr/share/storm/lib/libthrift7-0.7.0.jar:/usr/share/storm/lib/log4j.jar:/usr/share/storm/lib/jline-0.9.94.jar:/usr/share/storm/lib/compojure-1.1.3.jar:/usr/share/storm/lib/snakeyaml-1.9.jar:/usr/share/storm/lib/ring-servlet-0.3.11.jar:/usr/share/storm/lib/reflectasm-1.07-shaded.jar:/usr/share/storm/lib/servlet-api-2.5.jar:/usr/share/storm/log4j:/usr/share/storm/conf:/var/lib/storm/supervisor/stormdist/event-system-topology-5-1405127708/stormjar.jar
backtype.storm.daemon.worker event-system-topology-5-1405127708
3bdc4df5-145b-4929-97e2-82a8e08d6951 6701
0e995cd5-41c8-4072-8b72-1da2cce0a274
12 Jul 2014 01:15:10,264 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:10,765 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:11,266 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:11,767 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:12,268 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:12,768 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:13,269 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:13,770 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:14,270 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:15:14,773 INFO [Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
5c2f90e7-d550-47b2-8ff3-9f7cb27f472d still hasn't started
12 Jul 2014 01:16:24,628 INFO [Thread-2]
(clojure.tools.logging$eval1$fn__7.invoke:0) -
*TOPOLOGY IS GETTING KILLED AUTOMATICALLY, NOT SURE WHY?*
*Removing code for storm id event-system-topology-5-140512770812 Jul 2014
01:16:24,637 INFO *[Thread-3]
(clojure.tools.logging$eval1$fn__7.invoke:0) - Shutting down and clearing
state for id 5c2f90e7-d550-47b2-8ff3-9f7cb27f472d. Current supervisor time:
1405127784. State: :disallowed, Heartbeat:
#backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1405127784,
:storm-id "event-system-topology-5-1405127708", :executors #{[3 3] [35 35]
[67 67] [99 99] [7 7] [39 39] [71 71] [103 103] [11 11] [43 43] [75 75]
[107 107] [15 15] [47 47] [79 79] [111 111] [19 19] [51 51] [83 83] [115
115] [23 23] [55 55] [87 87] [119 119] [27 27] [59 59] [91 91] [123 123]
[31 31] [63 63] [95 95]}, :port 6700}
