Hi,
I've set up a small Storm cluster on AWS that is giving me headaches. At
the moment I've got a situation where the cluster (a nimbus node and 3
supervisors) will successfully run in local mode my particular topology.
However, as soon as I try to run the topology in distributed mode there are
no errors (that I can see) in any of the logs - the topology appears in the
UI but never shows any tuples emitted by the spouts.
In order to check it wasn't my topology I cloned the storm-starter project
and have tried with the RollingTopWords example - running locally I get a
stream of output but as soon as I specify that we should run in a
distributed fashion I get a success message (topology successfully
submitted). The topology appears on the UI as successfully submitted but I
see no progress.
It's a bit of a weird one as I don't know which logs to share to show the
problem as the whole point is I'm not seeing any errors!
The last entry in the supervisor logs pertains to the set up of the
supervisor (i.e. it's not logged anything about the topologies that have
been submitted)
The last entries in the nimbus logs are probably a bit more illuminating:
2015-04-13 14:09:25 b.s.d.nimbus [INFO] Executor
production-topology-2-1428932653:[2 2] not alive
2015-04-13 14:09:25 b.s.d.nimbus [INFO] Executor
production-topology-2-1428932653:[3 3] not alive
2015-04-13 14:09:25 b.s.d.nimbus [INFO] Executor
production-topology-2-1428932653:[4 4] not alive
2015-04-13 14:09:25 b.s.s.EvenScheduler [INFO] Available slots:
(["bd340179-5905-4233-ad5d-44b47b228177" 6702]
["bd340179-5905-4233-ad5d-44b47b228177" 6701]
["bd340179-5905-4233-ad5d-44b47b228177" 6700]
["9de97b6f-3d3f-4797-8ac0-e89501de48d3" 6703]
["9de97b6f-3d3f-4797-8ac0-e89501de48d3" 6702]
["9de97b6f-3d3f-4797-8ac0-e89501de48d3" 6701]
["9de97b6f-3d3f-4797-8ac0-e89501de48d3" 6700]
["2874d18d-e0d1-46e0-a50d-85ce645ec4aa" 6703]
["2874d18d-e0d1-46e0-a50d-85ce645ec4aa" 6702]
["2874d18d-e0d1-46e0-a50d-85ce645ec4aa" 6701]
["2874d18d-e0d1-46e0-a50d-85ce645ec4aa" 6700])
2015-04-13 14:09:25 b.s.d.nimbus [INFO] Reassigning
production-topology-2-1428932653 to 1 slots
2015-04-13 14:09:25 b.s.d.nimbus [INFO] Reassign executors: [[2 2] [3 3] [4
4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10] [11 11] [12 12] [13 13] [14 14]
[15 15] [1 1]]
2015-04-13 14:09:25 b.s.d.nimbus [INFO] Setting new assignment for topology
id production-topology-2-1428932653:
#backtype.storm.daemon.common.Assignment{:master-code-dir
"/app/storm/nimbus/stormdist/production-topology-2-1428932653", :node->host
{"bd340179-5905-4233-ad5d-44b47b228177" "*.compute.amazonaws.com"},
:executor->node+port {[2 2] ["bd340179-5905-4233-ad5d-44b47b228177" 6702],
[3 3] ["bd340179-5905-4233-ad5d-44b47b228177" 6702], [4 4]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [5 5]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [6 6]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [7 7]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [8 8]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [9 9]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [10 10]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [11 11]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [12 12]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [13 13]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [14 14]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [15 15]
["bd340179-5905-4233-ad5d-44b47b228177" 6702], [1 1]
["bd340179-5905-4233-ad5d-44b47b228177" 6702]}, :executor->start-time-secs
{[2 2] 1428934165, [3 3] 1428934165, [4 4] 1428934165, [5 5] 1428934165, [6
6] 1428934165, [7 7] 1428934165, [8 8] 1428934165, [9 9] 1428934165, [10
10] 1428934165, [11 11] 1428934165, [12 12] 1428934165, [13 13] 1428934165,
[14 14] 1428934165, [15 15] 1428934165, [1 1] 1428934165}}
If anybody can help that'd be greatly appreciated!
Thanks,
Matt
--
Check out my blog: dogdogfish.com
