Hi all,
After supervisor was killed, it failed to be restarted because of missing
file stormconf.ser. The log shows supervisor correctly "Removing code" for
the topology, but then it tried to "Launching worker with assignment" after
/storm-local/supervisor/stormdist/ is already cleaned up, thus, it got
exception about file does not exist. Is it a known bug? Could you please
provide some insights how this happened? After deleting /storm-local
directory, supervisor started successfully.
2014-01-28 16:51:02 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-01-28 16:51:02 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=xxx:2181/storm sessionTimeout=20000
watcher=com.netflix.curator.ConnectionState@252a78ee
2014-01-28 16:51:02 o.a.z.ClientCnxn [INFO] Opening socket connection to
server xxx:2181
2014-01-28 16:51:02 o.a.z.ClientCnxn [INFO] Socket connection established
to xxx:2181, initiating session
2014-01-28 16:51:02 o.a.z.ClientCnxn [INFO] Session establishment complete
on server xxx:2181, sessionid = 0x143b20bc2587b86, negotiated timeout =
20000
2014-01-28 16:51:02 b.s.d.supervisor [INFO] Starting supervisor with id
0aa5860a-248b-4814-b201-7b2f40ce701f at host xxx
2014-01-28 16:51:03 b.s.d.supervisor [INFO] Removing code for storm id
performance_topology-1-1390953444
2014-01-28 16:51:03 b.s.d.supervisor [INFO] Shutting down and clearing
state for id 03ccf938-7276-44f7-ab0e-f2cdfb76d394. Current supervisor time:
1390956663. State: :timed-out, Heartbeat:
#backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1390956441,
:storm-id "performance_topology-1-1390953444", :executors #
{[2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [-1 -1] [1 1]}
, :port 6797}
2014-01-28 16:51:03 b.s.d.supervisor [INFO] Shutting down
0aa5860a-248b-4814-b201-7b2f40ce701f:03ccf938-7276-44f7-ab0e-f2cdfb76d394
kill 5817: No such process
2014-01-28 16:51:03 b.s.util [INFO] Error when trying to kill 5817. Process
is probably already dead.
2014-01-28 16:51:03 b.s.d.supervisor [INFO] Shut down
0aa5860a-248b-4814-b201-7b2f40ce701f:03ccf938-7276-44f7-ab0e-f2cdfb76d394
2014-01-28 16:51:03 b.s.d.supervisor [INFO] Launching worker with
assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"performance_topology-1-1390953444", :executors ([6 6] [5 5] [7 7] [8 8] [3
3] [4 4] [2 2] [1 1])} for this supervisor
0aa5860a-248b-4814-b201-7b2f40ce701f on port 6797 with id
7a34f6bf-5494-4ee7-ba54-bbdfbbcbb861
2014-01-28 16:51:03 b.s.event [ERROR] Error when processing event
java.io.FileNotFoundException: File
'storm-local/supervisor/stormdist/performance_topology-1-1390953444/stormconf.ser'
does not exist
Thanks,
Thuy
