Thanks, this is great. If I do that super-quickly, I get a segfault in libzmq.so, specifically in zmq::socket_base_t::process_term. (If I'm slow, I get "No such file or directory" errors because the worker's trying to touch files in subdirs of /mnt/storm which no longer exist.)
This is a pretty horrible result, but it's progress. Best regards, Alex On Wed, Jan 15, 2014 at 3:21 PM, Jon Logan <[email protected]> wrote: > These issues are annoying to debug. My best solution has been to look in > the supervisor log, it'll print out a long command that it used to launch > (java -jar .....). Log into that machine, sudo as the storm user, and paste > that command and try to execute it yourself. It tends to be a problem > launching the worker, and this usually will display the error message. It's > a hack, but it usually works. > > > On Wed, Jan 15, 2014 at 3:18 PM, Alex Coventry <[email protected]> wrote: > >> Thanks, Jon. Yes, I have topology.debug set to true in the config map >> passed to StormSubmitter/SumbitTopology, and I've changed all the "INFO" >> settings to "DEBUG" in logback/cluster.xml. >> >> I think you might be onto something with checking whether the JVMs are >> running. I noticed in the supervisor log: "Error when trying to kill 8060. >> Process is probably already dead." Nathan Marz says in this >> thread<https://groups.google.com/forum/#!topic/storm-user/UxbMLIfEV1k> that >> it suggests native dependencies are not installed correctly. I'm looking >> for ways to test whether this is the case, now. Any suggestions are >> welcome. Is there a way to kick off a worker thread in the storm repl, to >> get some more info about how it's failing? >> >> Best regards, >> Alex >> >> On Wed, Jan 15, 2014 at 3:04 PM, Jon Logan <[email protected]> wrote: >> >>> I haven't looked at your logs, but just to clarify, the message you >>> describe is probably a debug message. Debug messages say things like the >>> emitting and receiving of tuples. I'm not sure if it's enabled by default >>> in local mode, but if you want those messages in normal mode, you need to >>> set topology.debug: true in the storm.yaml configuration file. >>> >>> >>> You can also look to see if the worker JVMs are actually running on >>> other machines. They should be in separate processes. >>> >>> >>> On Wed, Jan 15, 2014 at 1:57 PM, Alex Coventry <[email protected]>wrote: >>> >>>> Thanks for the confirmation. It appears that my topology is not >>>> running. When I run the word-count topology in local mode, I see messages >>>> like "Emitting: 3 default ["dog"]". I assume I should be seeing messages >>>> like that in the logs when I run the topology in distributed mode, but I'm >>>> not. I'm also not seeing any exceptions. Log messages related to >>>> communication between nimbus, workers and supervisor appear for about three >>>> minutes, then everything related to the topology seems to shut down, >>>> including the UI's report that it's running. >>>> >>>> I'm pretty stumped by this, and I'd be grateful for any help. I took a >>>> copy of the logs for nimbus, the supervisor, and the workers, for the >>>> lifecycle of the topology I just described: >>>> >>>> https://dl.dropboxusercontent.com/u/6414090/logs.tgz >>>> >>>> I'd really appreciate it if someone with more experience with storm >>>> could take a look at them and tell me where I'm going wrong. >>>> >>>> I'm using storm-0.9.0, storm-starter from git master. ZK and nimbus >>>> are running on the same machine, the supervisor/workers are running on >>>> another machine. Judging from the logs, they are all able to see each >>>> other. >>>> >>>> Best regards, >>>> Alex >>>> >>>> >>>> >>>> On Wed, Jan 15, 2014 at 7:26 AM, Jason Trost <[email protected]>wrote: >>>> >>>>> This should show up in one or more worker log in >>>>> $STORM_HOME/logs/worker-*.log. >>>>> >>>>> >>>>> On Tue, Jan 14, 2014 at 10:25 PM, Alex Coventry <[email protected]>wrote: >>>>> >>>>>> If I explicitly throw an exception in the storm-starter clojure >>>>>> example, as shown in the diff below, it shows up nicely when I run in >>>>>> local >>>>>> mode with >>>>>> >>>>>> coventry@samjoko:~/storm-starter$ lein run -m >>>>>> storm.starter.clj.word-count >>>>>> >>>>>> However, when I run it on a storm cluster with a command like >>>>>> >>>>>> coventry@samjoko:~/storm-starter$ >>>>>> ~/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar >>>>>> storm.starter.clj.word_count count >>>>>> >>>>>> I am not sure where these errors are being reported. Are they logged >>>>>> anywhere, and if not, can I get them to be? >>>>>> >>>>>> Best regards, >>>>>> Alex >>>>>> >>>>>> index ce2725d..c82fd0f 100644 >>>>>> --- a/src/clj/storm/starter/clj/word_count.clj >>>>>> +++ b/src/clj/storm/starter/clj/word_count.clj >>>>>> @@ -11,6 +11,7 @@ >>>>>> "an apple a day keeps the doctor away"]] >>>>>> (spout >>>>>> (nextTuple [] >>>>>> + (throw (Exception. "Where does this show up?")) >>>>>> (Thread/sleep 100) >>>>>> (emit-spout! collector [(rand-nth sentences)]) >>>>>> ) >>>>>> >>>>>> >>>>> >>>> >>> >> >
