I am using Storm w/ my bolts and spouts written in Python.
When I am running in a test environment locally I have a problem where if
the spout dies, the bolts consume 100% CPU and gradually increase their
memory. Here are the details:
My spout is reading from mongo. Sometimes it loses its cursor (due to a
network hiccup or something) and it raises an exception and bails out. I
can also reliably reproduce this problem by using "kill -9 <spout PID>".
Using strace and then a debugger, I figured out that the bolts are stuck in
this tight loop in the readMsg function inside storm.py (which ships with
storm)
while True:
line = sys.stdin.readline()[:-1]
if line == "end":
break
msg = msg + line + "\n"
The readline() call is a blocking call, but the bolt keeps getting blank
lines as input.
Note: the memory problem is because newlines keep getting appended. Since
the input is used as JSON, it would probably be safe to just remove the
addition of the newline (I'm not 100% positive because that might not work
if there are newlines in an embedded string within the JSON). But that
still doesn't fix the core issue.
I think the problem is that if the spout does not come down cleanly,
something in the Java keeps sending input to the bolts. I'm about to dig
into the Java code but I don't know any Java so I figured it was worth a
quick message to the Storm list to see if this is a known problem. Or even
if anyone has a pointer for where to look in the Java. I haven't
previously looked at the Storm source - it's just been a black box to this
point for me.
Thanks,
scot
