I came upon an issue with ScribeSource, though it's theoretically
applicable to any EventDrivenSource whose event generating thread(s)
die. Simple put, sending a bad packet to the thrift(scribe protocol)
port will result in it trying to allocate space for some arbitrarily
large packet resulting in an OOMException which kills the
thread(incidentally I thought this would be an issue in avro too, but it
throws an exception before making excessive allocation requests).
As far as flume is concerned, the component is still alive. stop() was
never called, so even monitoring the component state using jmx will not
notice anything wrong. This situation occurs from user error, but there
is potential for other errors leaving a zombie component. I think it
would be more user friendly to be able to recover from such errors.
I'm thinking of adding a StatusPollable interface that
EventDrivenSources can optionally implement(because we can't change the
interface without a version change). If implemented, the
EventDrivenSourceRunner would schedule a regular poll to check the
state. Upon failure it could either call stop() to signal it broke. With
autoRestartPolicy, the source would then get restarted by its supervisor.
Would appreciate any opinions before I put together a patch/post an issue.
