I'm trying to implement a bolt that needs to be notified of tuple failures (it puts incoming tuples on a queue, and gets notified separately when to emit the tuples on the queue. If a tuple fails in another bolt, I need to take it off the queue). However, I've come across multiple issues in implementing this:
Solution 1: The spout puts the failed msgIds onto a queue, and emits the failed msgIds into a 'failed' stream that is read by the bolt. Problem: The msgId used by the spout does not leave the spout. It looks like it is mapped onto a Long before being emitted. There is a getMessageId method on Tuple, but this only has access to the Longs, not the original msgId. Furthermore, the mapped msgIds do not get returned to the spout, so it cannot emit those instead, as it has no access to them. Solution 2: Specify a task hook in the bolt that gets called when a tuple is failed. Problem: SpoutFailInfo does not have the mapped msgId of the failed tuple, and EmitInfo does not have the original or mapped msgId at all. It also looks like the hook is not run locally to the bolt. Solution 3: Emit the msgId in the tuple itself, essentially implementing my own mechanism for doing this. This is problematic. Is there a suggested or recommended way of a bolt being notified of a tuple failure? I would prefer not to implement my own solution if there is one provided by storm. SimonC
