Thank you for reply eventually, :-). Its very helpful for me to understand this case. In the past month, I also find the two-goroutine patter is necessary when inject new pack in a filter, ;-) On Sep 9, 2014 2:14 AM, "Rob Miller" <[email protected]> wrote:
> The things you find when cleaning out your inbox... it's been over a > month, so this response may be moot, but I'm replying anyway for > posterity's sake. For those following along at home, the questions below > are referring to https://github.com/nemothekid/ > heka-greedy-filter-deadlock/. > > On 08/04/2014 09:07 PM, Tom Sawyer wrote: > >> Hi, Rob and Nimi. >> I have watched this thread this days. I have several questions about the >> deadlock situation of GreedyFilter. >> >> 1. Which channel is fully consumed and get frozen when deadlock occurs? >> I thought it was router.InChan and injectRecycleChan because Inject >> called by Input Plugin will deliver the pack into forunner.InChan while >> at the same time Filter Inject pack into forunner.InChan too. >> forunner.InChan maybe fully occupied because GreedyFilter has no >> chance take pack out of forunner.InChan when it is flushing. But >> Inject() of a filter will run a goroutine to do acctually inject >> operation and then GreedyFilter.flush() will return and get pack from >> forunner.InChan eventually. So I don't know where the deadlock happens. >> > Pretty sure it's the router that's getting backed up here. We have a > situation where messages that are intended for GreedyFilter are coming in > very quickly. This is fine, as long as GreedyFilter is processing them > quickly. But GreedyFilter's Flush method initializes and injects 10,000 > messages, which takes a very long time relative to most operations. While > this is happening, the incoming messages slowly pile up until the router is > backed up and can't accept any more messages. But GreedyFilter itself is > *also* trying to inject new messages back into the router. Since the router > is blocked, GreedyFilter gets stuck in the Flush method, it never returns > to the Run method to start clearing away the backlog, and we have a > deadlock. > >> 2. When deadlock happens why can "everybody calls Recycle or Inject" >> and the pool size is still exhausted? >> > Because the issue isn't that the message pool is exhausted, but that the > router is blocked because GreedyFilter isn't processing messages, and > GreedyFilter isn't processing messages because the router is blocked. > > Any filter or output that is going to be receiving a high volume of > messages should be doing what it can to process those messages as quickly > as possible, with a minimum of pauses. With outputs, there's a bit more > tolerance for pauses; the router may back up momentarily, but the > backpressure will alleviate when the output starts processing the incoming > messages again. Filters, however, are usually reinjecting messages back to > the router, so if they cause a router backup then it might be unrecoverable. > > The solution in this case was to move the processing out of a filter and > into a decoder, which is where the intended functionality should have been > in the first place. Any time you have a single input message generating > multiple output messages it's probably something that a decoder should be > doing instead of a filter. Slow operations in a decoder only slow down the > input that is feeding the decoder; the rest of Heka will keep churning > along undisturbed. If the work really needs to be done in a filter, the > solution would be to have the filter spin up two goroutines so it can > continue to process incoming messages while concurrently performing the > flush operation (or whatever it is that's taking a long time). > > Hope this helps clarify! > > -r > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
