Hi, We've come across an issue stress testing one of our setups. We have a 3 step setup, one (tcp) input, (aggregate) filter, and (cassandra) output. The way it works is on the input is a stream of key-values, which are then aggregated on key, and periodically flushed to the cassandra output.
What we began to see under a stress test was in the periodic aggregate step, the aggregate filter will begin to flush its memory by requesting and injecting pipeline packs (up to ~13k pipelinepacks, multiple times larger than the 100 pool size). Surprisingly, eventually the pipeline pack pool is exhausted and heka will freeze. After digging around, it seems the issue is: 1.) A large amount of pipeline packs are sent to the aggregate filter. 2.) The aggregate filter begins its aggregate step, and stops accepting packs from its InChan. 3.) The aggregate filter also begins to request PipelinePacks and inject them into the stream. 4.) Because the aggregate filter is no longer accepting requests, the MessageRouter is stuck blocking for the aggregate filter to accept a message. 5.) Because the MessageRouter isn't routing messages, the injected packs aren't going to the Cassandra output, and aren't ultimately being freed 6.) Because the packs aren't being freed, the pool is eventually exhausted 7.) Heka freezes even though everybody calls Recycle or Inject. If this doesn't make sense, I can code up an example filter/input plugin that should cause the deadlock. I'm sure if this is a legitimate bug, or just something we shouldn't do. Currently we have sidestepped the issue by copying the data and doing the flush in a separate goroutine (freeing up the filter to continue to accept packs). Another solution for us would be to completely sidestep the router and just pass the pipeline pack to the output directly. The docs claim this is possible, but I'm not entirely clear on how to achieve this (how do you get the reference to the target plugin InChan?). All in all, it seems that heka can freeze if the message router tries to deliver a pipelinepack to a filter that is in a busy loop requesting & injecting pipeline packs. Thanks, Nimi Wariboko Jr. [email protected]
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
