Re: [heka] heka wedged

[Eric LEMOINE]

> Buffering does help, yes. When buffering is in use, the shared pack from
the global pool is recycled immediately after the message is written to the
disk buffer. Each plugin that uses disk buffering uses its own packs
internally that don't intermingle with the global packs at all.

That makes sense. Thanks for confirming.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Idle packs and wedged Heka

[Eric LEMOINE]

Hi

We're running into a situation where Heka reports about "idle packs"
and is "wedged". See the Heka diagnostics in
.

So the inject and input recycle channels are empty – no free packs.
And the http_metrics_filter sandbox is blocked in an inject_message()
call waiting for a free inject pack.

We read other threads [*] discussing similar issues, but our case may
be a bit different. The http_metrics_fliter gets messages from
logstreamer inputs, and it injects messages that will be consumed by
the influxdb_accumulator_filter. I think our case is different because
the upstream filter (http_metrics_fliter), as opposed to the
downstream filter (influxdb_accumulator_filter), is blocked. How can
this be possible?

And I really wonder where the inject packs are! The inject recycle
channel is empty, so who holds the inject packs? There are 30 idle
inject packs attributed to the influxdb_accumulator_filter, although
that filter's match channel length is 0 (so is its input channel
length). So I do not understand how this sandbox filter can have idle
packs! And where are the remaining 70 inject packs? Any idea?

In case this is relevant: we use buffering for the output plugins
(with full_action drop), and no buffering for the filters.

This is blocking us big time. Any insight is welcome. Thanks!

[*] 
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] heka wedged

[Rob Miller]

On 04/04/2016 05:32 AM, Eric LEMOINE wrote:

On Wed, Mar 30, 2016 at 9:01 PM, Rob Miller  wrote:
> Heka can get wedged when there's a deadlock in the pipeline, usually related
> to pack exhaustion. For example, a filter might block because it's waiting
> for an empty pack with which to inject a new message, but there are no packs
> available because they're all tied up in the input channels for the filter.
> Tweaking the channel and pool sizes is usually helpful here.


Wondering if using buffering on filters could be a solution… I guess
that depends on when packs are recycled when buffering is used.

Buffering does help, yes. When buffering is in use, the shared pack from the 
global pool is recycled immediately after the message is written to the disk 
buffer. Each plugin that uses disk buffering uses its own packs internally that 
don't intermingle with the global packs at all.

-r



___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka