Re: [heka] Hinsight in pull mode

[Eric LEMOINE]

On Wed, Sep 21, 2016 at 10:48 AM, Mathieu Parent  wrote:
> Hello (again),
>
> Hindsight is using a push-model (i.e Nagios passive checks). This is
> great, but I want to plug it with Prometheus which uses pull-model
> [1].
>
> I see several ways to handle this:
> - use the prometheus push-gateway [2]. This has several drawbacks listed below
> - introduce pull model in hindsight
> - add a new daemon, based on lua_sandbox too, but using pull model
>
> The drawbacks of prometheus push-gateway are:
> - Unnecessary polling of data (data is grabbed even if not pulled by 
> prometheus
> - time lag, between data grabbing and data pulling
> - To sum up : to reduce time lag, you increase polling rate, when us
> decrease polling, you increase time lag.
>
> The push model may work like this:


You mean "pull" here I guess.


> - Adding pull_message_matcher config to inputs (defaults to FALSE)
> - Adding process_pull_message() function to inputs, returning a table
> of messages (or should it be inject_pull_message() + return 0?)
> - Adding request_pull_message() function to outputs, which maps to
> matching process_pull_message() and concatenates the results in a
> table. This function is blocking.
>
> Opinions?

If there was an "http listen" extension in lua_sandbox_extensions then
I think it would be easy to write an output plugin compatible with
Prometheus, without specific support for pull in Hindsight. The
problem is that there's "http listen" extension in
lua_sandbox_extensions right now, see [*] for a bit more information
on this.

[*] 
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] New Heka release?

[Eric LEMOINE]

On Thu, Sep 15, 2016 at 6:47 PM, Waqar Khan  wrote:
> Hi Eric,
>
> Heka is now deprecated, I do not think they will make any future releases.
> See the mail archive for more details.

I know. But the main branch (the "dev" branch) has evolved since the
0.10 release. So I just thought that creating 0.11 based on the
current "dev" branch would make sense.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] New Heka release?

[Eric LEMOINE]

Hi

Heka 0.10.0 was released in December 2015. Since then a number of
features and fixes have been added to the "dev" branch. For that
reason we currently use the "dev" branch, which we build ourselves.
Would releasing 0.11 make sense? I think that would make a lot people
happy :)

Thanks!
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Idle packs and wedged Heka

[Eric LEMOINE]

Le 14 avr. 2016 19:05, "Rob Miller"  a écrit :
>
> Sorry to be slow to respond, I've been away.
>
> Unfortunately I can't say for sure what's going on here. The idle packs
being associated w/ your influx accumulator filter appears to imply that
the packs are being injected into that filter but never recycled. It's
possible, however, that there's a bug in the diagnostics and that's not
accurate. Do you happen to be using buffering at all for any of your filter
or output plugins?

Buffering is only configured for output plugins.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Idle packs and wedged Heka

[Eric LEMOINE]

On Wed, Apr 6, 2016 at 8:48 PM, Rob Miller  wrote:
> I'm not sure exactly what is going on here, but I can provide you w/ some
> info that might help you debug further:
>
> * Any packs that are in a decoder's input channel can only have come from
> the input pool, i.e. inputRecycleChan. I see ~40 packs tied up there.
>
> * Any packs that are in a match channel or input channel for a filter or
> output plugin could have come from either the input pool or the inject pool,
> although they can only have come from the inject pool if they were injected
> by some other filter plugin first. If a plugin's matcher only matches
> messages that you know came directly from an input plugin, then they have to
> have come from the input pool. Most of these are empty, but there are 60
> packs sitting in the queues for http_metrics_filter.
>
> * Heka doesn't freeze the world while generating the report data, so it's
> possible that the data you're seeing doesn't represent a single point in
> time, which can cause the math to be weird. However, if Heka is truly wedged
> and no traffic is flowing, this is moot, since the state probably isn't
> changing at all during the entire reporting process.
>
> * It's possible for a filter or output to hold an arbitrary number of packs
> in its internal memory. These won't show up in the reporting at all, but
> neither will they be recycled. Without seeing the code that's running in
> your filters I can't say whether or not that's happening.
>
> I have to run into a meeting right now, hopefully this will help at least a
> bit, or at least help you come up with more specific questions.


Ok, I have done new tests with fewer plugins. I can reproduce the
"idle packs" issue but the diagnostic makes more sense, matching the
explanation you provided in the old thread (with Bruno Binet).

See 
for the new diagnostic trace.

This time, the downstream filter (influxdb_accumulator_filter) is the
one that is deadlocked. The recycle inject channel is empty, and this
filter has 60 packs in his queues (30 in the match queue and 30 in the
input queue). There are also 30 packs in the router's input queue,
which may be inject packs. So that makes for a maximum of 90 packs.
I'm still missing 10 packs! Any idea where they could be? Note that
the 6 packs in the http_metrics_filter plugin are input packs coming
from keystone_7_0_logstreamer input, they cannot be inject packs
because of the matcher used for that filter.

So I get a diagnostic that makes more sense, but I'm still missing 10
inject packs!

Sorry for insisting. Understanting this would help me troubleshoot
problems in the future.

PS: I am indeed not able to reproduce the problem with poolsize 200,
but I'd like to make sure the problem will not come back in the
future.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Idle packs and wedged Heka

[Eric LEMOINE]

On Wed, Apr 6, 2016 at 8:48 PM, Rob Miller  wrote:
> I'm not sure exactly what is going on here, but I can provide you w/ some
> info that might help you debug further:
>
> * Any packs that are in a decoder's input channel can only have come from
> the input pool, i.e. inputRecycleChan. I see ~40 packs tied up there.
>
> * Any packs that are in a match channel or input channel for a filter or
> output plugin could have come from either the input pool or the inject pool,
> although they can only have come from the inject pool if they were injected
> by some other filter plugin first. If a plugin's matcher only matches
> messages that you know came directly from an input plugin, then they have to
> have come from the input pool. Most of these are empty, but there are 60
> packs sitting in the queues for http_metrics_filter.
>
> * Heka doesn't freeze the world while generating the report data, so it's
> possible that the data you're seeing doesn't represent a single point in
> time, which can cause the math to be weird. However, if Heka is truly wedged
> and no traffic is flowing, this is moot, since the state probably isn't
> changing at all during the entire reporting process.
>
> * It's possible for a filter or output to hold an arbitrary number of packs
> in its internal memory. These won't show up in the reporting at all, but
> neither will they be recycled. Without seeing the code that's running in
> your filters I can't say whether or not that's happening.
>
> I have to run into a meeting right now, hopefully this will help at least a
> bit, or at least help you come up with more specific questions.


Thanks Rob.

I'll start with one very specific question.

In the diagnostic I have this for the idle inject packs:

2016/03/31 11:56:20 Diagnostics: 30 packs have been idle more than 120 seconds.
2016/03/31 11:56:20 Diagnostics: (inject) Plugin names and quantities
found on idle packs:
2016/03/31 11:56:20 Diagnostics:   influxdb_accumulator_filter: 30

(and no other plugins with idle inject packs!)

In the same diagnostic I have this:

influxdb_accumulator_filter:
InChanCapacity: 30
InChanLength: 0
MatchChanCapacity: 30
MatchChanLength: 0

My question: where can the 30 idle inject packs attributed to
influxdb_accumulator_filter be?

InChanLength is 0, so they're not in the influxdb_accumulator_filter
plugin's input channel. influxdb_accumulator_filter is a sandbox, and
looking at sandbox_filter.go code [*], I do not see how the filter
could be holding these packs.

At some point I thought that these packs could sit in another
filter/output's input channel, or be hold by another filter/channel.
But in that case the diagnostic should attribute the idle inject packs
to that other plugin as well. This is not case, the 30 idle packs are
only attributed to influxdb_accumulator_filter.

[*] 


I hope that my question is clear.

Thanks again!

PS: I hope to come up with more specific information about this problem.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] heka wedged

[Eric LEMOINE]

> Buffering does help, yes. When buffering is in use, the shared pack from
the global pool is recycled immediately after the message is written to the
disk buffer. Each plugin that uses disk buffering uses its own packs
internally that don't intermingle with the global packs at all.

That makes sense. Thanks for confirming.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Idle packs and wedged Heka

[Eric LEMOINE]

Hi

We're running into a situation where Heka reports about "idle packs"
and is "wedged". See the Heka diagnostics in
.

So the inject and input recycle channels are empty – no free packs.
And the http_metrics_filter sandbox is blocked in an inject_message()
call waiting for a free inject pack.

We read other threads [*] discussing similar issues, but our case may
be a bit different. The http_metrics_fliter gets messages from
logstreamer inputs, and it injects messages that will be consumed by
the influxdb_accumulator_filter. I think our case is different because
the upstream filter (http_metrics_fliter), as opposed to the
downstream filter (influxdb_accumulator_filter), is blocked. How can
this be possible?

And I really wonder where the inject packs are! The inject recycle
channel is empty, so who holds the inject packs? There are 30 idle
inject packs attributed to the influxdb_accumulator_filter, although
that filter's match channel length is 0 (so is its input channel
length). So I do not understand how this sandbox filter can have idle
packs! And where are the remaining 70 inject packs? Any idea?

In case this is relevant: we use buffering for the output plugins
(with full_action drop), and no buffering for the filters.

This is blocking us big time. Any insight is welcome. Thanks!

[*] 
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Heka consumes CPU doing nothing

[Eric LEMOINE]

Hi

This is to report what may be an issue in Heka, where Heka consumes
CPU while there's nothing to do.

My test-case is the following: LogStreamer with an rsyslog.lua decoder
reading logs from a number of static log files. After having processed
all the logs (from all the files) Heka still consumes CPU.

On my 8-core laptop, with 500 static log file, Heka consumes 400% CPU
while processing the logs, which is expected given that I set max_proc
to 4 in heka.conf. BUT it still consumes 25% of CPU after having
processed all the logs, and this for ever. The more log files to be
processed the more consumed by Heka. So this is related to the number
of decoders Heka creates.

Is this expected? Is this a bug in Heka.

My test-case is here , with
instructions on how to reproduce the problem.

Thanks!

PS: I'm happy to create a GitHub issue if you think it's appropriate.
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Heka consumes CPU doing nothing

[Eric LEMOINE]

On Tue, Jan 12, 2016 at 9:30 AM, Eric LEMOINE <elemo...@mirantis.com> wrote:
> Hi
>
> This is to report what may be an issue in Heka, where Heka consumes
> CPU while there's nothing to do.
>
> My test-case is the following: LogStreamer with an rsyslog.lua decoder
> reading logs from a number of static log files. After having processed
> all the logs (from all the files) Heka still consumes CPU.
>
> On my 8-core laptop, with 500 static log file, Heka consumes 400% CPU
> while processing the logs, which is expected given that I set max_proc
> to 4 in heka.conf. BUT it still consumes 25% of CPU after having
> processed all the logs, and this for ever. The more log files to be
> processed the more consumed by Heka. So this is related to the number
> of decoders Heka creates.
>
> Is this expected? Is this a bug in Heka.
>
> My test-case is here <https://github.com/elemoine/heka-cpu>, with
> instructions on how to reproduce the problem.
>
> Thanks!
>
> PS: I'm happy to create a GitHub issue if you think it's appropriate.


We figured the issue is related to the "check for more data interval"
which is hard-coded to 250ms in the code [*]. So that value might not
be appropriate when there are many log files to watch. 500 log files
may not correspond to a realistic scenario. With 100 log files, which
does sounds realistic to me, Heka still consumes about 8% of CPU on my
8-core laptop. Maybe 250ms is a bit too aggressive as a default value.

Thoughts?

[*] 
<https://github.com/mozilla-services/heka/blob/dev/plugins/logstreamer/logstreamer_input.go#L297>
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Logstreamer fails if log directory does not exist

[Eric LEMOINE]

Hi

First of all thank you for Heka!

The logstreamer plugin fails to initialize with "lstat /var/log/foo:
no such file or directory" if the log directory doesn't exist.

In my case I would like the logstreamer plugin to do nothing if the
log directory doesn't exist. And I'd like it to "wake up" and start
processing logs when it detects that the log directory now exists and
includes log files. It's like having Heka adapt to its environment.

Untested but this is the kind of patch I am thinking of:
.

What do you think? Would that behaviour make sense to you?

Thanks!
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka