Re: [heka] multiple file upload using HEKA

[Mathieu Parent]

2016-04-06 20:23 GMT+02:00 Rob Miller :
> If I understand what you're saying, you're using LogstreamerInput
> incorrectly. A single LogstreamerInput is meant to map to either a single
> log stream, or possibly multiple logstreams of identical structure, such as
> a set of access logs for a number of different domains.

(Correct me if I'm wrong).

Appart from the "different structure", your "differentiator" should
resolve differently for each stream. i.e heka keep only one opened
file descriptor per differentiator value. The docs should be updated,
but finding the good wordings is hard.

See: https://github.com/mozilla-services/heka/pull/1904

Regards

-- 
Mathieu Parent
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Idle packs and wedged Heka

[Rob Miller]

I'm not sure exactly what is going on here, but I can provide you w/ some info 
that might help you debug further:

* Any packs that are in a decoder's input channel can only have come from the 
input pool, i.e. inputRecycleChan. I see ~40 packs tied up there.

* Any packs that are in a match channel or input channel for a filter or output 
plugin could have come from either the input pool or the inject pool, although 
they can only have come from the inject pool if they were injected by some 
other filter plugin first. If a plugin's matcher only matches messages that you 
know came directly from an input plugin, then they have to have come from the 
input pool. Most of these are empty, but there are 60 packs sitting in the 
queues for http_metrics_filter.

* Heka doesn't freeze the world while generating the report data, so it's 
possible that the data you're seeing doesn't represent a single point in time, 
which can cause the math to be weird. However, if Heka is truly wedged and no 
traffic is flowing, this is moot, since the state probably isn't changing at 
all during the entire reporting process.

* It's possible for a filter or output to hold an arbitrary number of packs in 
its internal memory. These won't show up in the reporting at all, but neither 
will they be recycled. Without seeing the code that's running in your filters I 
can't say whether or not that's happening.

I have to run into a meeting right now, hopefully this will help at least a 
bit, or at least help you come up with more specific questions.

-r


On 04/04/2016 12:09 PM, Eric LEMOINE wrote:

Hi

We're running into a situation where Heka reports about "idle packs"
and is "wedged". See the Heka diagnostics in
.

So the inject and input recycle channels are empty – no free packs.
And the http_metrics_filter sandbox is blocked in an inject_message()
call waiting for a free inject pack.

We read other threads [*] discussing similar issues, but our case may
be a bit different. The http_metrics_fliter gets messages from
logstreamer inputs, and it injects messages that will be consumed by
the influxdb_accumulator_filter. I think our case is different because
the upstream filter (http_metrics_fliter), as opposed to the
downstream filter (influxdb_accumulator_filter), is blocked. How can
this be possible?

And I really wonder where the inject packs are! The inject recycle
channel is empty, so who holds the inject packs? There are 30 idle
inject packs attributed to the influxdb_accumulator_filter, although
that filter's match channel length is 0 (so is its input channel
length). So I do not understand how this sandbox filter can have idle
packs! And where are the remaining 70 inject packs? Any idea?

In case this is relevant: we use buffering for the output plugins
(with full_action drop), and no buffering for the filters.

This is blocking us big time. Any insight is welcome. Thanks!

[*] 
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka



___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] multiple file upload using HEKA

[Iñigo Ortiz de Urbina]

Merhaba Abhiman

If you are having trouble making heka work, I recommend you do some
"bottom-up" debugging to check every stage, from input to output. At
the very least:
  - is ´heka-logstreamer -config=/etc/heka/test.toml´ yielding the
expected output? If it does not show what you want, iterate the config
by adjusting the regex
  - what does the RstEncoder+LogOutput show? If it does not show what
you want, iterate the config by adjusting the msg matcher

Provided hekad is not wedged, the DashboardOutput also provides a nice
and easy to read overview [0].

I am also a bit confused by your use of the translation maps but I do
not have all the details so there may be a legit use for it.
I usually just expand the captured group in the differentiator setting
of Logstreamer.

Please check the Verifying Settings section of the docs [1] as well as
the RstEncoder+LogOutput combo [2].

HTH

[0] http://hekad.readthedocs.org/en/latest/config/outputs/dashboard.html
[1] http://hekad.readthedocs.org/en/latest/pluginconfig/logstreamer.html
[2] http://hekad.readthedocs.org/en/latest/config/encoders/rst.html



Este correo electrónico se ha enviado desde un
equipo libre de virus y protegido por Avast. https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail;
target="_blank" style="color: #4453ea;">www.avast.com   




On Wed, Apr 6, 2016 at 11:08 AM, Abhiman  wrote:
> I want to upload multiple files(which are non sequential) simultaneously
> using Heka. The files are conn.log, http.log, dhcp.log, dnp3.log, ftp.log,
> dns.log. I am trying with the following code
>
> [networklogs]
> Type = "LogstreamerInput"
> log_directory = "/opt/bro/logs/current"
> file_match= '(?P\d+)\.log'
> priority=["Year"]
> decoder="Json"
>
> [networklogs.translation.Year]
> conn = 1
> dhcp = 2
> dnp3 = 3
> dns = 4
> ftp = 5
> http = 6
>
> [Json]
> type = "SandboxDecoder"
> filename = "lua_decoders/json.lua"
>
> [Json.config]
> type = "raw.bro"
> debug = false
>
>
> [ESJsonEncoder]
> index = "test_2"
> type_name = "one"
>
>
> [ElasticSearchOutput]
> message_matcher = "TRUE"
> server = "http://localhost:9200;
> encoder = "ESJsonEncoder"
>
> but it's not working. When I am trying with only one file, I am succeeding
> but for multiple files, no luck.
>
> Regards,
>
> -
>
> Abhiman | Analyst
>
> M: +91 8004274160 | www.tcg-digital.com
>
>
> ___
> Heka mailing list
> Heka@mozilla.org
> https://mail.mozilla.org/listinfo/heka
>



-- 
"If you want to go fast, go alone. If you want to go far, go together."
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] multiple file upload using HEKA

[Abhiman]

I want to upload multiple files(which are non sequential) simultaneously using 
Heka. The files are conn.log, http.log, dhcp.log, dnp3.log, ftp.log, dns.log. I 
am trying with the following code

[networklogs]
Type = "LogstreamerInput"
log_directory = "/opt/bro/logs/current"
file_match= '(?P\d+)\.log'
priority=["Year"]
decoder="Json"

[networklogs.translation.Year]
conn = 1
dhcp = 2
dnp3 = 3
dns = 4
ftp = 5
http = 6

[Json]
type = "SandboxDecoder"
filename = "lua_decoders/json.lua"

[Json.config]
type = "raw.bro"
debug = false


[ESJsonEncoder]
index = "test_2"
type_name = "one"


[ElasticSearchOutput]
message_matcher = "TRUE"
server = "http://localhost:9200;
encoder = "ESJsonEncoder"


but it's not working. When I am trying with only one file, I am succeeding but 
for multiple files, no luck.

Regards,

-

Abhiman | Analyst

M: +91 8004274160 | www.tcg-digital.com
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka