Re: [heka] multiple file upload using HEKA

[Iñigo Ortiz de Urbina]

Merhaba Abhiman

If you are having trouble making heka work, I recommend you do some
"bottom-up" debugging to check every stage, from input to output. At
the very least:
  - is ´heka-logstreamer -config=/etc/heka/test.toml´ yielding the
expected output? If it does not show what you want, iterate the config
by adjusting the regex
  - what does the RstEncoder+LogOutput show? If it does not show what
you want, iterate the config by adjusting the msg matcher

Provided hekad is not wedged, the DashboardOutput also provides a nice
and easy to read overview [0].

I am also a bit confused by your use of the translation maps but I do
not have all the details so there may be a legit use for it.
I usually just expand the captured group in the differentiator setting
of Logstreamer.

Please check the Verifying Settings section of the docs [1] as well as
the RstEncoder+LogOutput combo [2].

HTH

[0] http://hekad.readthedocs.org/en/latest/config/outputs/dashboard.html
[1] http://hekad.readthedocs.org/en/latest/pluginconfig/logstreamer.html
[2] http://hekad.readthedocs.org/en/latest/config/encoders/rst.html



Este correo electrónico se ha enviado desde un
equipo libre de virus y protegido por Avast. https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail;
target="_blank" style="color: #4453ea;">www.avast.com   




On Wed, Apr 6, 2016 at 11:08 AM, Abhiman  wrote:
> I want to upload multiple files(which are non sequential) simultaneously
> using Heka. The files are conn.log, http.log, dhcp.log, dnp3.log, ftp.log,
> dns.log. I am trying with the following code
>
> [networklogs]
> Type = "LogstreamerInput"
> log_directory = "/opt/bro/logs/current"
> file_match= '(?P\d+)\.log'
> priority=["Year"]
> decoder="Json"
>
> [networklogs.translation.Year]
> conn = 1
> dhcp = 2
> dnp3 = 3
> dns = 4
> ftp = 5
> http = 6
>
> [Json]
> type = "SandboxDecoder"
> filename = "lua_decoders/json.lua"
>
> [Json.config]
> type = "raw.bro"
> debug = false
>
>
> [ESJsonEncoder]
> index = "test_2"
> type_name = "one"
>
>
> [ElasticSearchOutput]
> message_matcher = "TRUE"
> server = "http://localhost:9200;
> encoder = "ESJsonEncoder"
>
> but it's not working. When I am trying with only one file, I am succeeding
> but for multiple files, no luck.
>
> Regards,
>
> -
>
> Abhiman | Analyst
>
> M: +91 8004274160 | www.tcg-digital.com
>
>
> ___
> Heka mailing list
> Heka@mozilla.org
> https://mail.mozilla.org/listinfo/heka
>



-- 
"If you want to go fast, go alone. If you want to go far, go together."
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Any date for the 0.10.0 release?

[Iñigo Ortiz de Urbina]

Hi guys!

On Wed, Sep 16, 2015 at 10:49 PM, Simon Pasquier  wrote:
> Hi,
>
> On Wed, Sep 16, 2015 at 8:02 PM, Rob Miller  wrote:
>>
>> Unfortunately, as Inigo says in his other message, there are still some
>> issues w/ the buffering code that make me hesitant to release v0.10.0 final
>> just yet. There have been a lot of fixes since the last release, though, so
>> it's probably about time for at least another 0.10.0 beta release so more
>> folks can test the newer code w/o having to build from source.

I finally got around to deploy the latest build of versions/0.10.
I've re-enabled output buffering for both TCP and ES output plugins.

Let's see how it behaves. I'll keep you posted.

>> In further bad news, you've all probably noticed that releases are slowing
>> down a bit, and that bug triage / milestone maintenance in the issue tracker
>> is out of date. My time to work on the Heka core is limited by the fact that
>> I'm busy actually doing stuff *with* Heka (and Hindsight) here at Mozilla,
>> and there are only so many hours in a day. I'm still working on responding
>> to bug reports and pull requests, and more releases will be coming, but the
>> activity will probably be a bit bursty as I squeeze in hours when I have a
>> break from my other responsibilities.
>
>
> I'll try to look more closely at the issue tracker, at least to help
> triaging bugs that are duplicate, already fixed or invalid.
> Knowing that Hindsight is getting attention isn't a bad news IMO...

I for one will try to improve the docs when possible.

-- 
"If you want to go fast, go alone. If you want to go far, go together."
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Experiences with buffered outputs in 0.10.0b1

[Iñigo Ortiz de Urbina]

Hi all

On Fri, Sep 11, 2015 at 5:58 PM, Rob Miller  wrote:
> Thanks for the feedback, very glad to have folks like you finding and
> reporting the problems.

This is the least I can do! :)

> The problem with queue files failing to roll has already been resolved,
> running from the versions/0.10 branch should resolve that.
>
> The other issues may or may not still be happening, though. It would be
> great if you could build and report back re: which issues are still
> happening.

Let's do that. I'll build versions/0.10 and push the update and report back.

> The "shutdown" issue will still be happening. That one definitely deserves
> an issue to be opened.

And I can open that issue ;-)
Let me run some tests and collect more information.

-- 
"If you want to go fast, go alone. If you want to go far, go together."
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Errors porting TCPInput->AMQPOutput->AMQPInput configs from 0.8.3 to 0.10.0.b1

[Iñigo Ortiz de Urbina]

Hey Rob!

Many thanks for the prompt reply :-)

On Mon, Aug 31, 2015 at 7:41 PM, Rob Miller  wrote:
> AMQPOutput sets `use_framing = true` by default, so that means on the input
> side you'd want to use `splitter = "HekaFramingSplitter"`. Try adding that
> to the AMQPInput configuration and you'll hopefully start seeing data.
>
> Alternately, if you know you'll always only have one Heka message per AMQP
> message, you could set `use_framing = false` for the AMQPOutput, and use a
> NullSplitter with `use_message_bytes = true` on the AMQPInput side.
>
> Hope this helps!

Does help, thanks!

I did try to set the AMQPInput splitter to NullSplitter with
'use_message_bytes = true' (as AMQP didn't seem like a stream oriented
channel to me and I was expecting one heka msg per AMQP msg), but then
I missed the 'use_framing = false' setting in AMQPOutput. No wonder
why it wouldn't work ;-)

At any rate, it works now by setting the AMQPInput splitter to
HekaFramingSplitter.

Muchas gracias and sorry for the noise!

-- 
"If you want to go fast, go alone. If you want to go far, go together."
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Errors porting TCPInput->AMQPOutput->AMQPInput configs from 0.8.3 to 0.10.0.b1

[Iñigo Ortiz de Urbina]

Hi all

I am in the process of re-designing our logging pipeline using centos/heka.

Up until now, servers would use Logstreamers with custom decoders to
tail logs, assemble heka messages and AMQPOutput directly to two
load-balanced RabbitMQ servers.
Other heka instances would fetch the messages using AMQPInput and post
them to ES using ESOutput.

This has been running for a few months, processing millions of
messages perfectly fine. Thanks, guys! :-)

Now, I want to extend the pipeline for both internal and external systems.

In order to achieve this, I've updated the pipeline so that:
  - "heka-producers" have many logstreamers and one single TCPOutput
to stream *raw* messages to
  - "heka-aggregators" have one single TCPInput and many AMQPOutputs
to queue *raw* messages
  - "heka-consumers" have many AMQPInputs with MultiDecoders to
massage the data and one/many ESOutputs

The idea is that all servers send raw logs in the Payload field of a
heka message sent over TCP (leveraging buffering, output restart
behaviour and TLS), heka-aggregators decouple producers and consumers
by routing those logs to rabbitmq, and finally heka-consumers leverage
MultiDecoders to apply arbitrary lua decoders to the deserialised heka
messages (preserving the original Timestamp, Hostname etc set at the
source).

The following heka configs [1][2] used to work with 0.8.3.
I have put together a small testing environment (same configs, mostly)
and installed 0.10.0.b1.

When the messages start flowing through the testing environment, I can
confirm they are properly read and streamed to the heka-aggregators
(as seen by LogOutput+RstEncoder), but in the heka-consumer side I
get:

2015/08/31 14:33:40 Decoder
'system-rsyslog-mq1-input-system-rsyslog-multidecoder' error:
Subdecoder 'system-rsyslog-decoder' decode error: Failed parsing:
payload:
2015/08/31 14:33:40
:Timestamp: 1970-01-01 00:00:00 + UTC
:Type:
:Hostname:
:Pid: 0
:Uuid:
:Logger:
:Payload:
:EnvVersion:
:Severity: 7

The system-rsyslog-decoder fails at decoding the message as the
payload that is presented to it is empty. The timestamps are also set
to epoch. This suggests the problem lies in the deserialisation of the
heka message posted to RabbitMQ.

I suspect I have failed to align my configs with the splitter changes
introduced around 0.9. I have experimented customising the splitter
settings too - without any luck.

The AMQPOutput and AMQPInput have also seen some changes, but I
believe this is more of a PEBKAC than a bug.

Any help would be greatly appreciated!

[1] http://pastebin.com/LyDxrJtw
[2] http://pastebin.com/UuDR8gVG

-- 
"If you want to go fast, go alone. If you want to go far, go together."
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka