Re: [heka] question on output buffering

[Mathieu Parent]

2015-10-31 18:58 GMT+01:00 Timur Batyrshin :
> Hi,
>
> I’m using TCPOutput to relay data to another Heka instance and I quite often
> see that a on-disk
> buffer grows too much.

Rob Miller has proposed a patch at:
https://github.com/mozilla-services/heka/issues/1738#issuecomment-150404452

I've not had time to test it yet. Can you test it?

> Today’s case:
>
> /var/cache/hekad/output_queue/relay_output# ls -l
> total 9784788
> -rw-r--r-- 1 root root 10019573004 Oct 31 17:37 30.log
> -rw-r--r-- 1 root root  14 Oct 31 17:37 checkpoint.txt
> root@r-jp-cms:/var/cache/hekad/output_queue/relay_output# cat checkpoint.txt
> 30 10019808619
> (the difference in numbers could have been caused by me running cat command
> after some short delay)
>
> At the same time I see the metrics emitted from this host by this output as
> I actually get alerted on them which helped me to find this.
> After I restart Heka the file shrinks and starts to grow from the very low
> size.
>
>
> Here is a section from my config file for relay_output:
>
> [relay_output]
> type = "TcpOutput"
> address = "my.secret.domain:9123"
> message_matcher = "TRUE"
>
> I have no section for buffering for this plugin (nor for any other plugin)
> and have no other TcpOutputs.
>
> The docs at http://hekad.readthedocs.org/en/v0.10.0b1/buffering.html say
> that the file should grow
> no larger that 128Mb by default but in the above case it is already 10Gb and
> growing.

This may be a different problem. I had it too, but couldn't reproduce
it currently.

> Is it the default different to that specified in docs or do I miss something
> else in configuration?
>
>
> I’m running Heka 0.10.0b from GH releases page on Ubuntu 12.04
>
>
> Thanks,
> Timur
>
> ___
> Heka mailing list
> Heka@mozilla.org
> https://mail.mozilla.org/listinfo/heka
>



-- 
Mathieu
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Grok based parsing

[Mathieu Parent]

2015-09-24 14:21 GMT+02:00 Andre <andre-li...@fucs.org>:
[...]
> However the real advantage of grok is its user base and is reasonably
> broad support by tools (.e.g. LogStash, FluentD, Graylog2, and
> others)...
>
> This mean you gain an easy source of pre-defined parsing rules without
> having to rewrite them as a sandbox LPEG, e.g:
>
> https://github.com/whyscream/postfix-grok-patterns

This one has already been ported (by me), see :
https://gist.github.com/mozilla-services/heka/pull/1649

> https://github.com/elastic/logstash/tree/v1.4.2/patterns
> http://www.gregmefford.com/blog/2014/09/24/analyzing-cisco-asa-firewall-logs-with-logstash/

This is now at <https://github.com/logstash-plugins/logstash-patterns-core>.
Those are probably easy to port, I have a local lpeg decoder for
vendor syslog, and will propose a PR with it once it is enough tested
(it doesn't parse Cisco ASA yet, but it parses Cisco IOS and Nexus and
various Netapp). See it attached for now.

> http://seclists.org/snort/2013/q4/174

Very easy to do also. PR welcomes!


Regards

-- 
Mathieu
-- This Source Code Form is subject to the terms of the Mozilla Public
-- License, v. 2.0. If a copy of the MPL was not distributed with this
-- file, You can obtain one at http://mozilla.org/MPL/2.0/.

-- Copyright 2015 Mathieu Parent <math.par...@gmail.com>

--[[
Parses the syslog logs from vendor not respecting Syslog RFCs.

Config:

- hostname_keep (boolean, defaults to false)
Always preserve the original 'Hostname' field set by Logstreamer's 'hostname' configuration setting.

- cisco_ios (boolean, defaults to false)
Try to parse as a Cisco IOS Payload.

- cisco_nexus (boolean, defaults to false)
Try to parse as a Nexus Payload.

- cisco_wlc (boolean, defaults to false)
Try to parse as a Wireless LAN Controller Payload.

- netapp_cdot (boolean, defaults to false)
Try to parse as a NetApp Clustered Data ONTAP Payload.

- netapp_balance (boolean, defaults to false)
Try to parse as a NetApp OnCommand Balance Payload.

*Example Heka Configuration*
.. code-block:: ini

[FlexibleSyslogDecoder]
type = "SandboxDecoder"
filename = "lua_decoders/flexible_syslog.lua"

[FlexibleSyslogDecoder.config]
cisco_ios = true
cisco_nexus = true
cisco_wlc = true
netapp_cdot = true
netapp_balance = true

*Example Heka Message*

:Timestamp: 2014-01-10 07:04:56 -0800 PST
:Type: netapp_cdot
:Hostname: netapp1
:Pid: 0
:UUID: 8e414f01-9d7f-4a48-a5e1-ae92e5954df5
:Logger: SyslogInput
:Payload: Volume test@vserver:42424242-4242-4242-4242-424242424242 destroyed.
:EnvVersion:
:Severity: 6
:Fields:
| name:"syslogfacility" value:0
| name:"cdot_message" value:"wafl.vvol.destroyed"
| name:"cdot_severity" value:"info"
--]]

local l = require 'lpeg'
l.locale(l)
local dt = require 'date_time'
local math = require 'math'
local os = require 'os'

local hostname_keep = read_config('hostname_keep')
local grammar_enable = {}
grammar_enable.cisco_ios = read_config('cisco_ios')
grammar_enable.cisco_nexus = read_config('cisco_nexus')
grammar_enable.cisco_wlc = read_config('cisco_wlc')
grammar_enable.netapp_cdot = read_config('netapp_cdot')
grammar_enable.netapp_balance = read_config('netapp_balance')

-- Build grammars
local vendors = {}
-- FIXME DUP of syslog
local function convert_pri(pri)
pri = tonumber(pri)
local facility = math.floor(pri/8)
local severity = pri % 8

return {facility = facility, severity = severity}
end
local pri = l.P'<' * l.Cg(l.digit^-3 / convert_pri, 'pri') *  l.P'>'

---
-- Cisco IOS
-- <157>6628: Aug 21 14:51:56: %SYS-5-CONFIG_I: Configured from console by admin onvty0 (10.20.30.42)
-- <157>6629: SP:   Aug 21 14:52:06: %PFINIT-5-CONFIG_SYNC: Sync'ing the startup configuration to the standby Router.
-- <189>2105: *May  1 23:29:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/7, changed state to up
-- Ref http://www.cisco.com/c/en/us/products/collateral/services/high-availability/white_paper_c11-557812.html
local ios_date = l.Cg(l.Ct(
 l.Cg(l.Cc('') / function() return os.date('%Y') end, 'year')
   * dt.date_mabbr
   * l.P' '
   * dt.date_mday_sp
   * l.P' '
   * dt.time_hour
   * l.P':'
   * dt.time_minute
   * l.P':'
   * dt.time_second
   * (l.P'.' * l.Cg(l.digit^1 / function(n) return tonumber(n)/1000 end, 'sec_frac'))^-1
   ) / dt.time_to_ns, 'timestamp')
vendors.cisco_ios = l.Ct(
pri
  * l.digit^1 / tonumber -- message number since boot
  * l.P': '
  * (l.P'SP' * l.P':   ')^-1 -- supervisor
  * (
   

Re: [heka] multiple file upload using HEKA

[Mathieu Parent]

2016-04-06 20:23 GMT+02:00 Rob Miller <rmil...@mozilla.com>:
> If I understand what you're saying, you're using LogstreamerInput
> incorrectly. A single LogstreamerInput is meant to map to either a single
> log stream, or possibly multiple logstreams of identical structure, such as
> a set of access logs for a number of different domains.

(Correct me if I'm wrong).

Appart from the "different structure", your "differentiator" should
resolve differently for each stream. i.e heka keep only one opened
file descriptor per differentiator value. The docs should be updated,
but finding the good wordings is hard.

See: https://github.com/mozilla-services/heka/pull/1904

Regards

-- 
Mathieu Parent
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] State and future of Heka

[Mathieu Parent]

Hi Rob,

Thanks for those info about the future of Heka.

You're seeking help to keep heka alive, but what about Hindsight?

It has currently no mailing list, the docs are minimal, and there are
no configuration examples (except the benchmarks dir). Any plan to
improve this? Heka's doc is one of its "selling" points.

Regards


2016-05-06 19:51 GMT+02:00 Rob Miller :
> Hi everyone,
>
> I'm lng overdue in sending out an update about the current state of and
> plans for Heka. Unfortunately, what I have to share here will probably be
> disappointing for many of you, and it might impact whether or not you want
> to continue using it, as all signs point to Heka getting less support and
> fewer updates moving forward.
>
> The short version is that Heka has some design flaws that make it hard to
> incrementally improve it enough to meet the high throughput and reliability
> goals that we were hoping to achieve. While it would be possible to do a
> major overhaul of the code to resolve most of these issues, I don't have the
> personal bandwidth to do that work, since most of my time is consumed
> working on Mozilla's immediate data processing needs rather than general
> purpose tools these days. Hindsight (https://github.com/trink/hindsight),
> built around the same Lua sandbox technology as Heka, doesn't have these
> issues, and internally we're using it more and more instead of Heka, so
> there's no organizational imperative for me (or anyone else) to spend the
> time required to overhaul the Go code base.
>
> Heka is still in use here, though, especially on our edge nodes, so it will
> see a bit more improvement and at least a couple more releases. Most
> notably, it's on my list to switch to using the most recent Lua sandbox
> code, which will move most of the protobuf processing to custom C code, and
> will likely improve performance as well as remove a lot of the problematic
> cgo code, which is what's currently keeping us from being able to upgrade to
> a recent Go version.
>
> Beyond that, however, Heka's future is uncertain. The code that's there will
> still work, of course, but I may not be doing any further improvements, and
> my ability to keep up with support requests and PRs, already on the decline,
> will likely continue to wane.
>
> So what are the options? If you're using a significant amount of Lua based
> functionality, you might consider transitioning to Hindsight. Any Lua code
> that works in Heka will work in Hindsight. Hindsight is a much leaner and
> more solid foundation. Hindsight has far fewer i/o plugins than Heka,
> though, so for many it won't be a simple transition.
>
> Also, if there's someone out there (an organization, most likely) that has a
> strong interest in keeping Heka's codebase alive, through funding or coding
> contributions, I'd be happy to support that endeavor. Some restrictions
> apply, however; the work that needs to be done to improve Heka's foundation
> is not beginner level work, and my time to help is very limited, so I'm only
> willing to support folks who demonstrate that they are up to the task.
> Please contact me off-list if you or your organization is interested.
>
> Anyone casually following along can probably stop reading here. Those of you
> interested in the gory details can read on to hear more about what the
> issues are and how they might be resolved.
>
> First, I'll say that I think there's a lot that Heka got right. The basic
> composition of the pipeline (input -> split -> decode -> route -> process ->
> encode -> output) seems to hit a sweet spot for composability and reuse. The
> Lua sandbox, and especially the use of LPEG for text parsing and
> transformation, has proven to be extremely efficient and powerful; it's the
> most important and valuable part of the Heka stack. The routing
> infrastructure is efficient and solid. And, perhaps most importantly, Heka
> is useful; there are a lot of you out there using it to get work done.
>
> There was one fundamental mistake made, however, which is that we shouldn't
> have used channels. There are many competing opinions about Go channels. I'm
> not going to get in to whether or not they're *ever* a good idea, but I will
> say unequivocally that their use as the means of pushing messages through
> the Heka pipeline was a mistake, for a number of reasons.
>
> First, they don't perform well enough. While Heka performs many tasks faster
> than some other popular tools, we've consistently hit a throughput ceiling
> thanks to all of the synchronization that channels require. And this
> ceiling, sadly, is generally lower than is acceptable for the amount of data
> that we at Mozilla want to push through our aggregators single system.
>
> Second, they make it very hard to prevent message loss. If unbuffered
> channels are used everywhere, performance plummets unacceptably due to
> context-switching costs. But using buffered channels means that many
> messages are in flight at a 

Re: [heka] New plugins

[Mathieu Parent]

Hello,

2016-09-07 20:47 GMT+02:00 Michael Trinkala :
[...]
> FYI: Everything in the Mozilla production environments will be on the 1.0
> sandbox and Hindsight by the end of the year.

Does this means that you don't use LogStreamerInput? (or that you'll
port it before 2017?)

Can you explain your process for nginx logs (and maybe apache, and
maybe syslog)? (it seems to go thru amazonWS).

Thanks

-- 
Mathieu
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Lua coroutines: Does it scales?

[Mathieu Parent]

Hi,

I'm currently trying to write a "syslog tcp" input. I've read heka_tcp
input as inspiration which uses coroutines.

I'm wondering, given that "Only one coroutine ever runs at a time"
[1], does it scales well?

How are you handling this at Mozilla?

Cheers,

[1] http://lua-users.org/wiki/CoroutinesTutorial

-- 
Mathieu Parent
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] Hinsight in pull mode

[Mathieu Parent]

Hello (again),

Hindsight is using a push-model (i.e Nagios passive checks). This is
great, but I want to plug it with Prometheus which uses pull-model
[1].

I see several ways to handle this:
- use the prometheus push-gateway [2]. This has several drawbacks listed below
- introduce pull model in hindsight
- add a new daemon, based on lua_sandbox too, but using pull model

The drawbacks of prometheus push-gateway are:
- Unnecessary polling of data (data is grabbed even if not pulled by prometheus
- time lag, between data grabbing and data pulling
- To sum up : to reduce time lag, you increase polling rate, when us
decrease polling, you increase time lag.

The push model may work like this:
- Adding pull_message_matcher config to inputs (defaults to FALSE)
- Adding process_pull_message() function to inputs, returning a table
of messages (or should it be inject_pull_message() + return 0?)
- Adding request_pull_message() function to outputs, which maps to
matching process_pull_message() and concatenates the results in a
table. This function is blocking.

Opinions?

NB: I won't work on this before 2017 or 2018, but I need to discuss
this early to.

NB2: My current TODO list is at:
https://gist.github.com/sathieu/5a7e83d514638f396e17d462f13adee0

[1]: 
https://prometheus.io/docs/introduction/faq/#why-do-you-pull-rather-than-push?
[2]: https://prometheus.io/docs/instrumenting/pushing/

-- 
Mathieu Parent
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Lua coroutines: Does it scales?

[Mathieu Parent]

2016-09-21 16:01 GMT+02:00 Michael Trinkala :
> It depends on the application but there can be benefits see (it has some
> performance comparisons): http://www.lua.org/pil/9.4.html

Thanks.

This is what I need.

-- 
Mathieu
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] Hinsight in pull mode

[Mathieu Parent]

2016-09-21 15:57 GMT+02:00 Michael Trinkala :
> We should probably be using the Hindsight list
> https://mail.mozilla.org/listinfo/hindsight for this

Didn't knew about this list. Subscribing...

> but I will provide a
> short answer here: Hindsight is agnostic when it comes to push or pull in an
> I/O plugin.  It is a function of the plugin and there are currently examples
> of both.  However, as Eric commented there is no HTTP server implementation
> available in a sandbox yet (although there are a few options).  See
> https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/socket/sandboxes/heka/output/heka_tcp_matcher.lua
> for an example of a TCP subscription based output plugin.

OK. Still this is a push model, and the output plugin is listening to
the messages (i.e the messages are not requested by the output plugin,
but by ticker_interval or input process_message()).

There is no hurry. Leaving this question open for now.

-- 
Mathieu
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka