Re: [heka] Throttling email alerts

We usually throttle them at the source like https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/filters/http_status.lua#L85 but there is nothing to prevent you from adding global throttling or aggregation in the alert encoder (by host, plugin, application... whatever) Trink -

Re: [heka] [Heka] Test Sandboxes

Our sandbox unit tests can be found here: https://github.com/mozilla-services/heka/tree/dev/sandbox/plugins They should provide a reasonable example of how it can be done. Trink On Thu, Jan 22, 2015 at 10:42 AM, Djamel F. djamel...@gmail.com wrote: I wrote many sandboxes to treat data, and I

Re: [heka] Heka Release including Kafka Output

The current release as Kafka support. Trink On Tue, Feb 10, 2015 at 7:49 AM, Adriano Santos adriano.san...@gmail.com wrote: Hi All, Does anybody know when a new release of heka including Kafka will be out? Best, Adriano ___ Heka mailing

Re: [heka] Heka Release including Kafka Output

On Tue, Feb 10, 2015 at 10:10 AM, Michael Trinkala mtrink...@mozilla.com wrote: The current release as Kafka support. Trink On Tue, Feb 10, 2015 at 7:49 AM, Adriano Santos adriano.san...@gmail.com wrote: Hi All, Does anybody know when a new release of heka including Kafka

Re: [heka] load a large number of sandboxes.

It is only part of the build and not included in the installation/distribution. Since it just loads multiple versions of the same test sandbox it is only really useful for debugging during development. Trink On Mon, Feb 9, 2015 at 10:13 AM, Djamel F. djamel...@gmail.com wrote: recently I

[heka] Fwd: Timestamp Conversion

will be better able to assist you. Trink -- Forwarded message -- From: Madhukar Thota madhukar.th...@gmail.com Date: Sat, Mar 14, 2015 at 10:04 PM Subject: Re: [heka] Timestamp Conversion To: Michael Trinkala mtrink...@mozilla.com Getting following error: error: FATAL: process_message

Re: [heka] Timestamp Conversion

Just remove the capture group and the timestamp variable will contain the value you want. local timestamp = l.P[ * dt.build_strftime_grammar(%a %b %d %H:%M:%S.%s %Y) / dt.time_to_ns * ] if timestamp then msg.Timestamp = timestamp end Trink On Sat, Mar 14, 2015 at 8:58 PM, Madhukar Thota

Re: [heka] GeoIPdecoder with nginx access logs

You could just add it to the decoder if desired. Using the Lua geoip module it would look like this: require geoip.city local city_db = assert(geoip.city.open(read_config(geoip_city_db))) . . . fields.country_code = city_db:query_by_addr(fields.remote_addr, country_code) On Thu, Mar 5, 2015 at

Re: [heka] get ticker_interval value from a lua sandbox filter

Currently there is no way to access it, you would have to duplicate the entry in the config section of the plugin. Please file a feature request if desired, thanks. Trink On Mon, Mar 23, 2015 at 4:16 AM, bruno binet bruno.bi...@gmail.com wrote: Hi, I'd like to know the ticker_interval which

Re: [heka] Need help writing heka-s3 output plugin

You may want to take a look at https://github.com/mozilla-services/data-pipeline/blob/master/heka/plugins/s3splitfile/s3splitfile_output.go Trink On Wed, May 13, 2015 at 3:42 AM, Alex Jiao uohzx...@gmail.com wrote: Hi, I'm writing an S3 output plugin for heka. As I am new to golang and heka,

Re: [heka] Sandbox Decoder, not enough memory

The default limit is 8MiB, it can be adjusted with the memory_limit configuration parameter. https://hekad.readthedocs.org/en/latest/config/common_sandbox_parameter.html#config-common-sandbox-parameters I would recommend setting it to at least 10x the size of your largest document. The memory is

Re: [heka] Timestamp is set to UTC

Timestamp is the number on nanoseconds since the Unix epoch. The presentation is left to the consumer e.g., the RstEncoder will present it as UTC. Trink On Fri, Apr 3, 2015 at 1:35 AM, Cristian Falcas cristi.fal...@gmail.com wrote: Hello, I started playing with heka and I'm seeing that the

Re: [heka] Hindsight

). On 7 July 2015 at 16:17, Michael Trinkala mtrink...@mozilla.com wrote: Yeah, I have only been building on Ubuntu and haven't done any cross platform clean-up. Thanks for the build output I will fix those errors (unless you already have a patch). Trink On Tue, Jul 7, 2015 at 5:57 AM, bruno

Re: [heka] stock JSON decoder/encoder?

FYI: https://github.com/mozilla-services/heka/issues/826 On Mon, Aug 3, 2015 at 2:58 PM, Paul Bonser mister...@gmail.com wrote: I've written a Go JSON decoder which allows you to configure which JSON fields go into which Heka message struct fields. The rest of the fields get put in the Heka

Re: [heka] Hindsight

issues in github. Cheers, Bruno On 7 July 2015 at 18:26, Michael Trinkala mtrink...@mozilla.com wrote: I changed the checkpoint id to an unsigned long long. Can you test out the branch and add any other compilation errors to the issue (closing out this email thread). I am also taking

Re: [heka] Hindsight

, Michael Trinkala mtrink...@mozilla.com wrote: There are a few intentional changes between Heka and Hindsight. Looping messages in Heka has always been a bad idea so it was removed. Personally I like the looping messages feature in Heka as it is very flexible and could be useful to share ready

Re: [heka] Lua Decoder Parse Error

tname then > msg.Hostname = flds.hostname > flds.hostname = nil > end > > msg.Payload = log > msg.Fields = flds > > if not pcall(inject_message, msg) then return -1 end > return 0 > end > > I tried a few more things, but sti

Re: [heka] a question on heka message format

Always, it is by design as described in the docs for decode_message: https://hekad.readthedocs.org/en/latest/sandbox/index.html - message (table) The array based version of the message structure with the value member always being an array (even if there is only a single item). This

Re: [heka] Hindsight

. Thanks. Bruno On 10 June 2015 at 18:18, Michael Trinkala mtrink...@mozilla.com wrote: - It is usable and being actively developed with the intent to move it into production later this year. - We are currently running production data through it for testing but it is not deployed in an official

Re: [heka] My custom Lua decoder

Feedback added to the gist. Trink On Tue, Jul 7, 2015 at 12:06 PM, Rob Miller rmil...@mozilla.com wrote: On 07/07/2015 11:40 AM, Ali wrote: Hi, all! I'm finally done with my Lua decoder and thought I would post a link to it here, both for any constructive criticism anyone might have and

Re: [heka] Mirror of lpeg.trink.com for testing LPEG grammars?

Yeah, that VM was in a bad state I couldn't even ssh in. The reboot worked though so it is back up. Trink On Fri, Jul 10, 2015 at 8:54 AM, Ali h...@alijnabavi.info wrote: Hey, all. lpeg.trink.com seems to be down. Has anyone set up a publicly accessible instance of it? I'll see if I can

Re: [heka] Hindsight

b issue #7: https://github.com/trink/hindsight/pull/7 > > Cheers, > Bruno > > On 18 August 2015 at 06:18, Michael Trinkala <mtrink...@mozilla.com> > wrote: > >> Yes, I will get the plugins checked in shortly I just need to add a >> config option or two. >>

Re: [heka] apache logs not parsed

Did you set the decoder = "apache" in the LogstreamerInput configuration? Trink On Wed, Sep 23, 2015 at 1:16 AM, Cristian Falcas wrote: > Hello, > > I can't manage to make heka parse any apache logs. > > Here is my configuration: > > ## apache config > LogFormat "%h %l

Re: [heka] Writing a SandboxInput

Coroutines can be enabled (3 lines above the link Rob sent, same type of modification as the io tweak) Trink On Thu, Nov 19, 2015 at 1:50 AM, Kipras Mancevičius wrote: > Hi, thanks for all the info. I'll look into the options again. Most > probably going to recompile Heka

Re: [heka] State and future of Heka

Please don't mistake the small size of the documentation as it being incomplete. What is there is a full reference to the current functionality. Its small size (when compared to the Heka documentation) is due to the reduction of complexity in it's design (Hindsight is just a skeleton around the

Re: [heka] Hinsight in pull mode

We should probably be using the Hindsight list https://mail.mozilla.org/ listinfo/hindsight for this but I will provide a short answer here: Hindsight is agnostic when it comes to push or pull in an I/O plugin. It is a function of the plugin and there are currently examples of both. However, as

Re: [heka] Hinsight in pull mode

for Prometheus. If you wanted to go the other way the output plugin would be an HTTP client that POSTs the data to an HTTP server. Trink On Wed, Sep 21, 2016 at 7:06 AM, Mathieu Parent <math.par...@gmail.com> wrote: > 2016-09-21 15:57 GMT+02:00 Michael Trinkala <mtrink...@mozilla.com>

Re: [heka] Lua coroutines: Does it scales?

It depends on the application but there can be benefits see (it has some performance comparisons): http://www.lua.org/pil/9.4.html Trink On Wed, Sep 21, 2016 at 1:20 AM, Mathieu Parent wrote: > Hi, > > I'm currently trying to write a "syslog tcp" input. I've read