Okay, here's a lua filter and accompanying TOML config that should work
for you:
https://gist.github.com/rafrombrc/3858a7885e766fbefa61
Some notes:
* This should *replace* all of the 'nginx_access_stat' and
'nginx_access_stats' related sections in your original config.
* Depending on how large your stat set grows, you may bump up against
some sandbox output limits. You can increase these by raising
memory_limit, intruction_limit, or output_limit settings in the
SandboxFilter config (see
http://hekad.readthedocs.org/en/v0.9.1/config/common_sandbox_parameter.html#config-common-sandbox-parameters).
* I set up a LogOutput to capture any heka.sandbox-terminated messages.
This way if the filter crashes you'll have some console output
explaining why. The DashboardOutput shows this automatically, but I know
you're running w/o a dashboard until we get #1422 resolved.
* It's occurred to me while writing this that the bulk of the logic in
the 'timer_event' function should be added to a Lua module that we can
ship with Heka, so in the future generating all of the various counter
and timer values from the input stats data would just be a matter of
calling some functions. That would make it trivial to develop future
similar filters.
* Hopefully this is all clear enough, and a solid enough start, that you
can further customize and tailor it as your needs change.
Try this out and let me know how it goes... hopefully you'll see much
better throughput.
-r
On 04/01/2015 02:46 AM, Alex Muller wrote:
(In reply to https://mail.mozilla.org/pipermail/heka/2015-March/000388.html)
Hi Rob,
Thanks for your email a couple of weeks ago with some suggestions for
alleviating issues with Heka that we've seen.
> The easy solution is to remove the plugin_chansize change,
> and to bump your global pool_size setting up to about 130.
We've had this running for about a week now. The updated output from
sending a SIGUSR1 to the hekad process looks like this:
https://gist.github.com/alexmuller/73f824184f92adcc56d6
Unfortunately our machines which see most of our traffic (our cache nodes)
still can't keep up.
> This would probably be about 30 minutes of work, if you're interested in
> trying it out I'd be happy to whip something up for you to try out.
If you're willing to give this a go we'd be very happy to run it and see
what happens.
Cheers,
Alex
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
