Right now there are only two mechanisms for getting the information about Heka's current operational state. One is that to which you're referring, i.e. parsing the JSON from which the DashboardOutput generates its UI. You can do this by fetching this through the dashboard's HTTP interface, reading it from its location on the host's file system, or by setting up a different filter and/or output that listens for the same messages that the DashboardOutput is receiving, i.e. `Type == 'heka.all-report'`, where the JSON will be the message payload.
Unfortunately, these messages will only be generated if a DashboardOutput is actually included as a part of your configuration. Ideally it'd be possible to generate these messages without explicitly including a DashboardOutput. It would also be nice to be able to generate the messages with the relevant data encoded into message fields, rather than a big ol' JSON blob in the payload. Neither of those are huge tasks, but they'd require some amount of work to accomplish. The other mechanism for extracting this information is by sending a SIGUSR1 signal to the running Heka process. This will cause the same data that's included in the health report JSON to be spit out in a human readable plain text format to Heka's console output. This was added so admins would be able to see what's going on in cases where Heka's router is wedged and messages aren't flowing, since in those cases clearly the report messages aren't making it through. A DashboardOutput does not need to be instantiated in this case, but of course monitoring and processing this data is tricky. You'd have to tease it out of the other output that Heka is emitting, and then parse the text format, which was designed for human eyeballs and not really for machine parsing. Your best bet for this type of monitoring would probably be to write a filter that listens for `heka.all-report` messages, parses the included JSON, maintains a bit of state, and sends an alert if it notices that certain tell-tale signs of slowness are persisting for more than a couple of ticks. This requires having a DashboardOutput in your configuration, and it's self-monitoring, so it will only work in cases where data is still flowing in at least some capacity, but it should work for most of the cases you're interested in. Hope this helps, -r On 11/16/2015 09:00 AM, Timur Batyrshin wrote:
Hi, Is there a way to measure Heka’s load except by parsing the output of DashboardPlugin in a special way? I’ve just hit a case when one of my router Heka’s had HttpOutput with non-existent host which severely decreased it’s performance (to ~10-20kb/sec which is no more than 1k messages/sec). I’d like to catch such cases in future as well as the cases when I need to think of some capacity increasing in other ways. I see only memory statistics in the standard Heka router messages while I think I’d wanted to watch for load in router and inputRecycleChan. Is there a standard way to do so? Thanks, Timur _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
