Hi Rob, Parsing heka.all-report messages looks fine as well as having DashboardOutput. I was just hoping that there is already something ready for that :-)
One more question related to my original issue but a bit away from topic on performance: As I’ve mentioned earlier I run heka agents on multiple boxes. They send collect local data and send it via plain TCP to heka aggregator. This central heka server then sends traffic further to storage via HTTP. In my setup I see caches/logfiles on disk growing only on collectors but not on aggregator. This means that if network connectivity issues appear on aggregator the caches will fill up disks on end-boxes but not on aggregator, right? Also looks like they'll even start dropping the messages. So the question is: Is there a way to make HttpOutput cache outgoing traffic in the same way TcpOutput does? Or is there any other way to add persistence to it (without running external services)? Thanks, Timur On 16 Nov 2015 at 22:56:55, Rob Miller ([email protected]) wrote: Right now there are only two mechanisms for getting the information about Heka's current operational state. One is that to which you're referring, i.e. parsing the JSON from which the DashboardOutput generates its UI. You can do this by fetching this through the dashboard's HTTP interface, reading it from its location on the host's file system, or by setting up a different filter and/or output that listens for the same messages that the DashboardOutput is receiving, i.e. `Type == 'heka.all-report'`, where the JSON will be the message payload. Unfortunately, these messages will only be generated if a DashboardOutput is actually included as a part of your configuration. Ideally it'd be possible to generate these messages without explicitly including a DashboardOutput. It would also be nice to be able to generate the messages with the relevant data encoded into message fields, rather than a big ol' JSON blob in the payload. Neither of those are huge tasks, but they'd require some amount of work to accomplish. The other mechanism for extracting this information is by sending a SIGUSR1 signal to the running Heka process. This will cause the same data that's included in the health report JSON to be spit out in a human readable plain text format to Heka's console output. This was added so admins would be able to see what's going on in cases where Heka's router is wedged and messages aren't flowing, since in those cases clearly the report messages aren't making it through. A DashboardOutput does not need to be instantiated in this case, but of course monitoring and processing this data is tricky. You'd have to tease it out of the other output that Heka is emitting, and then parse the text format, which was designed for human eyeballs and not really for machine parsing. Your best bet for this type of monitoring would probably be to write a filter that listens for `heka.all-report` messages, parses the included JSON, maintains a bit of state, and sends an alert if it notices that certain tell-tale signs of slowness are persisting for more than a couple of ticks. This requires having a DashboardOutput in your configuration, and it's self-monitoring, so it will only work in cases where data is still flowing in at least some capacity, but it should work for most of the cases you're interested in. Hope this helps, -r On 11/16/2015 09:00 AM, Timur Batyrshin wrote: > Hi, > > Is there a way to measure Heka’s load except by parsing the output of > DashboardPlugin in a special way? > > I’ve just hit a case when one of my router Heka’s had HttpOutput with > non-existent host which severely decreased it’s performance (to > ~10-20kb/sec which is no more than 1k messages/sec). > I’d like to catch such cases in future as well as the cases when I need > to think of some capacity increasing in other ways. > > I see only memory statistics in the standard Heka router messages while > I think I’d wanted to watch for load in router and inputRecycleChan. > Is there a standard way to do so? > > Thanks, > Timur > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
