Any time there is an error that causes a buffered record to not be sent to ElasticSearch Heka should be emitting that error to its own console output. Do you not see any errors in your Heka output?
-r On 04/07/2015 09:47 AM, Ali wrote:
I fixed my original problem, but am wondering how to better diagnose what's going on with Heka in the future. I checked the ElasticSearch log and saw an error I mentioned on here a few days ago. An earlier incarnation of my decoder had a type containing capital letters. This was causing ElasticSearch to throw an error because it wouldn't create an index that wasn't lower case. Restarting the server didn't help because the output_queue still existed. I stopped hekad, deleted the output_queue, restarted hekad, and the records are now appearing in ElasticSearch. Is there a way I could have diagnosed this only using Heka, without looking at the ElasticSearch logs? -Ali On Tue, Apr 7, 2015 at 11:20 AM Ali <[email protected] <mailto:[email protected]>> wrote: Morning, all. Here's my hekad.toml: https://gist.github.com/hourback/36f29939a804becd9723 I am using a TcpInput for IBM HTTPServer access log data. I'm using a similar pipeline for Rsyslog messages coming from the same remote host (via nxlog) and things are working okay. However, for this TcpInput, nothing seems to be coming through, i.e., the input and decoder show up in the Heka dashboard, but the number of messages processed is always zero. I was originally using the unfortunately named IbmHttpserverAccessCommonLogDecoder as my TcpInput. When that wasn't showing results, I decided to test things by creating another decoder, the ApacheRegexDecoder of type PayloadRegexDecoder. My intention was to just grab everything sent to the decoder and throw it back out to ElasticSearch with the rest of the log data so I could verify that the data is actually getting to Heka from the remote host. However, nothing seems to be coming through here either. I'm sure I'm getting dozens of things wrong so whatever tips you have would be appreciated. :-) TIA, Ali _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
