Hi, I am running an instance of Heka, v0.9.1 on one server as a log shipper of Apache access logs to a remote Heka instance on another server. The log shipper instance is sending the logs via the TcpOutput plugin. The problem I’m experiencing is with the queue. Regardless of the settings the queue fills up and we either run out of disk space or the instance of Heka stops working. Below I’ve described the results of the different settings used in the config file followed by the configuration I’m using on the log shipper instance. I’ve set the ticker_interval down to 30 seconds due to the volume of Apache logs I’m processing. Even so, the queue files fill up moments after starting Heka.
1) With no queue settings (i.e. using the defaults), the queue files fill up, rotate and accumulate until the disk fills up. 2) With queue_full_action=shutdown, Heka dutifully stops when the first queue file reaches capacity. 3) With queue_fill_action=block, Heka pauses when the queue is full and reports a message to STDOUT: 2015/04/15 14:04:40 Plugin 'TcpOutput' error: Queue is full But, if I empty the queue file (i.e. cat /dev/null >| 1.log), log processing never resumes. Instead, it reports the following message to STDOUT. At this point, I have to kill -9 the instance of Heka. 2015/04/15 14:06:41 Diagnostics: 32 packs have been idle more than 120 seconds. 2015/04/15 14:06:41 Diagnostics: (input) Plugin names and quantities found on idle packs: 2015/04/15 14:06:41 Diagnostics: TcpOutput: 32 4) With queue_fill_action=drop, Heka pauses when the queue is full and reports the following message to STDOUT. At some point, it creates a new queue file, zero bytes in size that never gets used. And, the checkpoint.txt file even refers to the new queue file that got created. Yet, if I attempt to empty or delete the filled queue file, Heka never resume processing logs. 2015/04/15 14:10:05 Plugin 'TcpOutput' error: Queue is full I must add that all at the messages are properly being sent to the remote Heka instance. And, I would expect the TcpOutput plugin to only queue entries that don’t get sent. As would be the case if the the remote instance was unreachable. What is the purpose of the TcpOutput queue files and how do I manage them so I can use Heka as a log shipper running continuously on a production server. Thanks Chris [hekad] maxprocs = 2 [test_com] type = "LogstreamerInput" log_directory = "/export/test/apache2/test_com" file_match = '/(?P<Year>)\d+/(?P<Month>\d+)_(?P<Day>\d+)_access\.log' priority = ["Year", "Month", "Day"] [TcpOutput] address = "10.10.10.1:5565" message_matcher = "TRUE" # 30 seconds ticker_interval=30 # 1 GB queue_max_buffer_size=1073741824 queue_full_action="drop"
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
