Heka community:
I’m new to Heka and am having some difficulties setting up delivery of Apache
log messages from a local running Heka agent to a remote Heka instance via TCP
Output/Input plugins. The issue is directly coupled to using the Apache
Access Log Decoder configured on at the remote Heka instance. When this is
configured on my TCPInput there is no message Payload available to the [lua]
decoder. I have been able to create a working configuration that does not use
the Apache Access Log Decoder but would like to request assistance on how to
troubleshoot this issue further.
What I have done thus far is to modify the function process_message() in the
file: lua_decoders/apache_access.lua I’ve added a field called tcplog
containing the Payload to verify there is no message available to parse. The
modification I’ve made is described following the configurations below of my
working configuration and non-working configuration for comparison.
I must add that I am able to use the Apache Access Log Decoder with the
LogstreamerInput to process local files. So, this issue is specifically
related to the TCPInput/Apache Access Log Decoder combination.
My installation is Heka 0.9.1 on Ubuntu 14.04
# hekad -version
0.9.1
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty
I didn’t see any outstanding bugs related to my issues. Any advice would be
greatly appreciated.
Thanks,
Chris
Local running Heka agent used to ship Apache logs to remote Heka instance:
****************************************************************************************
[test_com]
type = "LogstreamerInput"
log_directory = "/export/test/apache2/test_com"
file_match = '/(?P<Year>)\d+/(?P<Month>\d+)_(?P<Day>\d+)_access\.log'
priority = ["Year", "Month", "Day"]
[aggregator_output]
type = "TcpOutput"
address = "10.10.10.1:5565"
message_matcher = “TRUE”
Remote Heka instance - Working config
****************************************************************************************
[TcpInput]
address = ":5565"
[Influxdb]
type = "SandboxEncoder"
filename = "lua_encoders/schema_influx.lua"
[Influxdb.config]
series = "%{logger}"
skip_fields = "Pid EnvVersion"
[FileOutput]
message_matcher = "TRUE"
path = "/home/giordano/heka/output.log"
perm = "775"
flush_count = 100
flush_operator = "OR"
encoder = "Influxdb"
Remote Heka instance - Configuration not working config
****************************************************************************************
[TcpInput]
address = ":5565"
decoder = "CombinedLogDecoder"
[CombinedLogDecoder]
type = "SandboxDecoder"
filename = "lua_decoders/apache_access.lua"
[CombinedLogDecoder.config]
type = "combinedutrack"
user_agent_transform = false
payload_keep = true
# combinedutrack log format
log_format = "%v %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\"
\"%{Cookie}i\""
[Influxdb]
type = "SandboxEncoder"
filename = "lua_encoders/schema_influx.lua"
[Influxdb.config]
series = "%{logger}"
skip_fields = "Pid EnvVersion"
[FileOutput]
message_matcher = "TRUE"
path = "/home/giordano/heka/output.log"
perm = "775"
flush_count = 100
flush_operator = "OR"
encoder = “Influxdb"
Additions to the apache access log decoder process_message() function
****************************************************************************************
function process_message ()
local log = read_message("Payload")
local fields = grammar:match(log)
— if not fields then return -1 end
if not fields then fields = {} end
fields.tcplog = log
msg.Timestamp = fields.time
fields.time = nil
…
Sample output from non working configuration
****************************************************************************************
[{"points":[[1427985263000,"combinedutrack","","","",7,""]],"name":"%{logger}",”columns":["time","Type","Payload","Hostname","Logger","Severity",”tcplog”]}]
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
