Hi,
I've got some logs in format:
yyyy-MM-ddTHH:mm:ssZ (Some header #1)
{below multiline message body, eg. PHP array dump}
yyyy-MM-ddTHH:mm:ssZ (Some header #2)
{below multiline message body, eg. PHP array dump}
Example:
2015-01-02T13:14:15+01:00 DB QUERY:
SELECT id, name
FROM table
WHERE id = 15
Namespace\Class:method /path/to/file/calleing/log/Class:123
2015-01-02T13:14:15+01:00 DB QUERY:
SELECT id, somethingelse
FROM table2
WHERE somethingelse = 'foobar'
Namespace\Class:method /path/to/file/calleing/log/Class:123
Another Example (diffrent log, similar problem):
2015-01-02T13:14:15+01:00 REQUEST -> http://123.45.67.89:0123
array (
0 => 'get_something_action',
1 =>
array (
'params' =>
array (
'someparam' => 'foo',
'otherparam' => 'bar,
)
)
)
2015-01-02T13:14:15+01:00 RESPONSE <- http://123.45.67.89:0123
array (
'state' => 'ok',
'params' =>
array (
'some_return_param' => 'foo_bar_baz'
)
)
I am useing RegexSplitter with delimiter configured like this:
* delimiter = '(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:\d{2} DB
QUERY:)\n'
* delimiter = '(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:\d{2} \S+
[<-][->] \S+.*)\n'
* delimiter = '(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:\d{2}
.+)\n'
(Diffrent delimiters are for diffrent logs, but their structure is the same -
the only diffrence is the "header" following the timestamp).
When I run hekad (with PayloadEncoder/LogOutput) I am getting alerts:
(note: empty lines = successful messages, becouse I have the "payload_keep"
param set to false)
(note2: failed parsing empty payload is also OK - it means the log file begins
with an empty line, which is not a valid message)
Example output:
2015/07/17 10:44:51
2015/07/17 10:44:51
2015/07/17 10:44:51 Decoder 'AcmeNLogInput-AcmeNLogDecoder-23' error: Failed
parsing: payload:
2015/07/17 10:44:51 Decoder 'AcmeDBLogInput-AcmeDBLogDecoder-6' error: Failed
parsing: payload: 15-04-12T15:06:45+02:00 DB QUERY:
select * from session where token = :token
{":token":"foobar"}
Acme\Database::isSessionExist /path/to/src/Acme/Database.php:123
2015/07/17 10:44:51
2015/07/17 10:44:51 Decoder 'AcmeDBLogInput-AcmeDBLogDecoder-5' error: Failed
parsing: payload: _name, hash
FROM devices
WHERE device_hash = :hash AND
client_name = :name AND
ip_address = :ip
{":ip":"123.45.67.89",":hash":"foobarbaz",":name":"ACME-01-NYC"}
Acme\Database::getDevice /path/to/src/Acme/Database.php:456
2015/07/17 10:44:51
2015/07/17 10:44:51 Decoder 'AcmeDBLogInput-AcmeDBLogDecoder-14' error: Failed
parsing: payload: arams /path/to/src/Acme/Database.php:234
2015/07/17 10:44:51
2015/07/17 10:44:51 Decoder 'AcmeDBLogInput-AcmeDBLogDecoder-9' error: Failed
parsing: payload: _hash
FROM devices
WHERE device_hash = :hash AND
client_name = :name AND
ip_address = :ip
{":ip":"98.76.54.32",":hash":"barfoobaz",":name":"ACME-01-NYC"}
Acme\Database::getDevice /path/to/src/Acme/Database.php:456
My question is:
* why does the payload start with "arams /path/to/src/Acme/Database
(...)" or "_hash (...)" or "15-04-12T15:06:45+02:00 DB QUERY (...)"?
it should ALWAYS start with yyyy-MM-ddTHH:mm:ss {Some Header} - all decoders
start with timestamp pattern and `delimiter_eol` is set to false (so it should
prepend the payload)
* I've found these messages in my log files, and they are no diffrent
(in format/pattern) from others
It seems like sometimes (?) the payload (for some reason) gets stripped?
My hekad config:
[hekad]
max_message_size = 15728640 # 15MB
maxprocs = 4
poolsize = 10
Example Splitter config:
[AcmeMainLogSplitter]
type = "RegexSplitter"
delimiter = '(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:\d{2} DEVICE:.*)\n'
delimiter_eol = false
Example Decoder config:
[AcmeMainLogDecoder]
type = "SandboxDecoder"
filename = "/etc/heka/lua_decoders/acme_main_log.lua"
memory_limit = 15728640 # 15MB
output_limit = 15728640 # 15MB
instruction_limit = 1500000 # 1.5 million
Any ideas why this happens?
---------------------------------------------
Siedziba: Getin Noble Bank SA, ul. Przyokopowa 33, 01-208 Warszawa
Sad rejestrowy: Sad Rejonowy dla m.st. Warszawy w Warszawie, XII Wydzial
Gospodarczy.
Numer KRS: 0000304735
NIP: 108-000-48-50
Wysokosc kapitalu zakladowego oplaconego w calosci: 2 650 143 319,00 zl
Zamieszczenie powyzszych danych identyfikujacych Getin Noble Bank SA stosownie
do art. 374 par.1 Kodeksu spolek handlowych nie jest rownoznaczne z handlowym
charakterem dostarczonej do Panstwa
wiadomosci e-mailowej i pozostaje bez wplywu na interpretacje zawartych w niej
oswiadczen.**********
Niniejszy e-mail oraz wszelkie zalaczone do niego pliki sa poufne i moga
podlegac ochronie prawnej. Jezeli nie jest Pan/Pani zamierzonym adresatem
powyzszej wiadomosci, nie moze jej Pan/Pani ujawniac, kopiowac, dystrybuowac,
ani tez w zaden inny sposob udostepniac lub wykorzystywac. O blednym
zaadresowaniu wiadomosci prosimy niezwlocznie poinformowac nadawce i usunac
wiadomosc.**********
This e-mail message may contain confidential and/or privileged information. If
you are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any unauthorized
copying, disclosure or distribution of the material in this e-mail is strictly
forbidden.
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
