On Tue, 3 Jun 2014, Duarte Silva wrote:
On Tuesday 03 June 2014 14:20:33 Rainer Gerhards wrote:
On Tue, Jun 3, 2014 at 2:11 PM, Duarte Silva <[email protected]>
wrote:
On Tuesday 03 June 2014 13:18:28 Rainer Gerhards wrote:
On Tue, Jun 3, 2014 at 1:01 PM, Duarte Silva <
[email protected]>
wrote:
Hi David,
thanks for the tip. I can now understand what Rsyslog is doing when it
receives the data :)
Since the appliance is sending the JSON in a multi-line format, the
last
line
isn't being interpreted as the message, but rather it's being
interpreted
as
all the other fields.
Since I can't modify the way the appliance sends the logs, is there a
way
in
Rsyslog to get around this?
Which protocol is used? Can you post a sample of the data **as seen on
the
wire**?
Quoting:
In tcpdump I clearly see the full message arriving:
# tcpdump -A -n -vvv -s2048 -i eth0 "tcp port 514"
(...)
=.2.....<164>appliance-892.alert: {
"alert": {
"action": "notified",
"explanation": "something"
},
"id": "892",
"occurred": "2014-06-02T15:00:24Z",
},
"appliance": "appliance",
"product": "product",
"version": "version"
}
.
So this is exactly what you see (not pretty-printed)? So it starts with:
<164>appliance-892.alert: {<LF>"alert": {<LF>"action": "notified",<LF>...
<LF> being US-ASCII LF
Yes, that is how it arrives.
If so, that appliance is sending horribly malformed messages with a totally
invalid framing. Honestly, I have no idea at all how a heuristic could
combine those lines. What do they say what the framing is (in other words:
how to know when the message is finished and the next one starts)?
In any case, it's not syslog protocol. See RFC5424 or even 3164 and you'll
see that this is something totally different. Among others, I'd
**strongly** suggest to file a bug report with them.
If you can get the information out of them how to know message borders, I
may be able to implement their "protocol", but that will be a custom
project (aka needs a sponsor).
From the appliance configuration, they are using syslog only as a transport for
the messages. The messages can then be XML or JSON. I don't think I will have
any luck in trying to swing the appliance maker to make the messages a one
liner. I will try to home brew something out.
They may be using the syslog port, but this isn't syslog transport.
is this being sent of TCP or UDP? can you send us a short tcpdump of the
messages?
if UDP, are they sending one message per packet? or can one message span
multiple packets? if one message can span multiple packets, then they are in
deep trouble because UDP is unreliable delivery and packets can get lost or
arrive out of order.
If this is TCP, then a parser module could read the stream and treat each
complete JSON object as a separate message. this would require a custom module.
What appliance is this?
Compared to what I'm sure you spent on the appliances, paying for a custom
module to receive these messages will be pretty cheap, talk with Rainer off of
the main list to get a quote for this. I've done it in the past. It's much
nicer to throw a little money at Adiscon and have it be part of the core rsyslog
than to hack something up and have to maintain it for future versions.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
