On Tuesday 03 June 2014 10:11:31 David Lang wrote: > On Tue, 3 Jun 2014, Duarte Silva wrote: > > On Tuesday 03 June 2014 14:20:33 Rainer Gerhards wrote: > >> On Tue, Jun 3, 2014 at 2:11 PM, Duarte Silva > >> <[email protected]> > >> > >> wrote: > >>> On Tuesday 03 June 2014 13:18:28 Rainer Gerhards wrote: > >>>> On Tue, Jun 3, 2014 at 1:01 PM, Duarte Silva < > >>> > >>> [email protected]> > >>> > >>>> wrote: > >>>>> Hi David, > >>>>> > >>>>> thanks for the tip. I can now understand what Rsyslog is doing when it > >>>>> receives the data :) > >>>>> > >>>>> Since the appliance is sending the JSON in a multi-line format, the > >>> > >>> last > >>> > >>>>> line > >>>>> isn't being interpreted as the message, but rather it's being > >>> > >>> interpreted > >>> > >>>>> as > >>>>> all the other fields. > >>>>> > >>>>> Since I can't modify the way the appliance sends the logs, is there a > >>> > >>> way > >>> > >>>>> in > >>>>> Rsyslog to get around this? > >>>> > >>>> Which protocol is used? Can you post a sample of the data **as seen on > >>> > >>> the > >>> > >>>> wire**? > >>> > >>> Quoting: > >>>>>> In tcpdump I clearly see the full message arriving: > >>>>>> > >>>>>> # tcpdump -A -n -vvv -s2048 -i eth0 "tcp port 514" > >>>>>> (...) > >>>>>> =.2.....<164>appliance-892.alert: { > >>>>>> > >>>>>> "alert": { > >>>>>> > >>>>>> "action": "notified", > >>>>>> "explanation": "something" > >>>>>> }, > >>>>>> "id": "892", > >>>>>> "occurred": "2014-06-02T15:00:24Z", > >>>>>> > >>>>>> }, > >>>>>> "appliance": "appliance", > >>>>>> "product": "product", > >>>>>> "version": "version" > >>>>>> > >>>>>> } > >>>>>> . > >> > >> So this is exactly what you see (not pretty-printed)? So it starts with: > >> > >> <164>appliance-892.alert: {<LF>"alert": {<LF>"action": "notified",<LF>... > >> > >> <LF> being US-ASCII LF > > > > Yes, that is how it arrives. > > > >> If so, that appliance is sending horribly malformed messages with a > >> totally > >> invalid framing. Honestly, I have no idea at all how a heuristic could > >> combine those lines. What do they say what the framing is (in other > >> words: > >> how to know when the message is finished and the next one starts)? > >> > >> In any case, it's not syslog protocol. See RFC5424 or even 3164 and > >> you'll > >> see that this is something totally different. Among others, I'd > >> **strongly** suggest to file a bug report with them. > >> > >> If you can get the information out of them how to know message borders, I > >> may be able to implement their "protocol", but that will be a custom > >> project (aka needs a sponsor). > > > > From the appliance configuration, they are using syslog only as a > > transport for the messages. The messages can then be XML or JSON. I don't > > think I will have any luck in trying to swing the appliance maker to make > > the messages a one liner. I will try to home brew something out. > > They may be using the syslog port, but this isn't syslog transport. > > is this being sent of TCP or UDP? can you send us a short tcpdump of the > messages?
It can be sent over TCP or UDP (the example I gave was TCP, check the tcpdump command line). Not really, sorry. > if UDP, are they sending one message per packet? or can one message span > multiple packets? if one message can span multiple packets, then they are in > deep trouble because UDP is unreliable delivery and packets can get lost or > arrive out of order. Yes, one of the problems I noticed was that the UDP notification wasn't contiguous (spanned throughout multiple packets), hence the switch to TCP in the appliance configuration. > If this is TCP, then a parser module could read the stream and treat each > complete JSON object as a separate message. this would require a custom > module. > > What appliance is this? Malware related, their logging is crap (for example they don't even allow a Rsyslsog server port change in the configuration). > Compared to what I'm sure you spent on the appliances, paying for a custom > module to receive these messages will be pretty cheap, talk with Rainer off > of the main list to get a quote for this. I've done it in the past. It's > much nicer to throw a little money at Adiscon and have it be part of the > core rsyslog than to hack something up and have to maintain it for future > versions. I decided to drop Rsyslog and went to Logstash. Not using the appliance Rsyslog notifications capabilities though. Used the appliance HTTP notifications instead (sends a POST with the JSON encoded notification using CURL). > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
