On Wed, Jun 12, 2013 at 3:21 PM, Radu Gheorghe <[email protected]>wrote:
> Hi Rainer, > > Thanks a lot for your reply! And quick hints are all I need, as I can > continue with researching and testing. > > Regarding tools: this is exactly why I decided to ask instead of testing, > because if a test fails, I'm not sure whether it's the tools fault or > rsyslog's. > > Tools aside, it seems like my options are: > - use UDP with RFC5424 > - stay away from multiline messages. I don't think this would be an option > for me, as I'd want to be able to send stacktraces over syslog. > IMHO multiline messages in syslog (or logging in general) are broken design. You get into so much trouble at so many places. I would concentrate on fixing them up as early as possible, doing escaping in order to provide them. There is no real standard to quote, but RFC5424 somewhat recommends (only via example) #ooo with o being octal digits - or the well-known \n. But... just my 2cts here. > > Not so sure about the following: > - syslog over TCP (5424 or not) is a no-no for multi-lined messages, > because you need a delimiter. I could change the delimiter from \n to > something else, but that might confuse a lot of things > well... rsyslog supports the same octet-counted framing that RFC5425 requires. With it, multiline messages *can* be transmitted without any problems (as far as the network is concerned). > - what about TLS (RFC 5425)? I see nice keywords there like "message lenth" > and "syslog frame". This should mean \n delimiters are not required, so one > could use this for sending multi-lined messages. > > yup You may also want to have a look at RFC 6587, where we elaborate about the multiline problem and framing. But again - even if you manage to transfer the messages without problems, almost all log processing tools expect a single log record to be on one line. So unless you have a total custom solution, you really, really will get into troubles. Rainer > Can you say if I got it right with the bullets above? Or comment on them? > > Thanks again! > Radu > > > > 2013/6/12 Rainer Gerhards <[email protected]> > > > I try to provide at least some quick hints while I am too busy to do the > > full writeup (would fill many pages). > > > > The core problem is > > > > a) traditional syslog > > b) the tooling around it > > > > So even if rsyslog processes the messages correctly, a lot of tools will > > miserably fail. The unfortunately is no accepted standard on how to treat > > things. RFC5424 strongly recomendes to use printable characters, only, in > > messages. > > > > With UDP, messages can include LF - confusion will probably happen later > in > > the toolchain. RFC5426 does also support it. > > > > My strong advise is to stay away from multi-line messages. > > > > HTH (a bit) > > Rainer > > > > > > On Wed, Jun 12, 2013 at 12:27 PM, Radu Gheorghe <[email protected] > > >wrote: > > > > > Hi, > > > > > > I'm sure the subject looks familiar, and indeed I found a lot of Q&As > > > regarding how to send multiline logs over syslog. But I still don't > see a > > > good solution, so I'm opening yet another thread. > > > > > > The only solution I found so far (credits go to David) was to escape > > > newline characters and un-escape them while reading the logs. I'd like > to > > > index logs in Elasticsearch, so I'd prefer to store them with the > proper > > > newline. Can I un-escape the newline in the template when I build the > > JSON > > > to send to ES? > > > > > > Either way, how would one: > > > - log a multi-line log to syslog from an application? > > > - configure rsyslog to read multi-line logs from a file (ie: > stacktraces, > > > if the line begins with the space, it belongs to the same event as the > > > previous line) > > > - forward multi-line logs from one rsyslog to another > > > > > > My understanding is that \n is normally a delimiter between log > messages. > > > Does that only apply to TCP syslog? What about UDP or TLS? Can multiple > > > lines fit in one packet and be treated as a single event? > > > > > > Finally, is there any difference between RFC 3164 and RFC 5424 syslog > for > > > multi-line logs? > > > > > > Thanks and best regards, > > > Radu > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
