Hello,
the RFC5424 is quite liberal to changes of control characters in MSG
part, probably for historical reasons, however, it does not allow for
such a replacement (or avoidance or escaping) in PARAM-VALUE, which is
arbitrary UTF-8 string.
I think this is why RFC5425 introduced the MSG-LEN framing for
stream-oriented TLS transports, we use the same framing for plain TCP
as well - "InputType Syslog_TLS" in im_tcp etc.
Note this is not just my excessive requirement to be RFC5424 compliant
in every detail:
- im_mseventlog and/or im_msvistalog already produce structured fields
with line breaks in their values (only in a fraction of events, but
from 20 servers in total, we are receiving such an event several times
an hour),
- software written in languages with exception handling often also
generate events with line breaks in MSG part: the stack trace -- some
syslog systems erroneously split such an event into several messages
line by line - I am trying to avoid this.
So what is our replacement for line breaks?
- in datagram transport (UDP, Unix datagram socket) - the events are
framed by the datagram,
- in stream transport (TCP, TLS) - MSG-LEN framing,
- in files - we are still using '\n' for the files to be well readable
by humans (and to be similar to plain old syslog) -- when reading back
such a file into NXLog, we use NXLog's xm_multiline module with
"HeaderLine /^<\d{1,3}>1 (- |\d{4}-)/" - this splits the input into
events with "high probability only", but more complicated regexps
would not increase the probability anyway.
For files, I have considered to escape line breaks in MSG as "\n" or
"\r", but 1) it is not allowed to do such an escape in PARAM-VALUE, 2)
it makes the multi-line events human-unreadable. So I ended up with
the xm_multiline and "high probability". Much better than replacing
line breaks with spaces, isn't it? :-)
Milan
On Wed, Jan 27, 2016 at 1:11 PM, Botond Botyanszki <b...@nxlog.org> wrote:
> Hi,
>
> On Tue, 26 Jan 2016 20:35:53 +0100
> Milan Krčmář <milan.krc...@gmail.com> wrote:
>
>> why does the to_syslog_ietf() replace line breaks ('\r','\n') in
>> message text with space (' ')? [in nx_logdata_to_syslog_rfc5424(),
>> almost at the end]
> That's done because linebreaks represent the end-of-record when syslog is
> transferred over TCP or TLS/SSL.
>
>> In RFC5424, the message text is allowed to contain any octets.
> This is what RFC5424 says:
> "6.4. MSG
> The syslog application SHOULD avoid octet values below 32 (the
> traditional US-ASCII control character range except DEL). These
> values are legal, but a syslog application MAY modify these
> characters upon reception. For example, it might change them into an
> escape sequence (e.g., value 0 may be changed to "\0"). "
>
>> The IETF standard
>> allows the line breaks to appear in structured data fields as well -
>> these are preserved. So, in output, you might get line breaks in
>> middle of whole event anyway.
> This might need to be reviewed to protect against such issues.
>
> Regards,
> Botond
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> nxlog-ce-users mailing list
> nxlog-ce-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
