catching up on e-mail, even with UDP syslog, linebreaks in the message can cause
problems.
While transporting such messages can be made to work, too many tools that then
process the messages will then break. It's far better to escape newlines in some
way.
David Lang
On Mon, 1 Feb 2016, Botond Botyanszki wrote:
Hi,
Perhaps an option could be added to xm_syslog not to replace linebreaks
but most users want it to work out of the box and line breaks will cause
the syslog to be malformed on the receiver, unless it is UDP syslog.
The Syslog_TLS style framing is is even more uncommon.
Regards,
Botond
On Fri, 29 Jan 2016 15:59:34 +0100
Milan Krčmář <milan.krc...@gmail.com> wrote:
Hello,
the RFC5424 is quite liberal to changes of control characters in MSG
part, probably for historical reasons, however, it does not allow for
such a replacement (or avoidance or escaping) in PARAM-VALUE, which is
arbitrary UTF-8 string.
I think this is why RFC5425 introduced the MSG-LEN framing for
stream-oriented TLS transports, we use the same framing for plain TCP
as well - "InputType Syslog_TLS" in im_tcp etc.
Note this is not just my excessive requirement to be RFC5424 compliant
in every detail:
- im_mseventlog and/or im_msvistalog already produce structured fields
with line breaks in their values (only in a fraction of events, but
from 20 servers in total, we are receiving such an event several times
an hour),
- software written in languages with exception handling often also
generate events with line breaks in MSG part: the stack trace -- some
syslog systems erroneously split such an event into several messages
line by line - I am trying to avoid this.
So what is our replacement for line breaks?
- in datagram transport (UDP, Unix datagram socket) - the events are
framed by the datagram,
- in stream transport (TCP, TLS) - MSG-LEN framing,
- in files - we are still using '\n' for the files to be well readable
by humans (and to be similar to plain old syslog) -- when reading back
such a file into NXLog, we use NXLog's xm_multiline module with
"HeaderLine /^<\d{1,3}>1 (- |\d{4}-)/" - this splits the input into
events with "high probability only", but more complicated regexps
would not increase the probability anyway.
For files, I have considered to escape line breaks in MSG as "\n" or
"\r", but 1) it is not allowed to do such an escape in PARAM-VALUE, 2)
it makes the multi-line events human-unreadable. So I ended up with
the xm_multiline and "high probability". Much better than replacing
line breaks with spaces, isn't it? :-)
Milan
On Wed, Jan 27, 2016 at 1:11 PM, Botond Botyanszki <b...@nxlog.org> wrote:
> Hi,
>
> On Tue, 26 Jan 2016 20:35:53 +0100
> Milan Krčmář <milan.krc...@gmail.com> wrote:
>
>> why does the to_syslog_ietf() replace line breaks ('\r','\n') in
>> message text with space (' ')? [in nx_logdata_to_syslog_rfc5424(),
>> almost at the end]
> That's done because linebreaks represent the end-of-record when syslog is
> transferred over TCP or TLS/SSL.
>
>> In RFC5424, the message text is allowed to contain any octets.
> This is what RFC5424 says:
> "6.4. MSG
> The syslog application SHOULD avoid octet values below 32 (the
> traditional US-ASCII control character range except DEL). These
> values are legal, but a syslog application MAY modify these
> characters upon reception. For example, it might change them into an
> escape sequence (e.g., value 0 may be changed to "\0"). "
>
>> The IETF standard
>> allows the line breaks to appear in structured data fields as well -
>> these are preserved. So, in output, you might get line breaks in
>> middle of whole event anyway.
> This might need to be reviewed to protect against such issues.
>
> Regards,
> Botond
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
> _______________________________________________
> nxlog-ce-users mailing list
> nxlog-ce-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
nxlog-ce-users mailing list
nxlog-ce-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nxlog-ce-users
