On Sat, 10 Jul 2021, Daniel Pocock via rsyslog wrote:
On 10/07/2021 22:38, David Lang wrote:
Just a note that in practice, CEE is pretty much dead. Pretty much all
that survived is the idea of using JSON to format the data and to use !
to be able to specify multi-level field names. Everything else in CEE
should be treated as an idea that may or may not be useful rather than a
RFC to be followed.
It is useful enough when trying to normalize logging from a range of
related free software projects
It is useful to use a document that somebody has written rather than
trying to write another one
If anybody knows what the original authors intended for "sev", "pri" and
"syslog!pri", if they ever made an explicit statement in a mailing list,
it would be useful to capture those details. It won't be useful to
everybody but I may put together a small document in the reSIProcate Git
repository about how we are using it with links to any previous
assertions about these fields.
you have pri, which is facility and severity combined, it doesn't make a
huge amount of sense to send all three. In practice the syslog facility
and severity values are what you are going to see.
What I'm doing is putting in the raw values from applications and then
normalizing them to Syslog severity / level values. There is no concept
of Syslog facility in many applications
Therefore, I have values like:
gstreamer!level
resiprocate!level
syslog!level
GStreamer and reSIProcate have more levels than Syslog severity.
However, I also normalize them and send a Syslog severity. I decided to
call it syslog!level - I don't fill in syslog!pri
One other thing to notice is that I'm not capturing any log messages
from regular system daemons. I'm only using this stack to aggregate
messages from SIP and WebRTC. That is why I don't care about using
Syslog facility in this implementation.
If it was a general purpose monitoring setup then I probably would want
to capture the facility value.
facility is fairly worthless nowdays, you should probably put in a localX
facility value, just so that if the logs end up processd by something looking at
it you don't get a '0' (kern) facility.
In practice, I ignore facility and just look at the application name.
What I do is to use a standard syslog header, and then in the JSON body
I don't repeat the info that's in the header (although I do like to
create a trusted!relay!foo set of values that populates info to track
what systems it's gone through, doing set $!trusted!relay!lasthop =
$!trusted!relay; before setting the new values)
Thanks, this is helpful feedback. In my case, I've got everything in a
lab environment where I can avoid multiple hops
yeah, in a lab you don't need it, but when you go to a multi-datacenter
geographically disteributed, redundant system environment, you will end up with
cases where one system in the chain misbehaves and capturing the hops makes it
much easier to troubleshoot :-)
The type of situation we look at in RTC systems involves something like
this, all the events are triggered by a single client. The client is
making independent connections to the WebSocket (TLS), the Media server
(UDP) and the TURN server (UDP). It is important for developers to
visualize those events in chronological order:
HTTP server: incoming connection
HTTP server: upgrade to WebSocket
SIP server: incoming WebSocket proxied by HTTP server
SIP server: incoming SIP INVITE with SDP
Kurento Media Server: incoming SDP offer
TURN server: incoming STUN request
Kurento Media Server: generate SDP answer
Kurento Media Server: incoming RTP media
Kurento Media Server: incoming RTCP report
Just one phone call can create over 1,000 events, that is why the
severity/level is quite important.
yep, make sure you use high precision timestamps and have something in place to
make sure your system clocks are synced. There are various cases where rsyslog
can re-order logs (multi-threaded work under heavy load, logs queued to disk
files, network outages, etc), so the order they appear at the destination is
usually pretty good, but you should not rely on it.
so either high precision timestamps, or a sequence number from the app.
David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
