On Mon, 5 Dec 2016, Bob Gregory wrote:
Hi David,
It's probably best if you _don't_ try to map syslog fields into
riemann fields because the two technologies are accomplishing different
things. Riemann is for processing metrics - numerical data about the state
of our systems, while syslog is about logs - narrative textual data about
our systems.
Service, tags, etc will need to be configured by the end-user; we shouldn't
be guessing what they might be based on our understanding of the log
message.
we need to try and come up with a reasonable default value for parameters.
The reason I would need a Riemann output is that I have three use cases
where I forward data in logs to Riemann from logstash -
1) Logstash's heartbeat (so I can measure latency on my processing pipeline)
2) ERROR and CRITICAL logs so I can alert on them
3) Metrics encoded into json logs by applications.
Service is the "Thing under measurement". The closest analogue would be
programname, but one program might have many services. For example: "http
response time ms", "Bytes read", "Active users", "messages received". Each
of the keys in the key/value messages raised by impstats is a single
service.
Tags are used to aggregate and filter services, they're arbitrary bits of
data; eg. "Message type", "User account type", "ec2 instance type", "site
map area". Our biggest use case for them is in asynchronous processing
pipelines, where we use them to tag the messages we're processing so that
we can see overall throughput and latency, but drill down when we have to.
The metric is the actual measurement, it's a number.
there is only one set of metrics per event (sint64 metric_sint64, double
metric_d, float metric_f), which do you use (or do you use multiple of them?).
Is there an expectation that you only use one?
how do you signal metric types?
i.e.:
values that are counters (running total of messages processed)
vaues that are gauges (the number of messages in a queue)
what are attributes? they are name-value pairs of strings, and you can have an
arbitrary number per event.
The closest analogue to severity is the "state", which is an arbitrary
string. Usually people use the statuses "ok", "warning", "error" etc. but
it's entirely arbitrary. They're mostly used to trigger state changes in
Riemann.
Description is a narrative description of an event. We only use these in a
single use-case, which is that we forward all logs of ERROR level and
higher to riemann so that it can count them, and send us roll-up emails
every hour, or trigger pagerduty. In this use-case, we set the description
to the incoming log message.
Lastly, the TTL is used to control how long a message should be held
in-memory by Riemann. It can be used to keep a snapshot of current state.
We use it for heartbeats - when an event's TTL expires, if we haven't
received another of the same event, we can raise an alert.
Hope that makes more sense - if you're interested in learning more about
Riemann, there's a great introductory video on the site. http://riemann.io/
The only fields that are required are the host, the service, and the metric.
so as I am understanding you, I would look at something along the lines of
host
default to $hostname, point at a variable
time is a 64 bit number
default to $timestamp, point at a either a timestamp variable (to be converted
to unix time) or a strong variable that should be convertable to a number
service is a string
default to programname, point at a variable
description is a string
default to $!msg if it exists, otherwise $msg, point at a variable
state is a strong
default to severity, point at a variable
tags are an array of one or more strings
default to $!event.tags, point at a json object that contains objects and/or
arrays, include only the values of those objects/arrays
TTL is a number
default to 0 if not defined, point at json string variable that should be
convertable to a number
metric is a number
no default,
point at a variable:
if the variable is a single json object, convert the value from a string to
a number
if the variable contains multiple objects, flatten them and append the
object name to the service and convert the value from a string to a number and
send each item separately
so if you have service = "a ", the json object {"foo": {"bar":"1", "baz":"2.5"}}
and pass $!foo as the metric, it will send two messages to riemann:
1) service 'a bar' metric 1
2) service 'a baz' metric 2.5
attributes are name-value pairs of items
no default
point at a json object that contains objects, flatten sub-objects and send as
name-value pairs
This would not require any different formats for impstats, you could take the
json output and feed it directly to this module.
you could also do some tweaking of the data before it's sent (using dyn-stats
names as part of the service)
the statsd module would be very similar, it should have the option to send
normal stats or the datadog extended stats (basically adding tags to the normal
statsd output)
I used foreach and a bunch of custom formats to do similar output to datadog
statsd.
David Lang
-- Bob
On Mon, 5 Dec 2016 at 00:06 David Lang <[email protected]> wrote:
On Mon, 5 Dec 2016, Dave Cottlehuber wrote:
https://github.com/algernon/riemann-c-client may be of interest to use
it directly -- its been dropped into collectd as a library now as well,
and is ported to Debian & FreeBSD already, that I know of. The protobuf
wire format is
https://github.com/algernon/riemann-c-client/blob/master/lib/riemann/proto/riemann.proto
if that's helpful.
it is.
What I've found useful with collectd and riemann was to be able to set
specific custom tags per instance (rsyslog server in our case) which
makes the sorting in riemann very easy prior to parsing any specific
message output. Mainly source & instance type:
it looks like the protobuf allows a lot of options in terms of how to store
the
data.
We can make educated guesses as to what makes sense fro the riemann point of
view, but they will only be guesses
as far as tags go, tagging it as being from rsyslog is an obvious item, and
if
we have tags from mmnormalize, they should go here. What else?
should service be the programname or the faclity?
where would facility/severity be stored? is severity == metric?
what sort of stuff normally goes in the description field?
for the attributes, one obvious one is the message, but beyond that it's
less
clear. Given that rsyslog internally tracks things as JSON, I think putting
each
json object as an attribute makes sense, but attributes can't be nested.
Internally to rsyslog, we deal with nested objects by flattening them and
seperating the tiers with a ! (i.e. {foo:{bar:baz}} == foo!bar:baz), is this
reasonable from a riemann point of view? should we use a different character
instead?
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
